3.12.2.1 (2024-09-16) --------------------- * Fix bug in upstream rocksdb to not leak file descriptors when iouring is used for RocksDB. 3.12.2 (2024-08-21) ------------------- * Add API GET /_admin/cluster/vpackSortMigration/status to query the status of the vpack sorting migration on dbservers, single servers and coordinators. * Use default language en_US (instead of en_US_POSIX as before in 3.12), if no LANGUAGE file is found and none of the locale environment variables LANG, LANGUAGE, LC_ALL and LC_COLLATE are set. This ensures more compatibility with 3.11. This fixes BTS-1941. * FE-461: fix JSON/table detection, disable CSV download when not a table. * FE-442: fix user avatar on user table. * FE-453: Upgrade arangojs version used in web UI. * FE-460: Query editor UI - fix buttons overlap when 'Options' tab is open. * Updated ArangoDB Starter to v0.19.4. * We already had the concept of individual log appenders (e.g., stdout, stderr, files, syslog). but log levels could only be configured globally. That is, all appenders always used the same log levels for all topics. This PR introduces appender specific log levels. For example, it is now possible to set the log levels for all topics to `error`, but leave the log levels for the log file unchanged. This is particularly useful for test runs because we avoid flooding the console with log messages, but still have all the information in the log files in case of test failures. The log level can be specified per appender via the command line by adding a comma separated list of topics with their respective level after the output definition, separated by a semicolon: `--log.output file:///path/to/file;queries=trace,requests=info` `--log.output -;all=error` The `_admin/log/level` endpoint has also been extended to with an optional parameter `withAppenders`. If that parameter is set to `true`, the response has the following form: ``` { "global": { : , ... }, "appenders": { : { : , ... }, ... } } ``` A `PUT` request to `_admin/log/level` with `withAppenders` set to true requires an input of the same format. Previously, requests to the `_admin/log/level` endpoint where not fully validated. A `PUT`' request that specified invalid topics or log levels would silently ignore those. This has been improved, so that invalid input is now always rejected with a proper error message - in that case _no_ changes are applied at all. The global level for some topic is always identical to the highest log level for that topic of all appenders. Thus, changing the log level for an appender will implicitly also adjust the global level accordingly. When a global level is set explicitly, this change is also applied to all appenders. * FE-459: fix failing foxx service download. * Fixed BTS-1742: `IN_RANGE` function now uses MDI index if possible. * FE-456: upgrade web UI to React 18. * FE-449: write concern is no longer set when creating a sattelite database * Add "useCache" option for graph nodes: The "useCache" AQL option is now supported for the following node types in addition: - TraversalNode - ShortestPathNode - EnumeratePathsNode (K_SHORTEST_PATHS, K_PATHS, ALL_SHORTEST_PATHS) The option controls whether the in-memory cache for edges is used for the graph operation. It defaults to `true`. It can be set to `false` to disable usage of the in-memory edge cache for the graph operation in question. Example: FOR v, e, p IN 1..2 OUTBOUND 'vertex/123' edges OPTIONS {useCache:false} ... * Fixed MDS-1225: reloading of AQL user-defined functions inside AQL queries could cause trouble if the `_aqlfunctions` collection is located on different DB server than the leader shards of other collections used in the same query. * Fix comparison of numeric values in AQL to bring it in line with the now correct VPack numerical sorting. * Include LANGUAGE file in hotbackups. This is necessary to be able to detect locale changes across a hotbackup create/restore process. * Fix sorting behaviour of VelocyPack values w.r.t. numbers. This has an impact on indexes indexing VPackValues. Therefore, after an upgrade the old sorting order will be retained to allow smooth upgrades. Newly started instances with a fresh database directory will only use the new sorting method. There is also a migration API under GET /_admin/cluster/vpackSortMigration/check and PUT /_admin/cluster/vpackSortMigration/migrate to check for problematic indexes and - provided there are none - to migrate the instance to the new sorting order. * Forcefully kill all running AQL queries on server shutdown. This allows for a faster server shutdown even if there are some long-running AQL queries ongoing. * Optional logging of AQL query metadata into `_queries` system collection. There is now a functionality to optionally log metadata of finished AQL queries into a system collection for later analysis. The query metadata includes the following attributes: - `id`: internal ID of the query - `database`: name of the database the query ran in - `user`: name of the user that executed the query - `query`: query string - `bindVars`: bind parameters values used by the query - `dataSources`: array of collection names / view names that were used in the query. - `started`: date/time the query started executing - `runTime`: total runtime of the query, in seconds - `state`: the state of the query (will always be "finished") - `stream`: whether or not the query was a streaming AQL query - `modificationQuery`: whether or not the query was a modification query - `warnings`: number of warnings issued by the query - `exitCode`: exit code of the query (0 = success, any other exit code indicates a specific error) Note the following restrictions for the above attributes: - `query` will only contain the query string if the startup option `--query.tracking-with-querystring` is set to `true`. This is the default. The query string will also be cut off after `--query.max-artifact-log-length` characters. - `bindVars` will only contain data if the startup option `--query.tracking-with-bindvars` is set to `true`. This is the default. - `dataSources` will only contain data if the startup option `--query.tracking-with-datasources` is set to `true`. This is not the default. Only finished queries will be logged to the `_queries` system collection. Queries are considered "finished" in case they have executed completely or have failed with an error. In-flight queries will not be logged, but should eventually become finished queries. Queries will only be logged to the `_queries` system collection if the startup option `--query.collection-logger-enabled` is set to `true`. Additionally, queries in the `_system` database are only logged if the option `--query.collection-logger-include-system-database` is also set to `true`. The option `--query.collection-logger-probability` can be used to specify a probability for logging queries to the system collection. If set to a value of `100`, all executed queries will be logged. If set to a value less than `100`, only the specified percentage of queries will be logged. This can be used to draw a random sample from the total set of queries on very busy systems on which tracking every individual query would have prohibitive overhead. For example, using `--query.collection-logger-probability=1` will log approximately every 100th query to the system collection, and ignore all others. Which exact queries are logged and which aren't is based on randomness. The startup option `--query.collection-logger-all-slow-queries` can be used to make all slow queries be logged, regardless of whether they were selected for sampling or not. The slow query time threshold is configurable as usual via the already existing startup options `--query.slow-threshold` and `--query.slow-streaming-threshold`. Query metadata is stored in the `_queries` collection in the `_system` database. This collection will be created automatically when needed. The collection is a normal system collection, so it can be queried by everyone with at least read access to the `_system` database and the `_queries` collection. Keep this in mind when enabling the query logging, as the data may include sensitive information, such as query strings and/or user names. Query metadata can be retrieved by running an AQL query on the `_queries` collection with filters and sort clauses of choice, e.g. FOR doc IN queries FILTER doc.runTime >= 10.0 FILTER doc.user == @user SORT doc.started RETURN doc FOR doc IN queries FILTER doc.started >= @start FILTER doc.started < @end COLLECT db = doc.database, user = doc.user WITH COUNT INTO count RETURN {db, user, count} When query logging is enabled, obsolete entries from the `_queries` collection are purged with a configurable schedule. The startup option `--query.collection-logger-retention-time` is used to determine how long after a query's start time it will be approximately retained in the system collection. For example, `--query.collection-logger-retention-time=86400` will keep query information around for approximately 1 day (86400 seconds). The default retention time is 28800 seconds, i.e. 8 hours. The actual cleanup of the system collection is only executed with a configurable interval as well, so that the cleanup process does not cause much traffic. The cleanup interval can be controlled by setting the option `--query.collection-logger-cleanup-interval`. The value is in milliseconds. The option `--query.collection-logger-push-interval` can be used to set a maximum wait time after which queries are logged to the system collection. This is a performance optimization, which helps to reduce the overhead of query logging. For example, `--query.collection-logger-push-interval=10000` will buffer queries in memory for at most 10000 milliseconds before they are actually written to the system collection. When additional queries arrive within this interval, they will be batched together into a single write operation to the system collection. This can amortize the cost of writing the query metadata to the system collection across multiple user queries. The option `--query.collection-logger-max-buffered-queries` can be used to limit the amount of query metadata buffered in memory before it is flushed to the system collection. Once this limit has been reached, no further query metadata will be buffered in memory and will be lost. To make this relatively unlikely, a flush is triggered automatically once a fourth of the maximum queries limit is reached. It is still possible however that the single-thread flush operation cannot keep up with the rate of incoming queries, so that the limit is reached and some query metadata are not logged. Any queries that are buffered in memory and are not yet flushed out to the system collection will be lost upon an instance shutdown or crash. This means that the query logging functionality should not be used for auditing, but rather for debugging and troubleshooting query issues, such as finding long-running queries, queries that produced warnings or errors, users that overuse the database etc. * FE-457: fix query UI result table headers & table/geo detection. * Add optimizer rule `push-limit-into-index` that pushes a limit down into index iteration and can optimize the query execution when compound index is used along with SORT and LIMIT clauses. * FE-423: Add a progress bar to reflect an index creation process. * Remove unused "mmap" log topic. * Support array and object destructuring for FOR loops that refer to simple expressions or collections. The following is now possible to write in AQL: FOR {value1, value2} IN collection FILTER value1 == @value1 && value2 == @value2 RETURN value1 This destructures every single object from `collection` and extracts its `value1` and `value2` attributes. It is equivalent to writing: FOR doc IN collection FILTER doc.value1 == @value1 && doc.value2 == @value2 RETURN doc.value1 It is also possible to destructure array inputs into separate variables. For example, the following query FOR doc IN collection FOR [key, value] IN ENTRIES(doc) RETURN {key, value} can be used to destructure all members from the expression `ENTRIES(doc)` and assign the first member to the variable `key`, and the second member to the variable `value`. This is equivalent to writing: FOR doc IN collection FOR entry IN ENTRIES(doc) LET key = entry[0] LET value = entry[1] RETURN {key, value} Accessing an undefined object attribute or an undefined array members results in a value of `null` being produced. Applying destructuring on any incompatible input types produces `null` as well. Destructuring is not supported for view-based FOR loops, for traversal FOR loops, SHORTEST_PATH FOR loops, K_PATHS FOR loops, K_SHORTEST_PATHS FOR loops nor ALL_SHORTEST_PATHS FOR loops. * Make constrained heap sort in AQL report all filtered rows as "filtered" in the query profile. * Adjust our usage of `_FORTIFY_SOURCE` in maintainer mode: - use `_FORTIFY_SOURCE` compile definition also when building with clang. Previously it was only used when compiling with gcc. - use `_FORTIFY_SOURCE=3`, which adds runtime memory safety checks (and some runtime overhead). Previously we used `_FORTIFY_SOURCE=2`. This only affects maintainer mode builds. Non-maintainer-mode builds are not affected, as `_FORTIFY_SOURCE` is not used on them. * Added array and object destructuring to AQL LET statements. - Array Destructuring: Array destructuring is the assignment of array values into one or multiple variables with a single `LET` assignment. This can be achieved by putting the target assignment variables of the `LET` assignment into angular brackets. For example, the statement: LET [y, z] = [1, 2] will assign the value `1` to variable `y` and the value `2` to variable `z`. The mapping of input array members to the target variables is positional. It is also possible to skip over unneeded array values. The following assignment will assign the value `2` to variable `y` and the value `3` to variable `z`. The array member with value `1` will not be assigned to any variable: LET [, y, z] = [1, 2, 3] When there are more variables assigned than there are array members, the target variables that are mapped to non-existing array members are populated with a value of `null`. The assigned target variables will also receive a value of `null` when the destructuring is used on a non-array. Destructuring can also be applied on nested arrays, e.g. LET [[sub1], [sub2, sub3]] = [["foo", "bar"], [1, 2, 3]] In the above example, the value of variable `sub1` will be `foo`, the value of variable `sub2` will be `1` and the value of variable `sub3` will be `2`. - Object Destructuring: Object destructuring is the assignment of multiple target variables from a source object value. This is achieved by using the curly brackets after the `LET` assignment token: LET { name, age } = { valid: true, age: 39, name: "John Doe" } The mapping of input object members to the target variables is by name. In the above example, the variable `name` will get a value of `John Doe`, the variable `age` will get a value of `39`. The attribute `valid` from the source object will be ignored. Object destructuring also works with nested objects, e.g. LET { name: {first, last} } = { name: { first: "John", middle: "J", last: "Doe" } } The above statement will assign the value `John` to the variable `first` and the value `Doe` to the variable `last`. The attribute `middle` from the source object is ignored. Note that here only variables `first` and `last` will be populated, but variable `name` is not. It is also possible for the target variable to get a different name than in the source object, e.g. LET { name: {first: firstName, last: lastName} } = { name: { first: "John", last: "Doe" } } The above statement assigns the value `John` to the target variable `firstName` and the value `Doe` to the target variable `lastName`. Note that neither of these attributes exist in the source object. * Added `verifyCertificates` attribute option for the `requests` JavaScript module. This option defaults to false, so that no certificates will be verified in an HTTPS connection made with the `requests` module. If the option is set to true, the server certificate of the remote server will be verified using the default certificate store of the system. There is also a `verifyDepth` attribute to limit the maximum length of the certificate chain that counts as valid. 3.12.1 (XXXX-XX-XX) ------------------- * Fix an "invalid document type" exception when using the AQL ENTRIES function on a database document or edge. * Fixed MDS-1216: restoring the previous value of the "padded" key generator could lead to the key generator's sequence being set to a too low value after recovery. * Updated ArangoDB Starter to v0.19.3. * Fix a nullptr access when aborting a move shard job for a already dropped collection. * Use __nss_configure_lookup to opt out of /etc/nsswitch.conf. Add the startup option --honor-nsswitch to cancel the opt-out. * Fix memory accounting in join executor. * Fix issues with late materialization being sometimes implicitly dependent on the optimizer rule "optimize-projections" being enabled. Now disable some late materialization optimizations in case this rule is not active. Also fix a specific case in which no results were being materialized in case the optimizer rule "optimize-projections" was disabled. * Fix serialization of multiple query plans using the `allPlans: true` hint when run `explain` on a query. * Support indexHint AQL option for traversals. The structure of traversal index hints is as follows: { collection: { direction: { level: indexes } } } with the following meanings: - collection: the name of an edge collection for which the index hint should be applied. Proper capitalization of the collection name is important. - direction: the direction for which to apply the index hint. Valid values are `inbound` and `outbound`, in lowercase. - level: the level/depth for which the index should be applied. Valid level values are the string `base` and any stringified numeric values greater or equal to zero. Example: FOR v, e, p IN 1..5 OUTBOUND startVertex edgeCollection OPTIONS { indexHint: { "edgeCollection": { "outbound": { "base": ["edge"], "1": "myIndex1", "2": ["myIndex2", "myIndex1"], "3": "myIndex3", } } } } FILTER p.edges[1].foo == 'bar' && p.edges[2].foo == 'bar' && p.edges[2].baz == 'qux' Because the collection names and levels/depths are used a object keys here, it is useful to enclose them in quotes. Omitting the quotes around them may result in parse errors for the query. Note that index hints for levels other than `base` will only be considered if the traversal actually uses a specific filter condition for the specified level. In the above example, this is true for levels 1 and 2, but not for level 3. Consequently, the index hint for level 3 will be ignored here. Currently index hints cannot be forced to be used for traversals, i.e. the `forceIndexHint` option is currently not supported for traversals. * Update timezone database as of 25.06.2024. * Rebuilt included rclone v1.65.2 with go1.22.4. * Swagger UI: Remove fragment identifiers from request URLs. They are merely used to disambiguate polymorphic endpoints in the OpenAPI descriptions of the HTTP API documentation. * BTS-1909, MDS-1232: Fixed a bug where COLLECT ... AGGREGATE x = UNIQUE(y) could miss some results when multiple shards were aggregated in a cluster. * Fixed an issue where the request would hang when retrieving the result of an asynchronous HEAD request. Now, the response body is cleared and the Content-Length header is set to 0, ensuring compliance with the HTTP protocol and preventing indefinite client waits. * Document modification operations (INSERT, UPDATE, REPLACE, REMOVE, and UPSERT) during an AQL query on DBServers no longer block threads while waiting for replication. * Fixed a bug where accessing a collection right after creation can sometimes fail. * OASIS-25850: Fix incomplete replacing of projection variables in traversals. This fixes a regression introduced in 3.12.0. * FE-454: Fix saving document in "Tree" mode. * FE-310: Query editor in web UI can now be resized vertically. * Fix a potential data corruption in a collection's Merkle tree, in case a write operation in a streaming transaction ran into the transaction's maximum size limit and failed. If the error was ignored and the transaction committed, the leftovers of the failed operation were not removed from the collection's in-memory Merkle tree buffer and thus be committed as well. This could later cause issues with shards not getting properly in sync. * Fix the handling of View objects in AQL template strings of the JavaScript API so that the behavior matches collection objects: let view = db._view("myView"); aql`FOR doc IN ${view} RETURN doc` * FE-451: update graph UI defaults for edge direction, edge/collection labels & colors. * Fix possible crashes or UB during some document operations on single- or DBServers. * Upgrade OpenSSL to 3.3.1 and used glibc build runtime to 2.39.0. * Updated ArangoDB Starter to v0.19.2. * Avoid dropping of followers in case a leader resigns and then comes back. * Fixed MDS-1229 (bad_function_call with batch materialization). * Make the `beginAsync()` operation for transactions fully async also in the coordinator case, where it could previously block. * Optimize performance of in-memory cache lookups under contention. * Reduce the possibility of cache stampedes during collection count cache refilling. Additionally, do not block or delay a foreground operation with refilling the collection count cache value from the DB-Servers. Instead the refilling is executed eventually via the scheduler in the background. * Move resolution of replication callbacks on leader DB-Servers to the scheduler's HIGH prio lane. The HIGH lane is justified here because the callback resolution must make progress, as it can unblock another thread waiting in an AQL write query for the replication to return. * Skip the scheduler for AQL query shutdown on coordinators. During query shutdown, we definitely want to skip the scheduler, as in some cases the thread that orders the query shutdown can be blocked and needs to wait synchronously until the shutdown requests have been responded to. * Fixed MDS-1230: added missing variable replacement for inline filters of EnumerateListNodes. * Fixed BTS-1813: Fixed an AQL error that could lead to range-check exceptions in certain queries with subqueries. * Move network retry requests to a dedicated thread. * Reduce frequency in which progress of hot backup upload / download progress is reported to the agency. This limits traffic from DB servers to the agency during hot backup uploads and downloads. * Pass all relevant compiler optimization flags down to relevant CMake sub projects which are built using configure/make. * Added ENTRIES function, that takes an object and produces a list of key-value pairs. The order of pairs is unspecified. * Prevent spurious rclone errors in S3. * Use posix_spawn instead of fork/exec for subprocesses. This solves a performance issue during hotbackup upload. * Added new REST API endpoint HTTP DELETE `/_admin/log/level` to reset all log levels to their startup values. The log levels will be reset to their factory defaults unless they were overridden via a configuration file or command-line options. In this case, the log levels will be reset to the value they were configured to at startup. All modifications to the log levels since the instance startup will be lost when calling this API. This API is useful for tools that temporarily change log levels but do not want to fetch and remember the previous log levels settings. Such tools can now simply call the new API to restore the original log levels. * Fixed OASIS-25823: when late materialization was used together with skipping over documents or the `fullCount` operation, it was possible for AQL queries to fail with errors such as `bad_function_call`. * Fixed the listDatabases API: Directly after adding a new empty DBServer, or after restoring a Hotbackup the listDatabases() would return an empty list of existing databases, which goes back to normal quickly. This bug only effected the APIs exposing the list of database names, all databases in fact still exist and are fully functional. * Fix a potential race in collectRebalanceInformation. * FE-447: fix query spotlight search, make it case-insensitive. * FE-443: Fix inconsistent tooltip padding in Create Index dialog. * Fix potentially hanging threads during index creation if starting one of the parallel index creation threads returned an error. * Fix connection retry attempts for cluster-internal TLS connections that ran into the 15 seconds timeout during the connection establishing attempt. In this case, the low-level socket was repurposed, but not reset properly. This could leave the connection in an improper state and lead to callbacks for some requests to not being called as expected. The connection timeout was also increased from 15 seconds to 60 seconds. * FE-448: auto-repair collection document JSON on save. * Improved the time required to create a new Collection in a database with hundreds of collections. This also improves times for indexes and dropping of collections. * Added startup option `--cache.max-cache-value-size` to limit the maximum size of individual values stored in the in-memory cache used for - edge indexes - persistent indexes with caching enabled - collections with document caches enabled The startup option defaults to 4MB, meaning that no individual values larger than 4MB will be stored in the in-memory cache in these areas. Limiting the size of cache values helps to avoid storing very large values in the cache for huge documents or for all the connections of super nodes, and thus leaves more memory for other purposes. * BTS-1856: Fix a data-race inside the Query class during accesses on a private property (`execStats`). * FE-432: allow row clicks on all UI tables. * FE-441: Fix collection unstable default sort in multi-shard collections UI. * Changed default value of `--log.max-queued-entries` from 10000 to 16384. * Added metric `arangodb_logger_messages_dropped_total` to count the number of log messages that were discarded by the logger if its log queue was full. * Added option `skipFastLockRound` for transactions. The option can be set in a streaming write transaction's options (i.e. `{"skipFastLockRound":true}`) to disable the fast lock round inside the transaction. The option defaults to `false, so the fast lock round is tried where possible. Disabling the fast lock round is normally not necessary, but it can make sense when there are many concurrent streaming transactions queued that all try to access the same collection and lock it exclusively. In this case, the fast lock round actually makes things worse, because there will always be some exclusive locks in the way. In this use case it will be better to skip the fast lock round. Disabling the fast mode makes each actual locking operation take longer than when in fast mode, but it guarantees a deterministic locking order and thus avoids deadlocking and retrying that is possible with the fast mode. Overall disabling the fast lock mode can be faster in this case. Setting this option is irrelevant for read-only transactions and will degrade performance when there are no concurrent transactions that use exclusive locks on the same collection. * Fixed SEARCH-485 (ES-2010): Search files disappeared during creation of HotBackup. * Bundle ICU library version 64 with the build, so that persistent indexes and views relying on the old sort order can still be used without problems. The prevent issues with RocksDB going into read-only mode after an upgrade from 3.11, if it encounters persistent index values in an unexpected sort order. RocksDB's error message in this case was "Corruption: Compaction sees out-of-order keys". The out-of-order keys were caused by a changed sort order for some Unicode characters. The newer ICU 73 version that ArangoDB 3.12 bundles includes the most recent changes to the Unicode standard that have been accepted until recently. This leads to problems, because old versions of ArangoDB were using ICU version 64, which was based on an older version of the Unicode standard, and ArangoDB versions from 3.12 onwards were using ICU 73, which implements a very recent version of the Unicode standard. For example, in the Unicode standard implemented by ICU 64, the ASCII apostrophe character \u0027 compares lower than the Unicode apostrophe character \u2019. In ICU 73 both characters compare equal. The sort order for existing on-disk indexes however must not change, because RocksDB relies on the index files being sorted according to a non-changing sort order. ArangoDB from 3.12.0.2 onwards will thus bundle ICU versions 64 and 73, so that indexes in RocksDB and Arangosearch views will continue to use ICU version 64 as they did in previous versions. The V8 engine relies on newer versions of ICU and thus will use ICU version 73. * Make AQL query optimizer create less query execution plan permutations when a query contains multiple occurrences of COLLECT statements under certain conditions: - COLLECT statements that are contained in a single loop that is driven by a traversal or an enumeration over an array now prefer the hashed collect variant. - if the hashed collect variant is explicitly requested via the `OPTIONS` clause of the COLLECT statement (i.e. COLLECT ... OPTIONS {method: "hash"}) then the hashed variant is now always used, even if the maximum number of query execution plans is already reached. Previously only the sorted variant would be used despite the explicit `OPTIONS` hint. * Fix an issue that can cause AQL related RestHandlers to wind up in a WAITING state and never get woken up. This implies that the associated query snippet and transaction cannot be released. If the query contains modification operations, this also implies that the associated collection write lock is not released and can therefore prevent other threads from acquiring the exclusive lock (which is required by e.g., the replication). * Fix an issue that can cause some background load in parallel traversals while waiting for data from upstream. For full details see https://github.com/arangodb/arangodb/pull/20768. * BTS-1848: Issue with Collection Names Containing Non-ASCII Characters in Web UI. * Upgrade bundled version of nghttp2 library to version 1.61.0. * Added internal APIs to dump the contents of the AQL query registry from DB-Servers plus the contents of the transaction manager on coordinators and DB-Servers. * Prioritize requests for commiting or aborting streaming transactions on leaders and followers, because they can unblock other operations. Also prioritize requests in already started streaming transactions and AQL queries over new requests because it is assumed that already started streaming transactions and AQL queries can block other operations from starting, so these should be completed first. The following request priorities have changed: - cluster-internal requests for continuing already started AQL queries have changed their priority from low to medium. This excludes requests that setup AQL queries on DB servers, which still run with low priority. - requests that include the transaction id header are now elevated to run with medium priority if they originally ran with low priority. This excludes requests that begin new transactions or AQL queries. - requests to commit or abort an already running streaming transaction will be elevated from low to medium priority. - requests to HTTP GET `/_api/collection//shards` and GET `/_api/collection//reponsibleShard` are now running with high priority instead of low. Such requests are only used for inspection and have no dependencies. - requests to push further an existing query cursor via the `/_api/cursor` are now running with medium priority instead of low priority. Requests to start new queries still run with low priority. - follower requests that acquire the lock on the leader while the follower tries to get in sync are now running with low instead of medium priority. * Obsolete startup option `--server.max-number-detached-threads`. This option is now obsolete because coroutines are used instead of blocking function calls. * Log an info message if a request was queued in the scheduler queue for 30s or longer. * Add upgrade task to drop `_pregel_queries` system collections. * Fixed an upgrade issue on Agents. When agents are started with `--database.auto-upgrade true` and `--cluster.force-one-shard true`. The upgrade procedure got into an incorrect state. Please note `--cluster.force-one-shard` is not required on the Agent instances. * Fix potential deadlocks in "fast lock round" when starting transactions in cluster. The "fast lock round" is used to send out transaction begin requests to multiple leaders concurrently, without caring about the order in which requests are send out. This can potentially lead to deadlock with other out-of-order requests. Thus the intention of the "fast lock round" is to use a very low timeout for these requests, so that deadlocks would be detected and rolled back quickly. However, there was a code path that triggered the "fast lock round" with long request timeouts, which could lead to long wait times until deadlocks were detected and rolled back. * EE-Only bug-fix MDS-1190: When using arangorestore to restore a SmartGraph, the restore process may generate new internal collection ids. These generated ids were not always updated in the restored SmartGraph schema. If the ids diverge, the restored SmartGraph still works as expected, all data is there and can be queried and modified. Only the schema of the restored SmartGraph could not be changed anymore, e.g. no collections could be added to or removed from the graph. The SmartGraph schema handling now does not consider collection ids anymore, so that schemas can again be modified, even those of existing SmartGraphs that were originally affected by the issue. * FE-316: Edge collection import example now correctly reflects the required `from` and `to` attributes. * FE-437: reinitialize analyzer form when type changes. * Remove "read lock" acquisition calls from shard synchronization because they are now unnecessary. This slightly simplifies the shard synchronization protocol. * Repair shard rebalancer if some server has no leaders of a collection but followers. Previously, this server was then not always considered for leader movements. * Change the default log format. The option log.thread is now by default true. The option log.time-format is now by default `utc-datestring-micros`. * Guarantee conditional-only evaluation of the operands of the AQL ternary operator and the logical AND and OR operators, if any of the operands is a subquery. Given an AQL ternary condition such as condition ? truePart : falsePart the `truePart` expression is now executed only if `condition` is truthy, and the `falsePart` is evaluated only if `condition` is falsy. In previous versions of ArangoDB the `truePart` and `falsePart` operands of the ternary operator were evaluated eagerly and out-of-line if the operands were subqueries. The ternary operator's condition is now pulled out and made available under a separate temporary variable. For example, the query RETURN condition ? truePart : falsePart will internally be rewritten as LET tmp1 = condition RETURN tmp1 ? truePart : falsePart In case the operands are subqueries and must be executed conditionally, the query is evaluated as LET tmp1 = condition LET tmp2 = ( FILTER tmp1 truePart ) LET tmp3 = ( FILTER ! tmp1 falsePart ) RETURN tmp1 ? tmp2 : tmp3 This guarantees that the condition is only evaluated once, and that the `truePart` and `falsePart` operands are only executed if `condition` was truthy or false, respectively. Also only evaluate the right-hand operands of logical AND and logical OR if the left-hand operand evalutes to a truthy value (logical AND) or a falsy value (logical OR). * Allow limiting the amount of async prefetching operations for AQL queries. Since version 3.12, AQL queries can make use of async prefetching. While this is normally beneficial for query performance, the async prefetch operations may cause congestion in the ArangoDB scheduler and interfere with other operations. In order to control the maximum amount of prefetch operations, the following startup options have been added: - `--query.max-total-async-prefetch-slots`: the maximum total number of slots available for asynchronous prefetching, across all AQL queries. - `--query.max-query-async-prefetch-slots`: the maximum per-query number of slots available for asynchronous prefetching inside any AQL query. That means the total number of concurrent prefetch operations across all AQL queries can be limited using the first option, and the maximum number of prefetch operations in every single AQL query can be capped with the second option. These options prevent that running a lot of AQL queries with async prefetching fully congests the scheduler queue, and also they prevent large AQL queries to use up all async prefetching capacity on their own. * Fix BTS-1807: transactions are bound to the user who started it, and the database they were started for. However, there was a code path that was missing the database check, which allowed the transaction id to be used in a request to a different database. The request would not return an error, but it would actually be executed in the context of the original database, and not the one specified in the request. With this fix, such a request will now return an ERROR_TRANSACTION_NOT_FOUND. 3.12.0 (2024-03-21) ------------------- * Revert arangodump exclusion the following cluster collection properties when writing collection parameters files in a single server dump: - replicationFactor - writeConcern - numberOfShards - minReplicationFactor - shardKeys - shards * Accept language settings found in the LANGUAGE file which previous versions have detected for the "C" locale. This is to allow upgrades from 3.11 if the database directory was created with the "C" locale. * FE-437: reinitialize analyzer form when type changes. 3.12.0-rc.1 (2024-03-12) ------------------------ * Rebuilt included rclone v1.65.2 with go1.21.8. * Repair shard rebalancer if some server has no leaders of a collection but followers. Previously, this server was then not always considered for leader movements. * Updated ArangoDB Starter to v0.19.0. 3.12.0-beta.1 (2024-03-07) -------------------------- * Updated ArangoDB Starter to v0.19.0-preview-1. 3.12.0-alpha.1 (2024-03-06) --------------------------- * Updated ArangoDB Starter to v0.18.3. * FE-436: rename `multi_delimiter` analyzer property to `delimiters` on web UI. * FE-424: Adjusted Foxx service install button placement. * Add startup progress reporting for RocksDB .sst file checksum computation. * Upgrade OpenSSL to 3.2.1. * Remove arangosync binary. * Updated rclone to v1.65.2 (built with go1.21.6). * FE-426: Better error messages for Foxx installation failures. When installing Foxx services from the web UI, the error messages for failures or timeouts during the initial download of the service bundle now provide more information about the underlying cause. Additionally the timeout threshold for the initial download of Foxx service bundles from remote sources has been increased from 10 seconds to 60 seconds for both the web UI and the Foxx HTTP API. * Added startup options `--network.compression-method` and `--network.compress-request-threshold` to optionally compress relevant cluster-internal traffic. If `--network.compression-method` is set to 'none', then no compression will be performed. To enable compression for cluster-internal requests, set this option to either 'deflate', 'gzip', 'lz4' or 'auto'. The 'deflate' and 'gzip' compression methods are general purpose, but can have significant CPU overhead for performing the compression work. The 'lz4' compression method compresses slightly worse, but has a lot lower CPU overhead for performing the compression. The 'auto' compression method will use 'deflate' by default, and 'lz4' for requests which have a size that is at least 3 times the configured threshold size. The compression method only matters if `--network.compress-request-threshold` is set to value greater than zero. This option configures a threshold value from which on the outgoing requests will be compressed. If the threshold is set to value of 0, then no compression will be performed. If the threshold is set to a value greater than 0, then the size of the request body will be compared against the threshold value, and compression will happen if the uncompressed request body size exceeds the threshold value. The threshold can thus be used to avoid futile compression attempts for too small requests. Compression for all agency traffic is turned off regardless of the settings of these options. This is important so that the agent instances do not get overwhelmed by the potential extra CPU utilization caused by compression/decompression. * Save some traffic in cluster-internal requests: - Remove disk usage metrics from heartbeat messages (`freeDiskSpaceBytes` and `freeDiskSpacePercent`), as these are not processed by the agency supervision at all. - Do not return `id` and `endpoint` attributes in responses to empty appendEntries requests in the agency, as these attributes are not picked up by the response message handler. - Do not include `x-arango-source` HTTP header in any outgoing requests that the agency makes to other servers. - Do not include `x-arango-hlc` HTTP header in heartbeat messages to the agency's transient store. - Do not return the following HTTP response headers in cluster-internal traffic: - `x-arango-queue-time-seconds` - `x-content-type-options` - `content-security-policy` - `cache-control` - `pragma` - `expires` - `strict-transport-security` - `server` The following responses **do not count** as cluster internal traffic, so that the mentioned HTTP response headers are fully retained for them: - responses by single server instances. - responses by coordinators if the `x-arango-source` HTTP header was not set in the request. The following responses count as cluster-internal traffic, so the HTTP response headers will be removed: - all responses by agents and DB servers, regardless from where the requests were initiated. - all responses from coordinators, if the `x-arango-source` HTTP header was set in the request and is set to the id of a coordinator or DB server. * FE-409: add deletion success notifications in views on web UI. * FE-320: Database name maximum length validation during input. * BTS-1733: prevent dropping of collection if the collection is part of an existing graph. This PR prevents the unintentional dropping of collections that are part of any existing graph, so that graph definitions remain intact even if the user attempts to drop a member collection. This introduces a change in behavior: while previously it was allowed to drop collections that were part of an existing graph, trying to do so will now result in error `ERROR_GRAPH_MUST_NOT_DROP_COLLECTION` (error code `1942` and error message `must not drop collection while part of graph` with more specifics following). This may require a change in client application code that drops individual collections from graphs in order to clean up. The recommended way to drop these collections is to drop the graph instead. Dropping the graph can optionally also delete all associated collections of the graph. Even if this flag is not used, collections that are not part of a graph definition anymore can later be dropped regularly. * Allow changing writeConcern of dependent collections. Previously changing the `writeConcern` property of collections that also have their `distributeShardsLike` property set was disallowed. This is now allowed. * FE-425: adds missing Analyzer types on web UI - "multi_delimiter" and "wildcard". * BTS-1660: When updating (PATCH request) edges with API Gharial and only specifying one of the two edge attributes `_from` or `_to` (not both), the content of the edge was not fully validated. This has been fixed now. * BTS-1661: In addition to BTS-1259: For some operations where a collection is being used, but not known by the graph definition itself, we've introduced a new, more precise error, called: `ERROR_GRAPH_COLLECTION_NOT_PART_OF_THE_GRAPH`. This error will be reported in case a collection is being used, but not part of the graph definition at all. Whereas the `ERROR_GRAPH_REFERENCED_VERTEX_COLLECTION_NOT_USED` will be thrown in such cases like a not known vertex is being used inside an edge documents `_from` or `_to` attribute. * APM-665: RocksDB memory tracking Changed the default value of the following command line options - `--rocksdb.reserve-table-builder-memory` - `--rocksdb.reserve-table-reader-memory` - `--rocksdb.reserve-file-metadata-memory` from `false` to `true` by default to account for the memory usage of the above RocksDB components in the RocksDB block cache and thus the metric `rocksdb_block_cache_usage`. This also allows to better limit the total memory usage of RocksDB. * Index progress visibility for indexes, which are created in background. * Added support for simple external versioning to document operations. UPDATE and REPLACE operations can now be given an optional `versionAttribute` property. If set, the attribute with the name specified by the property is looked up in the to-be-updated/to-be-replaced document. If no such attribute exists, the UPDATE/REPLACE is performed as usual. If such attribute exists, its content is read and compared numerically to the value of the versioning attribute in the document that updates/replaces it. If the version number in the new document is higher than in the document that exists in the database, the UPDATE/REPLACE is performed normally. If the version number in the new document is lower or equal to what exists in the database, the UPDATE/REPLACE is not performed and behaves like a no-op. No error will be returned to the user in this case. This simple versioning can help to avoid overwriting existing data with older versions in case data is transferred from an external system into ArangoDB and the copies are currently not in sync. The `versionAttribute` property can also be used for INSERT operations with `overwriteMode: "update"` or `overwriteMode: "replace"`. It can also be used inside AQL queries by specifying it in the `OPTIONS` clause of an UPDATE/REPLACE/INSERT operation. Examples: // insert normally db.collection.insert({_key: "test", value: 1, ver: 1}); // updates because version attribute value is higher in new document db.collection.update("test", {value: 2, ver: 2}, {versionAttribute: "ver"}); // does not update because version attribute value is lower in new //document db.collection.update("test", {value: 3, ver: 1}, {versionAttribute: "ver"}); // updates because key already exists and version attribute value is //higher in new document db.collection.insert({_key: "test", value: 4, ver: 3}, {overwriteMode: "update", versionAttribute: "ver"}); db._query("UPDATE 'test' WITH {value: 5, ver: 4} IN collection OPTIONS {versionAttribute: 'ver'}"); Note that versioning is opt-in, and that no version checking is performed for operations for which the `versionAttribute` property was not set as part of the update or replace operation or as an option in the AQL query. Also note that version checking only kicks in if both the existing version of the document in the database and the new document version contain the version attribute and they contain numeric values between 0 and 18446744073709551615. If either the existing document in the database or the new document version do not contain the version attribute, or if the version attribute in any of the two is not a number inside the valid range, the update/replace operation will behave as if no version checking was requested. Also note that document removal operations do not support versioning. Removal operations are always carried out normally without checking the version attribute, even if it is specified. * FE-349: show full error message on web UI when moving shards. * Upgraded bundled version of V8 to 12.1.165. * Upgraded bundled version of ICU to 73. * FE-296: Update shard distribution UI. * FE-367: ability to hide columns in table web UI. * Bug-Fix: Also fix the copy (mentioned directly below here) in the ArangoShell. * Bug-Fix: Fixes the unnecessary copy of the `node_modules` folder located in the frontend directory. This will also improve the startup speed of the V8 feature. * No longer use getpwuid, getpwnam, getgrgid and getgrnam on Linux to avoid crashes due to /etc/nsswitch.conf on older Linux distributions. * Escape control characters (e.g. newlines) in the query strings that are logged in an instance's audit log. * FE-377: remove broken link from JSON editor. * Expose REST API endpoints for server options discovery: - GET `/_admin/options-description`: returns a JSON description of all available server options. - GET `/_admin/options`: returns a JSON object with the currently set server options. As these API may reveal sensitive data about the deployment, they can only be accessed from inside the `_system` database. In addition, there is a policy control startup option `--server.options-api` that determines if and to whom the APIs are made available. This option can have the following values: - `disabled`: options APIs is disabled. - `jwt`: options APIs can only be accessed via superuser JWT. - `admin`: options APIs can be accessed by admin users in the `_system` database only. - `public`: everyone with access to `_system` database can access the options APIs. * Only schedule a single compaction per RocksDB column family at any given time. Scheduling mutliple concurrent compactions on the same column family can lead to scheduler threads being stalled under some adverse conditions that lead to a stall inside RocksDB. * Added REST API `/_api/key-generators` to query the available key generators for collections. * Make AQL optimizer rule `move-filters-into-enumerate` also move filters into EnumerateListNodes for early pruning. Previously it could only pull filters into IndexNodes and EnumerateCollectionNodes. * Validate that smartgraph attribute exists and is not changed. Previously this check was performed in the KeyGenerator and therefore only performed for inserts, but not for replace or update operations. Therefore replace or update operations could modify or even remove this attribute. In addition, we now return the error code `ERROR_NO_SMART_GRAPH_ATTRIBUTE` instead of `ERROR_KEY_MUST_BE_PREFIXED_WITH_SMART_GRAPH_ATTRIBUTE` if the attribute is not set. * Fix BTS-1636: Under certain circumstances the path found by a weighted shortest path search was not the path of smallest weight. * Added CMake option `USE_JEMALLOC_CHECKS` to toggle the usage of extra safety checks in jemalloc. The option is currently enabled by default, but can be turned off when there are performance concerns. * Fixed BTS-1703: Cannot chain multiple UPSERTS. * Added AQL function `TO_CHAR(number)`, which produces a character from the given character number. * Added AQL function `REPEAT(string, count, separator)`, which repeats a string for a configurable number of times, with an optional separator. Note that the maximum size of the output string is limited to 16 MB to avoid very large outputs that would consume lots of memory. Exceeding the maximum output string length will make the function return `null` and issue a warning. * Fixed issue #17673: Adds timezone conversion as an optional parameter in all relevant date functions. * Activate more checks in jemalloc: --enable-opt-safety-checks --enable-opt-size-checks This is supposed to help find bugs w.r.t. memory manangement. * FE-406: update docs links on web UI. * Added an optimizer rule `optimize-enumerate-paths` that pulls filters of the form FILTER pathVar.vertices[* RETURN ...] ALL == value or FILTER pathVar.edges[* RETURN ...] ALL == value inside path searches (ALL_SHORTEST_PATHS, K_SHORTEST_PATHS, K_PATHS) into the path search itself. this should lead to better path searches if the filter is selective enough. * BTS-1633: One Shard databases would incorrectly not be indicated as being One Shard databases in the new database view in the frontend. * Added option `--http.handle-content-encoding-for-unauthenticated-requests` for arangod. If set to `true`, the arangod server will automatically uncompress incoming HTTP requests with Content-Encodings gzip and deflate even if the request is not authenticated. If the option is set to `false`, any unauthenticated request that has a Content-Encoding header set will be rejected. This is the default setting. * Added options `--compress-transfer` and `--compress-request-threshold` to all client tools. These options can be used to enable transparent compression of the data that is sent between the client tools and the ArangoDB server. If the option `--compress-transfer` is set to `true`, the client-tools will add an extra HTTP header `Accept-Encoding: deflate` to all requests they make to the server. This allows the server to compress its responses before sending them back to the client tools. If the option is set to `true`, the client will also transparently compress their own requests to the server if the size of the request body (in bytes) is at least the value of the startup option `--compress-request-treshold`. The default value for this option is `0`, which disables compression of the request bodies in the client tools. To opt in to sending compressed data, the option must be set to a value greater than zero. Request body compression will be performed using the deflate compression algorithm. The client tools will also add a `Content-Encoding: deflate` header to the request when the request body was compressed. As compression will use CPU cycles, it should be activated only in case the network communication between the client tools and the server is slow and there is enough CPU capacity left for the extra compression/decompression work. Furthermore, requests should only be compressed when they exceed a certain minimum size, e.g. 250 bytes. Too small requests are typically not compressible very well. * Added startup option `--http.compress-response-threshold` to the ArangoDB server. The value of this specifies the threshold value (in bytes) from which on response bodies will be sent out compressed by the server. The default value for this option is `0`, which disables sending out compressed response bodies. To enable compression, the option should be set to a value greater than 0. The selected value should be large enough to justify the compression overhead. Too small response bodies are typically not well compressible. Regardless of the value of this option, response body compression will only happen in case the client advertised that it expects a compressed response body by sending an `Accept-Encoding: gzip` or `Accept-Encoding: deflate` header with its request. If that header is missing, no response compression will be performed by the server. As compression will use CPU cycles, it should be activated only in case the network communication between the server and clients is slow and there is enough CPU capacity left for the extra compression/decompression work. work. Furthermore, responses should only be compressed when they exceed a certain minimum size, e.g. 250 bytes. * Fixed BTS-1698: Inverted index can be hinted/forced for UPSERT, which is invalid. * Issue a warning when an UPSERT is used on a collection with multiple shards or on a single-shard collection when a DistributeNode is used. In these cases, the UPSERT cannot guarantee it has observed its own writes. The warning can be opted-out from explicitly by setting the `readOwnWrites` option of the UPSERT operation to `false`. * Added "readOwnWrites" option for AQL UPSERT operations. If set to `true` (which is the default), an UPSERT operation will process its inputs one by one. That way an UPSERT can observe its own writes and can handle modifying the same target document multiple times in the same query. When the "readOwnWrite" option is set to `false`, an UPSERT can process its input in batches (a batch is normally 1000 inputs), so it can execute a lot faster. However, when using batches, an UPSERT cannot observe its own writes. So the "readOwnWrite" option should only be set to `false` if it can be guaranteed that the input to the UPSERT will lead to disjoint documents being inserted/updated/replaced. * Optimize equality lookups and prefix matches that use the AQL like function. The optimization will turn `LIKE(doc.attribute, 'fixed string')` into an equality lookup, and `LIKE(doc.attribute, 'prefix%')` into a range query. That allows indexes to be used in case they are present on the underlying attribute. The optimization is applied only if the second parameter to the `LIKE` function is a fixed string and if third parameter to the `LIKE` function is either not set or set to a value of `false`. * FE-388: Update UI colors & fonts to new styles. * Change default value for arangoimport's `--type` file type option from "json" to "auto". The default value of "auto" will guess the type of the import file based on the file extension. The following file extensions will be automatically detected: - .json - .jsonl - .csv - .tsv If the file extension cannot be mapped to any of the above types, the import will assume that the input is JSON. * BTS-1700: arangoimport parser should abort after a reasonable amount of parse errors. This introduces an option `--max-errors` for arangoimport. This option can be used to limit the amount of error messages displayed by arangoimport, and to abort the import after at least this many errors have occurred. The default value for the option is 20, so that imports will quickly stop in case many errors occur. This is a behavior change compared to previous versions, in which the import would continue even when there were many errors. To achive the same behavior with the new version, the value of the arangoimport startup option `--max-errors` can be set to a high value. * Make AQL optimizer rule "interchange-adjacent-enumerations" generate less plan permutations for the case the a non-collection FOR loop is followed by a collection-based FOR loop. This change can reduce the number of plans in complex AQL queries, and thus can save time needed for plan optimization. * FE-372: Truncate long saved query names in UI. * EE Only: When creating a disjoint SmartGraph or a SmartJoin collection where the list of shard names would go from a k digit number to a k+1 digit number, some AQL queries would mismatch the shard list of one collection with the others. The data is all in the correct place, this is an issue only in read operations. * Added metric `arangodb_aql_cursors_memory_usage` to show the memory usage of open, not yet fully consumed server-side query cursors. * Added AQL functions `PARSE_COLLECTION` and `PARSE_KEY` to quickly extract the collection name and document key from an `_id` value without much overhead. * FE-286: refactor user permissions screen. * FE-396: fix JSON editor undo for inverted index addition. * Support alternative syntax for UPSERT queries, so that arbitrary filter conditions can be used. The alternative syntax is: UPSERT FILTER UPDATE|REPLACE ... The key change here is that the UPSERT will produce a new variable `$CURRENT` with the to-be-found document. This variable can be used in the following filter condition to apply arbitrary filter conditions and not just equality matches. * Added AQL statistics for additional document lookups. Also, the JoinNode now exposes additional statistics: - `scannedIndex` - `filtered` - `seeks` - `documentLookups` * Updated jemalloc to the most recent available version of the code. There has not been a release in a while, and this code fixes compile problems using jemalloc with newer versions of clang in ArangoDB's release build environment. * Fix propagation of collection `writeConcern` changes to DB servers. * Change default value of arangodump startup option `--use-parallel-dump` from `false` to `true`. This enables the more parallel dump procedure by default, which allows dumps to be produced more quickly. The parallel dump protocol needs server-side support, and the first ArangoDB version that completely supports it is 3.12. In order to use a 3.12 arangodump to create dumps from servers that run an older version of arangod, the option should be set to `false` when invoking arangodump. The parallel dump option also allows the usage of the `--split-files` option in arangodump. When enabled, the data of each collection/shard can be dumped into multiple independent files, which can be written in parallel, and that can also be restored in parallel. Please note that you will need an arangorestore from version 3.12 or higher to be able to restore a dump that was taken with the `--split-files` option. * BTS-1602: Added new flag `--ignore-collection` to the arangodump binary. Collections can now be excluded by using this flag. This flag can be specified multiple times to exclude multiple collections. * Added `--write-concern` parameter similar to `--replication-factor` to arangorestore. This enables the user to override the write concern for one or multiple collections. * BTS-1259: Not all CRUD requests based on vertex and edge operations using `_api/gharial` performed a check against the actual graph definition. This behavior is now fixed. In case a collection is being used but not known by the graph definition itself, this will now properly be reported by either `ERROR_GRAPH_REFERENCED_VERTEX_COLLECTION_NOT_USED`, in case of a vertex, or by `ERROR_GRAPH_EDGE_COLLECTION_NOT_USED`, in case of an edge. * Added metric `arangodb_aql_cursors_active` to show the number of active AQL query cursors. * Add AQL optimizer rule "async-prefetch", which allows certain AQL query parts to asynchronously prefetch data while other parts of the query are still operating. Several parts of the same AQL query operating concurrently can lead to query performance improvements in case there is still reserve (scheduler) capacity. The following parts of an AQL query can be executed in parallel to other parts: - CalculationNode - CollectNode - EnumerateCollectionNode - EnumerateViewNode - FilterNode - IndexNode - SortNode Asynchronous prefetching is disabled for the following parts of an AQL query - LimitNode: asynchronously prefetching data for a LimitNode must be disabled because otherwise we may overfetch data and the LIMIT values could be violated. - NoResultsNode: the NoResultsNode produces an empty result, and is always quick to execute. There is no need to make it fetch its dependent data asynchronously. - ReturnNode: the ReturnNode normally only passes through existing values and should be quick to execute. - SingletonNode: the SingletonNode only produces a single value and is always quick to execute. It also doesn't have any dependencies. - any nodes inside a subquery If the following nodes occur inside an AQL query, asynchronous prefetching is disabled for the entire query: - InsertNode, UpdateNode, ReplaceNode, RemoveNode, UpsertNode: prefetching is currently not compatible with these node types. - EnumeratePathsNode, ShortestPathNode, TraversalNode: graph nodes currently do not support prefetching in all cases, thus the functionality is turned off. AQL queries in which some nodes are eligible for asynchronous prefetching will be marked with the optimizer rule "async-prefetch" having been applied. The explain output for queries is now extended with a new "Par" column, which shows which nodes in the query are eligible for asynchronous prefetching. All nodes denoted with "✓" will support asynchronous prefetching at runtime. The profiling output for queries is now extended with a new "Par" column, which shows the actual number of parallel calls during query execution. Note that every call to a query node will be counted in "Calls" in addition. "Calls" contains the total number of calls made to the query node, and "Par" contains the number of successful asynchronous prefetch calls. Note that asynchronous prefetching is done on a best effort basis, but there is no guarantee that query parts actually execute concurrently. The reason for this is that asynchronous prefetching tasks are posted to the server's scheduler with normal priority, so they compete for scheduler threads in the same way as other operations do. The more capacity the server's scheduler still has, the more likely it is that query parts can actually run concurrently. * FE-385: fix web UI query import stuck on loading for faulty JSON. * FE-378: fix fields containing spaces during index creation on web UI. * FE-288: Improve collection index UI. * FE-321: add validation for query name in web UI. * Change default value for `--database.extended-names` from `false` to `true`. This will allow end users to use Unicode characters inside database names, collection names, view names and index names by default, unless the functionality is explicitly turned off. Note: once a server in the deployment has been started with the flag set to `true`, it will store that setting permanently. Switching the startup option back to `false` will raise a warning about the option change at startup, but not block the startup. Existing databases, collections, views and indexes with extended names can still be used even with the option set back to `false`, but no new database objects with extended names can be created with the option set back to `false`. So this state is only meant to facilitate downgrading or reverting the option change. When the option is supposed to be left at a value of `false`, all database objects with extended names that were created in the meantime should be removed manually. * Allow the usage of the `_id` attribute in the stored values of persistent indexes. * Improve performance of AQL LimitExecutor for higher limit values. The LimitExecutor now fast-forwards all input rows instead of working on the individual input rows. * Remove special handling for backups taken with ArangoDB 3.5 in hot backup restore. * Extend the internal state of ReadWriteLocks from 32-bit to 64-bit to prevent potential overflows. Previously creating more than 64k streaming transactions at the same time could produce an overflow that would effectively result in undefined behavior. * Remove "database" attribute from the serialized version of AQL IndexNodes, EnumerateCollectionNodes, EnumerateViewNodes, GraphNodes and RemoteNodes. That attribute was never read back. Also removed the "reverse" attribute from the serialized version of AQL IndexNodes, because it was also not read back except from tests. * FE-376: Fix Query UI crash on null values. * FE-375: Adds document counts to collection info screen. * Make replication WAL tailing schedule prefetch requests via the global scheduler, in the same way as the initial dump phase does. In the best case, this can speed up the getting-in-sync phase when getting shards in sync. * Fixed BTS-1239: Restoring a single server dump with system collections into a cluster caused some of those system collections to lose the `distributeShardsLike` setting and also changed the `shardingStrategy` to `enterprise-compat`. Now system collections in the cluster are always restored with `distributeShardsLike` and `shardingStrategy` set to the correct default values. * Added the following server startup options that can be used to limit the resource usage of parallel arangodump invocations: - `--dump.max-memory-usage`: Maximum memory usage (in bytes) to be used by the server-side parts of all ongoing arangodump invocations. This option can be used to limit the amount of memory for prefetching and keeping results on the server side when arangodump is invoked with the `--parallel-dump` option. It does not have an effect for arangodump invocations that did not use the `--parallel-dump` option. Note that the memory usage limit is not exact and that it can be slightly exceeded in some situations to guarantee progress. - `--dump.max-docs-per-batch`: Maximum number of documents per batch that can be used in a dump. If an arangodump invocation requests higher values than configured here, the value will automatically be capped to this value. Will only be followed for arangodump invocations that use the `--parallel-dump` option. - `--dump.max-batch-size`: Maximum batch size value (in bytes) that can be used in a dump. If an arangodump invocation requests larger batch sizes than configured here, the actual batch sizes will be capped to this value. Will only be followed for arangodump invocations that use the `--parallel-dump` option. - `--dump.max-parallelism`: Maximum parallelism (number of server-side threads) that can be used in a dump. If an arangodump invocation requests a higher number of prefetch threads than configured here, the actual number of server-side prefetch threads will be capped to this value. Will only be followed for arangodump invocations that use the `--parallel-dump` option. * Added the following metrics to observe the behavior of parallel arangodump on the server: - `arangodb_dump_memory_usage`: Current memory usage of all ongoing arangodump operations on the server. - `arangodb_dump_ongoing`: Number of currently ongoing arangodump operations on the server. - `arangodb_dump_threads_blocked_total`: Number of times a server-side dump thread was blocked because it honored the server-side memory limit for dumps. * When dropping a database, immediately also mark all collections of the database as dropped. This has the advantage that ongoing operations on these collections can be stopped earlier, and memory for the underlying collections and indexes can be reclaimed sooner. This changes the behavior for some operations, for example: ``` db._createDatabase("testDatabase"); db._useDatabase("testDatabase"); let c = db._create("testCollection"); db._useDatabase("_system"); db._dropDatabase("testDatabase"); // the following operation will now return "collection not found". // in previous versions, it could still be executed. print(c.count()); ``` The new behavior is more consistent with dropping collections, after which operations on the dropped collection also fail immediately. * Fixed BTS-1489: Race condition in AsioSocket shutdown when using SSL. * Added startup option `--transaction.streaming-max-transaction-size` to make the maximum size of streaming transactions configurable. Previously the maximum size of streaming transactions was hard-coded to 128 MiB. This value is also used as default value for the option now. * Track memory usage for write operations in AQL queries. The memory usage of write operations in AQL queries was previously not tracked and not accounted for in AQL queries' own memory usage. The memory used for write operations also was not accounted for when setting a memory limit for the query. The memory used for write operations is now properly tracked, including in-memory write batches and locked keys in RocksDB (for conflict checking). With that change, the memory usage reported for AQL queries that perform write operations will be significantly higher than before, but the values now will reflect the reality a lot better. This has the consequence that previously working AQL write queries can now fail because their actual memory hits the configured per-query or global memory limit threshold. In these cases it is useful to either revisit the per-query and/or global memory limit settings, turn on intermediate commits for the query, or adjust the query so that it uses less memory. * Added metrics for tracking the memory usage of write operations outside AQL: * `arangodb_transactions_rest_memory_usage`: memory of user-initiated write operations. * `arangodb_transactions_internal_memory_usage`: memory usage of internal operations, such as background statistics, TTL index maintenance, replication etc. * Fixed BTS-1531: Async cancel Job returns different error messages. Unify error messages for canceled jobs (error code 21) to "request canceled". Previously, the server could return different messages, either "request canceled", "request has been canceled by user" or "handler canceled". * Fixed BTS-1543: - client tools (e.g. arangoexport) don't refuse to start up with the error `output directory ... is not empty, use --overwrite true to overwrite the data in it if the output directory *only* contains the `ENCRYPTION` file. This is a special case, which can easily occur when invoking any of the client tools with wrong arguments. In this case no harm is done when allowing the tool to run anyway, even without the `--overwrite true` flag. This will potentially clobber the `ENCRYPTION` file, but it is safe to do because we restrict this to the situation when the directory *only* contains the `ENCRYPTION` file. - allow invoking client tools and arangod with `https://`- and `http://`- prefixed endpoints. This is more user-friendly than the previously supported `ssl://`- and `tcp://`-prefixed endpoints. End users are familiar with using `https://` and `http://` than our `ssl://` or `tcp://` endpoints. Internally, `https://` will be converted to `ssl://`, and `http://` will be converted to `tcp://`. * Improve Pregel garbage collection for dropped databases. * Refuse server startup with outdated little-endian on-disk format. The little-endian on-disk format was used for deployments that were created with either ArangoDB 3.2 or 3.3 when using the RocksDB storage engine. Since ArangoDB 3.4, a big-endian on-disk format was used for the RocksDB storage engine, which is more performant. Deployments that were set up with the RocksDB storage engine using ArangoDB 3.2 or 3.3 and that have been upgraded since then will still use the old format. This should not affect many users, because the default storage engine in ArangoDB 3.2 and 3.3 was the MMFiles storage engine. Furthermore, deployments that have been recreated from an arangodump since ArangoDB 3.4 will not be affected, because restoring an arangodump into a fresh deployment will also make ArangoDB use the big-endian on-disk format. When the outdated little-endian on-disk format is detected at startup, the server will bail out with a descriptive error message. The recommend way to migrate the database from the little-endian storage format to the big-endian storage format is to 1. create a full logical backup of the database using arangodump 2. stop the database servers in the deployment 3. wipe the existing database directories 4. restart the servers in the deployment 5. restore the logical dump into the deployment It will not be sufficient to take a hot backup of a little-endian dataset and restore it, because when restoring a hot backup, the original database format will be restored as it was at time of the backup. * BTS-1380: Added traversal order (dfs, bfs, weighted) to explain output. * arangodump and arangorestore will now try to bump up the limit for the number of open file descriptors at startup, on Linux and macOS. This way the tools are allowed to open more files concurrently, which can be useful when dump/restore with many threads. * Added startup option `--descriptors-minimum` to arangodump and arangorestore. This option can be used to set a lower bound value for the open file descriptors limit that is required to invoke arangodump/arangorestore. Setting the option to a value of 0 makes arangodump/arangorestore start regardless of the actual limit for the number of open file descriptors. This option is available on Linux and macOS only. * Renamed arangodump startup option `--use-experimental-dump` to `--parallel-dump`. * Improved experimental dump functionality in arangodump to work in single server mode and to support transport compression and dumping velocypack data. * Obsolete the following options of arangodump: - `--envelope`: setting this option to true previously wrapped every dumped document into a {data, type} envelope. This was useful for the MMFiles storage engine, where dumps could also include document removals. With the RocksDB storage engine, the envelope only caused overhead and increased the size of the dumps. The default value of `--envelope` was changed to `false` in ArangoDB 3.9 already, so by default all arangodump invocations since then created non-envelope dumps. With the option being removed now, all arangodump invocations will unconditionally create non-envelope dumps. - `--tick-start`: setting this option allowed to restrict the dumped data to some time range with the MMFiles storage engine. It had no effect for the RocksDB storage engine and so it is removed now. - `--tick-end`: setting this option allowed to restrict the dumped data to some time range with the MMFiles storage engine. It had no effect for the RocksDB storage engine and so it is removed now. * Added experimental options to arangodump: - `--compress-transfer`: setting this option enables gzip-compression for the server responses to dump requests. Enabling it can significantly reduce traffic volume from the server back to arangodump, in exchange for some CPU overhead for compression on the server and decompression in arangodump. The default value for this option is false. Setting this option only affects the transfer, but not the on-disk format of dumps. - `--dump-vpack`: setting this option will dump data in raw velocypack format. This will make arangodump operate faster and create smaller dumps. The default value for this option is false, as enabling it would incompatibly change the on-disk format of dumps. * FE-300: Implement new Views list UI. * Fix a timeout discrepancy between HeartbeatThread and DBServerAgencySync during shutdown. The HeartbeatThread on DB servers waited for at most 65 seconds for the DBServerAgencySync to complete, but DBServerAgencySync could make requests to the agency with a timeout of 120 seconds. The timeouts have now been unified, so that the request timeout is 60 seconds and that the HeartbeatThread waits for at most 65 seconds (as before). * BTS-842: make the error message for edge document validation slightly clearer. * FE-248: Improve collections list UI. * BTS-1124: wait for license to be applied locally before responding to the request. * BTS-1566: remove redundant responsibility for creating local databases on coordinators. * Prepare internal NetwortkFeature to transparently handle gzip-encoded responses. * FE-309: Store created_at timestamp for queries in web UI. * Remove attribute `dfdb` from response of storage engine API (GET `/_api/engine`). * The storage engine API (GET `/_api/engine`) now returns an `endianness` attribute for the RocksDB storage engine on single servers and database servers. * Print arangod executable's build-id in startup and crash messages. * Removed --jslint from arangosh and scripts/jslint.sh along with the outdated copy of eslint; updated CONTRIBUTING.md to reflect the changes and hint at scripts/eslint.sh which can be used to lint javascript code with a locally installed version of eslint * Fix Pregel running on OneShard deployments. * FE-94: Support for query options and optimizer rules on web UI. * FE-308: Support for graph editing on web UI. * FE-313: Add support for minHash analyzers & offset field on web UI. * Improve crash handler by adding the state of the application server (e.g. startup, running, shutting down) into the crash message. * Remove long-deprecated graph compatibility functions, which were implemented as JavaScript user-defined AQL functions since ArangoDB 3.0: - arangodb::GRAPH_EDGES(...) - arangodb::GRAPH_VERTICES(...) - arangodb::GRAPH_NEIGHBORS(...) - arangodb::GRAPH_COMMON_NEIGHBORS(...) - arangodb::GRAPH_COMMON_PROPERTIES(...) - arangodb::GRAPH_PATHS(...) - arangodb::GRAPH_SHORTEST_PATH(...) - arangodb::GRAPH_DISTANCE_TO(...) - arangodb::GRAPH_ABSOLUTE_ECCENTRICTIY(...) - arangodb::GRAPH_ECCENTRICTIY(...) - arangodb::GRAPH_ABSOLUTE_CLOSENESS(...) - arangodb::GRAPH_CLOSENESS(...) - arangodb::GRAPH_ABSOLUTE_BETWEENNESS(...) - arangodb::GRAPH_BETWEENNESS(...) - arangodb::GRAPH_RADIUS(...) - arangodb::GRAPH_DIAMETER(...) These functions were only available previously after explicitly calling the `_registerCompatibilityFunctions()` function from any of the JavaScript graph modules. The `_registerCompatibilityFunctions()` exports have also been removed from the JavaScript graph modules. * Removed the following long-deprecated features for the HTTP server: - overriding the HTTP method by setting one of the HTTP headers - `x-http-method` - `x-http-method-override` - `x-method-override` This functionality could previously be enabled by starting the server with the startup option `--http.allow-method-override`. The functionality has now been removed and setting the startup option does nothing. - optionally hiding ArangoDB's `server` response header. This functionality could optionally be enabled by starting the server with the startup option `--http.hide-product-header`. The functionality has now been removed and setting the startup option does nothing. * Fixed BTS-1302: make "silent" option for document operations work in cluster. When using the "silent" options for document operations in the cluster, the coordinator will now only return the errors that happened, as it did in single server for a long time. This is a bugfix and feature alignment, but still a downwards-incompatible change compared to previous versions, where the "silent" option in document operations had no effect in the cluster. This change also unifies the behavior of batch document operations in single server and cluster with the behavior of document operations for SmartGraph edge collections: previously, batch document operations in SmartGraph edge collections stopped upon the first error, whereas they were (intentionally) continued for the remaining operations in the batch in single server and cluster. This is a downwards-incompatible change compared to previous versions, but it can be considered a bugfix. * FE-291: Refactor Query screen on web UI to React. * FE-313: Upgrade arangojs to v8.x on web UI. * Expose the "high water multiplier" for the in-memory cache subsystem via a new startup option `--cache.high-water-multiplier`. The high water multiplier is used to calculate the effective memory usage limit for the in-memory cache subsystem. The cache's configured memory usage limit (`--cache.size`) is multiplied by the high water multiplier, and the resulting value is used as the effective memory limit. It defaults to 56% of the configured cache size. * Wrap imports into SmartGraph edge collections via the REST import API (POST `/_api/import`) into a managed transaction, so that edge data is either completely inserted or not at all. * Changed default values for the following options related to the in-memory cache subsystem: - `--cache.ideal-lower-fill-ratio` changed from 0.04 to 0.08 - `--cache.ideal-upper-fill-ratio` changed from 0.25 to 0.33 This allows slightly better hash table utilization and earlier memory reclamation. * Renamed memory usage metrics for better naming consistency: - `arangodb_internal_index_estimates_memory` was renamed to `arangodb_index_estimates_memory_usage` - `arangodb_scheduler_queue_memory` was renamed to `arangodb_scheduler_queue_memory_usage` - `arangodb_scheduler_stack_memory` was renamed to `arangodb_scheduler_stack_memory_usage` These metrics were introduced in 3.12, so no previous versions are affected. * FE-301: Unify Users View on web UI. * Added deprecation message for parts of createCollection API. So far this API ignored unknown options, and therefore did not warn about typos. Furthermore some combinations of values that are not sensible are accepted. In some future version of ArangoDB those will be rejected by the API. The documented part of the API does not change. Watch out for an error with logid: ee638, this indicates you call deprecated parts. * Remove TTL and Observe/Unobserve from the Agency API. * FE-290: Unify Graphs View on web UI. * FE-289: Improve database table UI. * BTS-744: Disable development mode for system services. * Lower default value of `--query.log-memory-usage-threshold` from 4GB to 1GB. This leads to queries that consume 1GB or more memory to be logged so that they can be investigated and improved. * Set threshold for RocksDB automatic WAL file deletion by age from 30 days to 30 years. This should effectively not change any behavior, as we don't rely on RocksDB's built-in WAL file deletion mechanism. Instead, we tell RocksDB explicitly to delete which WAL files, depending on what other functionality (e.g. replication, index creation) still needs them or not. * Deprecate the startup option `--rocksdb.wal-archive-size-limit` because it can be unsafe to use in environments that use replication (cluster, active failover, single-to-single replication). When using the option and limiting the total size of the WAL files in the archive, this may lead to still-needed WAL files being pruned too early. This can have side effects, such as aborting ongoing replication or index creation attempts. * Don't show "write query options" for AQL write queries in the explain and profile output. The write query options normally don't produce any valuable insights and only take up screen real estate. * If a startup option has a dynamic default value (e.g. if the default value depends on the amount of available RAM), display a "dynamic default" hint when invoking `--help`. * FE-229: Improve shard rebalancing UI. * FE-262: Improve Analyzer table UI. * FE-223: refactor arangosearch, handle nested fields special characters. * FE-166: Handle a space separated list to display multiple node labels in the Graph Viewer. * FE-15: Removed green checkmark next to the version number and made keyboard shortcuts button easier to discover. * BTS-846: Relaxed setup cost for calculation of default sort cost. There was a hardcoded value of 100 being added to the sort cost that would be dominant when there was a small amount of documents in the collection. Even if an index would be the best candidate, if it doesn't cover a sort node present in a query, it might end up not being chosen because the default sort cost would be minimum 100 because of the former hardcoded value. * BTS-1396: Restricted followers from attempting to send telemetrics data to the endpoint on active failover mode. * APM-671: index selectivity estimates (cuckoo estimator) memory tracking. This adds a new metric "arangodb_internal_index_estimates_memory" that reports the total memory usage of all index selectivity estimates combined. The metric includes memory used for buffering index write operations that still have to applied. * APM-663: revision tree memory tracking. Tracking memory used for buffered updates to all revision trees in a new metric `arangodb_revision_tree_buffered_memory_usage`. * Optimize hibernation and resurrection of revision trees so that there are no recurring hibernations and resurrections of the same tree quickly after each other. This is achieved by tracking the time of the last hibernation attempt for each revision tree and not hibernating it if the last attempt was less than a minute ago. * Removed the long-deprecated JavaScript-based traversal functionality, including the following APIs: - REST API endpoint `/_api/traversal` - JavaScript module `@arangodb/graph/traversal` The functionality provided by these APIs was deprecated and unmaintained for many years. JavaScript-based traversals were replaced with AQL traversals in ArangoDB version 3.0. Additionally, the JavaScript-based traversals could not handle larger amounts of data and were thus very limited. * Updated bundled version of libunwind to 1.7-rc2. * Remove function `getFishbowlStorage()` from bundled JavaScript module `@arangodb/foxx/manager`. This function was publicly exposed in previous versions, and returned a handle to the `_fishbowl` system collection. Applications should not have depended on it, because that collection was only used to temporarily store the list of Foxx applications stored in a central Github repository maintained by ArangoDB. * BTS-486: fishbowl collection creation interferes with hot backup. Do not dynamically create the `_fishbowl` system collections anymore to store the list of Foxx applications stored from a central Github repository maintained by ArangoDB. Instead, store the list in memory. This has the advantage that the `_fishbowl` system collection does not need to be created on the fly, which could have interfered with backups running at the very same time. Additionally the previous `_fishbowl` collection was database-specific, so one instance of the collection could be created per database in a deployment. Now, the list of applications stored in memory is shared between all databases in a deployment. * Fixed GitHub issue #17291: Fixed a server crash which could occur in case an AQL query using a PRUNE or FILTER statement, combined with UDFs (user defined functions), got executed. v3.11.8 (2024-02-22) -------------------- * Updated arangosync to v2.19.6. * Updated OpenSSL to 3.0.13 and OpenLDAP to 2.6.7. * Updated ArangoDB Starter to v0.18.2. * Rebuilt included rclone v1.62.2 with go1.21.6. * ES-1892: Fix hot restores missing user defined analyzers. * Fix: we cannot update a link, but have to drop and recreate it. Until now, this new index had the same set of labels as the old one. However, on followers (and with replication2 also on the leader), the DropIndex and EnsureIndex actions could run concurrently. I.e., we could try to create the new index before the old one was fully removed. In this case we could get a duplicate metric exception, preventing the index from being created. Such errors are not really handled ATM - they are simply logged and otherwise ignored. That means the index will simply not be available on the affected server, since we will also do not retry to create it at a later time. To avoid this, we add a new `indexId` label to the metric to make it unique. v3.11.7 (2024-01-29) -------------------- * BTS-1751: Strange error message when executing a query while creating an index in the background. * Reduce default amount of per-collection document removals by TTL background index creation thread from 1m to 100k in each iteration. This change gives other collections a chance of being cleaned up as well. * Do not make AQL query runtime timeout affect TTL index background thread removal queries. * New API to show progress in background index creation. * Updated ArangoDB Starter to v0.18.0. * BTS-1741: fix updates of values in unique persistent indexes with stored values defined for them. When such an index value was updated, it was possible that the stored value was not correctly updated, so that subsequent reads of the index value would run into exceptions such as `Expecting type Array or Object`. * APM-828: Per collection/database/user monitoring. This adds optional metrics for tracking per-shard requests on DB-Servers. The exported metrics are: - `arangodb_collection_leader_reads_total`: number of read requests on leaders, per shard, and optionally also split by user. - `arangodb_collection_leader_writes_total`: number of write requests on leaders, per shard, and optionally also split by user. The new startup option `--server.export-shard-usage-metrics` can be used to opt in to these metrics. It can be set to one of the following values on DB-Servers: - `disabled`: no shard usage metrics are recorded nor exported. This is the default value. - `enabled-per-shard`: this will make DB-Servers collect per-shard usage metrics. - `enabled-per-shard-per-user`: this will make DB-Servers collect per-shard and per-user metrics. This is more granular than `enabled-per-shard` but can produce a lot of metrics. If enabled, the metrics are only exposed on DB servers and not on Coordinators or single servers. Whenever a shard is accessed in read or write mode by one of the following operations, the metrics are populated dynamically, either with a per-user label or not, depending on the above setting. The metrics are retained in memory on DB-Servers. Removing databases, collections or users that are already included in the metrics won't remove the metrics for these databases, collections or users until the DB-Server is restarted. The following operations increase the metrics: - AQL queries: an AQL query will increase the read or write counters exactly once for each involved shard. For shards that are accessed in read/write mode, only the write counter will be increased. - Single-document insert, update, replace, and remove operations: for each such operation, the write counter will be increased once for the affected shard. - Multi-document insert, update, replace, and remove operations: for each such operation, the write counter will be increased once for each shard that is affected by the operation. Note that this includes collection truncate operations. - Single- and multi-document read operations: for each such operation, the read counter will be increased once for each shard that is affected by the operation. The metrics are increased when any of the above operations start, and they are not decreased should an operation abort or if an operation does not lead to any actual reads or writes. Note that internal operations, such as internal queries executed for statistics gathering, internal garbage collection, and TTL index cleanup are not counted in these metrics. Additionally, all requests that use the superuser JWT for authentication and that do not have a specific user set, are not counted. Requests are also only counted if they have an ArangoDB user associated with them, so that the cluster must also be running with authentication turned on. As there can be many of these dynamic metrics based on the number of shards and/or users in the deployment, these metrics are turned off by default (see above), and if turned on, are only exposed only via a new HTTP REST API endpoint GET `/_admin/usage-metrics`. They are not exposed via the existing metrics API endpoint GET `/_admin/metrics`. Add additional metrics for tracking the number of bytes read or written per shard on DB-Servers: - `arangodb_collection_requests_bytes_read_total`: This metric exposes the per-shard number of bytes read by read operation requests on DB-Servers. It is increased by AQL queries that read documents or edges and for single- or multi-document read operations. The metric is normally increased only on the leader, but it can also increase on followers if "reads from followers" are enabled. For every read operation, the metric will be increased by the approximate number of bytes read to retrieve the underlying document or edge data. This is also true if a document or edge is served from an in-memory cache. If an operation reads multiple documents/edges, it will increase the counter multiple times, each time with the approximate number of bytes read for the particular document/edge. The numbers reported by this metric normally relate to the cumulated sizes of documents/edges read. The metric is also increased for transactions that are started but later aborted. Note that the metric is not increased for secondary index point lookups or scans, or for scans in a collection that iterate over documents but do not read them. - `arangodb_collection_requests_bytes_written_total`: This metric exposes the per-shard number of bytes written by write operation requests on DB-Servers, on both leaders and followers. It is increased by AQL queries and single-/multi-document write operations. The metric is first increased only the leader, but for every replication request to followers it is also increased on followers. For every write operation, the metric will be increased by the approximate number of bytes written for the document or edge in question. If an operation writes multiple documents/edges, it will increase the counter multiple times, each time with the approximate number of bytes written for the particular document/edge. An AQL query will also increase the counter for every document or edge written, each time with the approximate number of bytes written for document/edge. The numbers reported by this metric normally relate to the cumulated sizes of documents/edges written. For remove operations however only a fixed number of bytes is counted per removed document/edge. For truncate operations, the metrics will be affected differently depending on how the truncate is executed internally. For truncates on smaller shards, the truncate operation will be executed as the removal of the individual documents in the shard. Thus the metric will also be increased as if the documents were removed individually. Truncate operations on larger shards however will be executed via a special operation in the storage engine, which marks a whole range of documents as removed, but defers the actual removal until much later (compaction process). If a truncate is executed like this, the metric will not be increased at all. Writes into secondary indexes are not counted at all. The metric is also increased for transactions that are started but later aborted. These metrics are not exposed by default. It is only present if the startup option `--server.export-shard-usage-metrics` is set to either `enabled-per-shard` or `enabled-per-shard-per-user`. With the former setting, the metric will have different labels for each shard that was read from. With the latter setting, the metric will have different labels for each combination of shard and user that accessed the shard. If enabled, the metrics are only exposed on DB servers and not on Coordinators or single servers. Note that internal operations, such as internal queries executed for statistics gathering, internal garbage collection, and TTL index cleanup are not counted in these metrics. Additionally, all requests that use the superuser JWT for authentication and that do not have a specific user set, are not counted. Requests are also only counted if they have an ArangoDB user associated with them, so that the cluster must also be running with authentication turned on. * Validate that the attribute stored in the `smartGraphAttribute` of SmartGraph vertex collections exists and is not changed afterwards by update or replace operations. Previously the `smartGraphAttribute` value was checked only when inserting documents into a SmartGraph vertex collection, but not for update or replace operations, although the documentation always stated that the `smartGraphAttribute` value must not be changed after the initial creation of the document. The missing checks on update/replace allowed users to retroactively modify the value of the `smartGraphAttribute` for existing documents, which could have led to problems when the data of such a SmartGraph vertex collection was replicated to a new follower shard. On the new follower shard, the documents went through the full validation, and led to documents with modified `smartGraphAttribute` values being rejected on the follower. This could have led to follower shards not getting in sync. Now the value of the `smartGraphAttribute` will be fully validated with every insert, update or replace operation, and every attempt to modify the value of the `smartGraphAttribute` retroactively will fail with error 4003 (`ERROR_KEY_MUST_BE_PREFIXED_WITH_SMART_GRAPH_ATTRIBUTE`, error message "in smart vertex collections _key must be a string and prefixed with the value of the smart graph attribute"). Additionally, if upon insert the `smartGraphAttribute` is missing for a SmartGraph vertex, the error code will be error 4001 ( `ERROR_NO_SMART_GRAPH_ATTRIBUTE`, error message "smart graph attribute not given") instead of error 4003. To retroactively repair the data in any of the affected collections, it is possible to update every (affected) document with the correct value of the `smartGraphAttribute` via an AQL query as follows: FOR doc IN @@collection LET expected = SUBSTRING(doc._key, 0, FIND_FIRST(doc._key, ':')) LET actual = doc.@attr FILTER expected != actual UPDATE doc WITH {@attr: expected} IN @@collection COLLECT WITH COUNT INTO updated RETURN updated This will update all documents with the correct (expected) value of the `smartGraphAttribute` if it deviates from the expected value. The query will return the number of updated documents as well. The bind parameters necessary to run this query are: - `@@collection`: name of a SmartGraph vertex collection to be updated - `@attr`: attribute name of the `smartGraphAttribute` of the collection * FE-385: fix query import. * BTS-1731: protect streaming transaction garbage-collection from deletion of the transaction's underlying database. * BTS-1727: Return proper EXIT_UPGRADE_REQUIRED in cluster mode. * Removal artificial upper bound value of `128` for the startup option `--rocksdb.max-background-jobs`. * Added stored values support for ZKD indexes. * Fixed a crash during recursive AstNode creation when an exception was thrown. This can happen on DB servers when a query plan snippet is created from VelocyPack, and the query plan snippet would use more memory than is allowed by the query memory limit. * Fix cmake setup for jemalloc library for the case of memory profiling. * BTS-1714: Within writing AQL queries the lock timeout was accidentally set to 2 seconds for all write operations on followers. This could lead to dropped followers when an index of a large shard was finalized on an follower. * Fixed BTS-1701: Assertion triggered in AQL Traversal on edge PRUNE. * Fix an issue when forced index hints were used in a query, but the optimizer selected a query execution plan that would not use the selected index. Previously, the query failed with a "Could not serve index hint" error message. With the fix, the optimizer will select the next best plan(s) until all index hints are either satisfied, or there are no further plans to select from. In the latter case, the query still aborts with said error. v3.11.6 (2023-11-29) -------------------- * FE-403: Fix loader instantiation. * Fix display of "unique" column in indexes overview of collections. * Updated OpenSSL to 3.0.12. * Fixed MDS-1170: Significant performance degradation when queries are executed within transactions that involve edits. * Solve a potential blockage during hotbackup by not stopping read-only transactions from committing during hotbackup. Furthermore, improve behavior of authentication in the case that the user cache is outdated. * FE-395: Fix query editor map not loading. * Stabilize detaching of threads test. * Rebuilt included rclone v1.62.2 with go1.21.4. * Updated ArangoDB Starter to v0.17.2 and arangosync to v2.19.5. * Track memory usage of internal connection statistics and request statistics: - `arangodb_connection_statistics_memory_usage` - `arangodb_requests_statistics_memory_usage` These metrics will remain at 0 if the server is started with the option `--server.statistics false`. Otherwise they will contain the memory usage used by connection and request statistics. Memory usage should remain pretty constant over time, unless there are bursts of new connections and/or requests. * Avoid memory leak in case an arangod instance is started with the option `--server.statistics false`. Previously, with that setting the request and connection statistics were built up in memory, but were never released because the statistics background thread was not running. * Avoid memory leak in case an arangod instance is started with the option `--server.statistics false`. Previously, with that setting the request and connection statistics were built up in memory, but were never released because the statistics background thread was not running. * Remove version check on startup of arangosh. This can speed up the startup of the arangosh considerably because it won't do a network request to www.arangodb.com. v3.11.5 (2023-11-09) -------------------- * Fixed a problem in ReadWriteLock which could prevent waiting readers from being woken up, when a write lock acquire timed out. * Allow a scheduler thread to detach itself from the scheduler if it sees that it has to perform a potentially long running task like waiting for a lock. This allows a new scheduler thread to be started and avoids that it can happen that all threads are blocked waiting for a lock, which has in the past led to deadlock situations. The number of detached threads is limited by a configurable option. Currently, only threads waiting for more than 1 second on a collection lock will detach themselves. * MDS-1164: Added Publisher and Company to the Windows binaries. * Silence TSAN for shutdown for access to the SchedulerFeature::SCHEDULER pointer using atomic references. * Fixed a race in controlled leader change which could lead to a situation in which a shard follower is dropped when the first write operation happens. This fixes BTS-1647. * Fixed BTS-1273: While queueing an async server log message, we could be blocked by IO on the log-writer thread. This could slow down the main path. In case the log-writer is configured to use slow device (e.g. using syslog) this could have significant impact. * Introduced an upper bound of queued async log messages. If we would log more messages then the background logger thread can actually process, we start to write log messages synchronously. This is to prevent the queue to grow indefinitely. The upper bound is configurable via the startup option `--log.max-queued-entries`. The default value is 10000. * Fixed an unnecessary follower drop in controlled leader change, which will speed up leader changes. This fixes BTS-1658. * Fixed BTS-1669 Transaction Manager returns Error if we Abort an Expired Transaction. There was a small time window of around 2 seconds in which aborting expired transactions would return "transaction aborted" instead of returning success. The time window was between when a transaction expired (according to its TTL value) and when the transaction manager's garbage collection aborted the transaction. The issue only happened for transactions which outlived their TTL value and for which an abort operation was attempted in that time window. * Backport multiple fixes for the scheduler behavior during ArangoDB shutdown. * FE-374: Query UI - fix switching from table to JSON. * Make AQL query cursor garbage-collection clean up more expired cursors in a single run than before. This change can help to reclaim memory of expired cursors quicker than in previous versions. * Updated OpenSSL to 3.0.11. * Reduce number of atomic shared_ptr copies in in-memory cache subsystem. * Fixed BTS-1610: In certain situations with at least three levels of nested subqueries, of which some of the outer iterations don't return any results, results of later iterations could be lost. * Updated arangosync to v2.19.4. * Add cluster-internal connectivity checks. This makes Coordinators and DB-Servers in a cluster periodically send check requests to each other, in order to check if all servers can connect to each other. If a cluster-internal connection to another Coordinator or DB-Server cannot be established within 10 seconds, a warning will be logged. The new startup option `--cluster.connectivity-check-interval` can be used to control the frequency of the connectivity check, in seconds. If set to a value greater than zero, the initial connectivity check is performed approximately 15 seconds after the instance start, and subsequent connectivity checks are executed with the specified frequency. If set to a value of zero, connectivity checks are disabled. This change also adds the metrics `arangodb_network_connectivity_failures_coordinators` and `arangodb_network_connectivity_failures_dbservers`, which can be monitored for detecting temporary or permanent connectivity issues. * Fix ineffective startup option `--rocksdb.partition-files-for-documents`. Setting this option had no effect unless RocksDB version 8 was used. v3.11.4 (2023-10-04) -------------------- * Fix updating of collection properties (schema) when the collection has a view on top. * FE-323: allow 'nested' property in view JSON UI. * Slightly extended hot backup release lock timeout on coordinators from `timeout + 5` seconds to `timeout + 30` seconds, to prevent premature release of hot backup commit locks on coordinators. * Improve logging in case hot backup locks cannot be taken on coordinators. * BTS-1618: Preventing the problem from ever occurring on 3.11. * Speed up incremental hotbackup upload by parallelization of remote-to-remote copies. Tolerate deletion of old backups during incremental upload. * Added the following metrics to improve observability of the in-memory cache subsystem: - `rocksdb_cache_free_memory_tasks_total`: total number of `freeMemory` tasks scheduled - `rocksdb_cache_migrate_tasks_total`: total number of `migrate` tasks scheduled - `rocksdb_cache_free_memory_tasks_duration_total`: total time (microseconds) spent in `freeMemory` tasks - `rocksdb_cache_migrate_tasks_duration_total`: total time (microseconds) spent in `migrate` tasks * Improve performance of the in-memory cache's memory reclamation procedure. The previous implementation acquired too many locks, which could drive system CPU time up. * Fix MDS-1157: taking hot backups with --allow-inconsistent=false option always reported a warning that the backup was potentially inconsistent, although taking the backup actually succeeded. The warning message was `Failed to get write lock before proceeding with backup. Backup may contain some inconsistencies.`, but it was a false positive and the backups taken were actually consistent. * Fixed issue with `keepNull=false` updates not being properly replicated to followers. * Updated JavaScript dependencies: qs: 6.11.0 -> 6.11.2 semver: 7.3.8 -> 7.5.4 This addresses CVE-2022-25883 in the semver module. * FE-355: fix ctrl + A for search box. * FE-365: fix query import breaking due to extra fields. * Fixed BTS-1613: Fixed processing analyzers imported from Cluster dump to the Single server database. * Removed DocuBlocks and obsolete documentation tooling. The HTTP API descriptions are now in the `arangodb/docs-hugo` repository. * Added metric `rocksdb_cache_edge_empty_inserts_total` to count the number of inserts into the edge cache for non-connected edges. * Renamed two edge-cache related metrics to improve naming consistency: - `rocksdb_cache_edge_effective_entries_size` was renamed to `rocksdb_cache_edge_inserts_effective_entries_size_total` and was changed from a gauge to a counter. - `rocksdb_cache_edge_uncompressed_entries_size` was renamed to `rocksdb_cache_edge_inserts_uncompressed_entries_size_total` and is now also a counter instead of a gauge. * Do not auto-reload entries into in-memory edge cache for edge index entries that were previously not contained in the in-memory edge cache. This change will help to keep the hot set in memory rather than evicting it in favor of "random" other index entries. * Properly track memory usage for allocated objects in the in-memory cache (e.g. the edge cache or the in-memory cache for other persistent indexes). Previously the memory used for the underlying hash tables was accounted for correctly, but the sizes of the cache payloads (keys and values) were not accounted for under all circumstances (at least for the initial entries in the caches). This change leads to more accurate memory usage tracking and reporting by the in-memory cache subsystem, and to the cache subsystem not exceeding its configured memory usage limit. The cache subsystem was also changed so that it can use as much memory as configured by the global cache memory limit (configurable via startup options `--cache.size` and `--cache.high-water-multiplier`). Previously the cache subsystem was freeing memory as soon as it hit 56% of the configured limit. Overall, the effective memory usage of the cache subsystem can be different to the cache memory usage in previous versions. In previous versions the configured memory usage limit could be temporarily exceeded, but in most cases the cache used considerably less memory than allowed by the limit. Effectively the memory usage was capped at 56% of the configured limit. Now the cache will try to use up to as much memory as allowed by the configured memory usage limit (i.e. `--cache.size` multiplied by the high water multiplier `--cache.high-water-multiplier`). The default value for the high water multiplier is set to 56% in this version to keep compatibility with previous versions. * Expose the "high water multiplier" for the in-memory cache subsystem. The high water multiplier is used to calculate the effective memory usage limit for the in-memory cache subsystem. The cache's configured memory usage limit (`--cache.size`) is multiplied by the high water multiplier, and the resulting value is used as the effective memory limit. It defaults to 56% to ensure compatibility with previous versions, in which the threshold was effectively hard-coded to the same value. * Reduce memory usage for empty in-memory edge caches by ~40%. This is achieved by allocating each cache's statistics objects only lazily, when actually needed. * Added the following metrics for the in-memory edge cache: - `rocksdb_cache_edge_inserts_total`: total number of insertions into the in-memory edge cache. - `rocksdb_cache_edge_compressed_inserts_total`: total number of insertions into the in-memory edge cache that used compression. * Added the startup option `--cache.max-spare-memory-usage` to control memory usage for spare, unused hash tables in the in-memory caching subsystem. This option can be used to cap the memory used by spare tables. It can be set to a value of 0 to not use any memory except for active hash tables. * Fixed BTS-1556: Potential shutdown race, when the server is shutting down and still AgencyCommunication was going on it could use the scheduler which was already deleted, causing a crash right before a normal shutdown. * Fixed BTS-1556: Potential shutdown race, when the server is shutting down and still AgencyCommunication was going on it could use the scheduler which was already deleted, causing a crash right before a normal shutdown. This should not have any negative effect, as all state preserving operations are done by then, it just wasn't a clean exit. * Updated ArangoDB Starter to 0.17.1. * Fixed BTS-1541: Old legacy little endian key format for RocksDB database (created in ArangoDB 3.2 and 3.3) showed wrong behavior on newer versions. This fixes a persistent index corruption bug with the old format. * Fixed a loophole in COLLECT variable name validation: in COLLECT INTO expressions it was possible to refer to variables that the COLLECT just introduced. This was undefined behavior and not caught by the previous version of COLLECT's variable checking. * BTS-1598: fix race in agency. * ES-1727: Fix `UPDATE`, `REPLACE`, and `UPSERT ... UPDATE/REPLACE` failing with "conflict, _rev values do not match" for non-local edges in Smart- and EnterpriseGraphs, when `OPTIONS { ignoreRevs: false }` is supplied. * Upgraded Swagger-UI to v5.4.1. * Fixed BTS-1590: Fixed potentially undefined behavior in NetworkFeature, when it was referring to an options object that could have been destroyed already. * Allow compression content-negotation in metrics API, so that responses from the metrics API at `/_admin/metrics` can be sent compressed if the client supports it. v3.11.3 (2023-08-17) -------------------- * Fixed BTS-1553: Fixed a rare occuring issue during AQL queries where the inner amount of a limit in the LimitExecutor is set to a wrong value which lead to some data rows not being returned to the client. * BTS-1544: The _system database now properly reports its sharding value. * Fixed BTS-1554: wrong aggregation count when an "in" or "or" condition was executed through an index lookup. * BTS-1549: adjust permission handling in experimental dump to the same behavior as in non-experimental dump. * Fixed AQL WINDOW statement for OneShard databases: Whenever WINDOW is used in the row based variant like (e.g. WINDOW { preceding: 1, following: 1 }) it errored with: mandatory variable "inVariable" not found. This variable is now correctly treated as optional. * Fixed AQL WINDOW statement for OneShard databases: Whenever WINDOW is used on a OneShardDatabase, or on data from a collection that only has one shard, the preceding and following clauses were flipped. * Updated arangosync to v2.19.3. * Harden HTTP/2 internal callback functions against exceptions. These callback functions are called from C code which cannot handle exceptions in any way. Instead, we now turn any exception into the return code `HPE_INTERNAL` to signal that an error occurred. * Added better diagnostic messages in case documents<->primarx index corruption occurs. * Prevent potential buffer overflow in the crash handler. * Updated OpenSSL to 3.0.10 and OpenLDAP to 2.6.6. * Updated ArangoDB Starter to 0.17.0. * ES-1566: instead of potentially accessing a nullptr inside graph traversal setup, throw an exception. This will be handled properly and returned with a proper error message. * BTS-1511: AQL: Fixed access of integers in the ranges [-36028797018963968, -281474976710657] and [281474976710656, 36028797018963968], i.e. those whose representation require 7 bytes. These values were misinterpreted as different integers. Simple passing of values (i.e. writing to or reading from documents) was not affected: Only accesses of those values as numbers by AQL was. E.g. arithmetic (addition, multiplication, ...), certain AQL functions, comparing/sorting - the latter only if the numbers are compared directly, but not as part of a value. For example, `SORT x` lead to an unexpected order if x was such a number. `SORT [x]` however worked as expected. * Added experimental startup options for RocksDB .sst file partitioning: - `--rocksdb.partition-files-for-documents` - `--rocksdb.partition-files-for-primary-index` - `--rocksdb.partition-files-for-edge-index` - `--rocksdb.partition-files-for-persistent-index` Enabling any of these options will make RocksDB's compaction write the data for different collections/shards/indexes into different .sst files. Otherwise the document data from different collections/shards/indexes can be mixed and written into the same .sst files. Enabling these options usually has the benefit of making the RocksDB compaction more efficient when a lot of different collections/shards/indexes are written to in parallel. The disavantage of enabling this option is that there can be more .sst files than when the option is turned off, and the disk space used by these .sst files can be higher than if there are fewer .sst files (this is because there is some per-.sst file overhead). In particular on deployments with many collections/shards/indexes this can lead to a very high number of .sst files, with the potential of outgrowing the maximum number of file descriptors the ArangoDB process can open. Thus the option should only be enabled on deployments with a limited number of collections/shards/indexes. * Fixed potential deadlock() in cache::Manager::unprepareTask() in case a MigrateTask could not be scheduled successfully (in case of a full scheduler queue on DB servers). v3.11.2 (2023-07-21) -------------------- * Fix SEARCH-465: Truncate on ArangoSearch index could release a WAL if commit passed. * Updated ArangoDB Starter to 0.16.0. * Avoid recursive lock during agency startup. * Fixed issue with lock starvation when an AQL insert operation with multiple static documents was executed as part of a streaming transaction. * Added startup option `--database.max-databases` to limit the maximum number of databases that can exist in parallel on a deployment. This option can be used to limit resources used by database objects. If the option is used and there already exist as many databases as configured by this option, any attempt to create an additional database will fail with error 32 (`ERROR_RESOURCE_LIMIT`). Additional databases can then only be created if other databases are dropped first. The default value for this option is unlimited, so technically an arbitrary amount of databases can be created (although effectively the number of databases is limited by memory and processing resources). * FE-304, FE-305: use navigator.onLine for checking internet connection, correct path for running/slow queries. * Enforce that server always returns an empty body if the return code is 204. * Arangodump retries dump requests in more cases: read, write and connection errors. * Whenever there is a query request timeout in AQL (e.g., a server died) which causes the query to fail, the DBServers will now all directly kill their potentially ongoing parts of the query and not wait for garbage collection. * Use libunwind in jemalloc profiling to make it available with libmusl and static binaries. * BTS-1331: Reduced the amount of required network calls when using traversals combined with FILTER and/or PRUNE statements in a GeneralGraph in combination with a clustered environment. * Added transparent LZ4 compression for values in the in-memory edge cache if their size exceeds a configurable threshold. This is configurable as an opt-in functionality. LZ4 compression of edge index cache values allows to store more data in main memory than without compression, so the available memory can be used more efficiently. The compression is transparent and does not require any change to queries or applications. The compression can add CPU overhead for compressing values when storing them in the cache, and for decompressing values when fetching them from the cache. The new startup option `--cache.min-value-size-for-edge-compression` can be used to set a threshold value size for compression edge index cache payload values. The default value is `1GB`, which will effectively turn compression off. Setting the option to a lower value (e.g. `100`) will turn on the compression for any payloads whose size exceeds this value. The new startup option `--cache.acceleration-factor-for-edge-compression` can be used to fine-tune the compression. It controls the LZ4-internal "acceleration" factor used for the compression. The default value is `1`. Higher values typically mean less compression but faster speeds. The following new metrics can be used to determine the usefulness of compression: - `rocksdb_cache_edge_effective_entries_size`: will return the total number of bytes of all entries that were stored in the in-memory edge cache, after compression was attempted/applied. This metric will be populated regardless of whether compression is used or not. - `rocksdb_cache_edge_uncompressed_entries_size`: will return the total number of bytes of all entries that were ever stored in the in-memory edge cache, before compression was applied. This metric will be populated regardless of whether compression is used or not. - `rocksdb_cache_edge_compression_ratio`: will return the effective compression ratio for all edge cache entries ever stored in the cache. Note that these metrics will be increased upon every insertion into the edge cache, but not decreased when data gets evicted from the cache. * Optimize runtime performance of exclusive locks. * Added startup option `--replication.active-failover-leader-grace-period`. This startup option can be used to set the amount of time (in seconds) for which the current leader in an active failover setup will continue to assume its leadership even if it lost connection to the agency. In case the leader cannot contact the agency anymore, the agency will elect a new leader after the supervision grace period has elapsed. In order to avoid a split-brain situation with multiple servers assuming leadership, this option can be used to make a disconnected leader refuse any incoming write operations after the grace period controled by this option has elapsed. Ideally the startup option should be given a value greater than the value of the supervision grace period, in order to avoid a temporarily disconnected leader giving up leadership too early and unnecessarily. The default value is 120 seconds. Setting the option to a value of 0 will keep the existing behavior, in which a disconnected leader will not refuse incoming write operations. * Added new startup options `--cache.ideal-lower-fill-ratio` and `--cache.ideal-upper-fill-ratio` to control the minimum and maximum fill ratios for cache tables that trigger shrinking and growing of the table by the cache rebalancer. The default values are: - `0.04` (i.e. 4%) for the lower bound that triggers shrinking - `0.25` (i.e. 25%) for the upper bound that triggers growing These values were hard-coded in previous versions of ArangoDB. * Remove temporary `CREATING_{number}` directories from hot backup in case a hot backup runs into an error. * BTS-1490: Allow performing AQL updates locally without using DISTRIBUTE in case the update AQL is of the pattern FOR doc IN collection UPDATE IN collection Previously the optimization was only possible if the update AQL was of the pattern FOR doc IN collection UPDATE WITH IN collection Both and refer to data from the collection enumeration variable `doc` here. Also fix the optimization in case a shard key attribute is updated with a value from a different attribute, e.g. FOR doc IN collection UPDATE { _key: doc.abc, abc: doc._key } IN collection In this case the optimization was previously applied although it shouldn't. * BTS-1490: query can use a lot more memory when using COLLECT WITH COUNT. * Attempt to avoid busy looping when locking collections with a timeout. * Add a `description` field to OptimizerRule and dump the explanations via the `GET /_api/query/rules` endpoint. * FE-20: fix UI placeholder format for locale when creating analyzers. * FE-240: disable JSON editor when viewing inverted index. * FE-287: fix number validation for replicationFactor and writeConcern. * FE-285: Fix query download - use post request for query. * Fixed Github issue #19175. This fixes a problem in traversal query optimization that was introduced in 3.11 and that can lead to traversal queries being aborted with an error `AQL: cannot and-combine normalized condition`. * Fixed two possible deadlocks which could occur if all medium priority threads are busy. One is that in this case the AgencyCache could no longer receive updates from the agency and another that queries could no longer be finished. This fixes BTS-1475 and BTS-1486. v3.11.1 (2023-06-12) -------------------- * Updated arangosync to v2.18.1. * SEARCH-480: Speedup ArangoSearch recovery. * SEARCH-476: Fix bug in fst builder. * BTS-1325: AQL: Fixed a possible deadlock with multiple parallel traversals. * Updated OpenSSL to 3.0.9. * BTS-1435: fixed invalid AQL optimization and added a safeguard. * APM-766, SEARCH-479: Reduce memory overhead for ArangoSearch removes. * Improve precision for ArangoSearch GEO_IN_RANGE function. * Updated ArangoDB Starter to 0.15.8. * OASIS-25262: Fixed undefined behavior in IN lookup in unique indexes when the lookup array had to be rebuilt in memory. * Invalid keys are now reported as individual errors for batch insert operations and no longer abort the whole batch. * BTS-1255: Fix sporadic memory usage accounting underflows in in-memory cache subsystem. Also turn the algorithm for freeing memory from a cache's buckets from a non-deterministic one that did not guarantee progress into a bounded algorithm with guaranteed progress. * ECONNABORTED is treated as a ConnectionClosed error in fuerte. * Database drop operation no longer fails if we cannot remove the corresponding permissions from the _users collection. * Added startup option `--query.max-collections-per-query` to adjust the limit for the maximum number of collections/shards per query. The option defaults to `2048`, which is equivalent to the previous hardcoded value. * BTS-1261 For some named graphs in cluster, when creating a debugDump for a traversal query that would use the graph's name, the graph wouldn't be able to be recreated because the info gathered in the process of creating a debugDump was broken. It uses the result form an aql explain to get the graph info and use in the debugDump, but this wouldn't be available because, instead of having the name of the graph as key in the graph object of the explain, there would be an array with edge collection names. It was changed for the case when there's a named graph, but maintained when there's no access to the graph's name. * When trying to read multiple documents, the coordinator will now handle empty lists gracefully and return an empty result set instead of an error. * Added metric "rocksdb_total_sst_files" to count the number of sst files, aggregated over all levels of the LSM tree. * Increase too short timeouts for requests made from coordinators to DB-servers when retrieving the number of documents or the index selectivity estimates for SmartGraph edge collection parts. These parts were treated like system collections because of their naming convention, and the requests were run with a timeout of only 5s. * In batched query results, when executing requests for `/_api/cursor//`, removed the restriction that the user would only be able to fetch the next batch using the next batch id in if the query option `allowRetry` was set to true, but maintained the restriction that the user can only retrieve the latest batch if the query option `allowRetry` is set to true. * Add a startup parameter `--rclone.argument` which can be utilised to enable debugging with logfiles in hot backup RClone upload operacions: `--rclone.argument=--log-level=DEBUG` `--rclone.argument=-log-file=/tmp/rclone.log` * Fix issue #18982: query editor null/undefined check filters out bindParams with value 0. v3.11.0 (2023-05-23) -------------------- * Convert v3.11.0-rc.2 into v3.11.0. * SEARCH-477: stabilize ArangoSearch (stage of BTS-1416). v3.11.0-rc.2 (2023-05-17) ------------------------- * Internal bug-fixes and stabilization improvements. v3.11.0-rc.1 (2023-05-13) ------------------------- * FE-211: Allow admins to edit gravatar email * Fixed BTS-1398: GEO_DISTANCE() for ArangoSearch geo_s2 analyzer. * Improved error reporting for system calls on Windows and fixed a Windows-specific issue of the IOHeartbeatThread. * FE-256: Minor Graph Viewer improvements: - re-order right click menu - Make Right click menu and toolbar work in full screen - Remove editable attributes in edit and delete (nodes and edges) modals - Add "tree"-mode to editor in edit (nodes and edges) modals * FE-266: Use the method 'fromGeoJson()' instead of 'geometry.coordinates' for GeoJSON. * Fixed issue #18942: arangorestore ignores the HTTP status code. * FE-267: fix custom analyzer & features not showing up in inverted index view, and support for basic fields definition. * FE-265: Fix user permissions radio button shadow. * FE-268: remove 'switch to new graph' when no defined graph is present. * FE-258: allow specifying optimizeTopK during arangosearch view creation. * FE-257: allow additional properties in inverted index creation. * UI Fix - allow empty keys in document key validation. * BTS-1181: fix a data race on the collection list in TransactionState. This race could happen in the cluster when a collection is sharded by a different attribute than `_key` and a document lookup by _key is performed. This requires the lookup to be sent to each shard, since we cannot deduce the shard based on the `_key` value. If this lookup is done as part of a stream transaction, and if the collection has not been declared in the transaction's `read` list, then the collection is added lazily. Previously this would result in a race if multiple shards are located on the same server. For transactions with `allowDirtyReads` set to true read collections are always added lazily, which made this race more likely. * BTS-1340: Fixed telemetrics api making shell hang when logging into it and leaving it too quickly so the telemetrics API doesn't have time to send the data to the warehouse. The thread could be hanging on SSL_connect() until the connection timeout if the socket is blocking, so the socket was made non-blocking for the thread to leave SSL_connect() after the connection is interrupted when leaving the shell. Also created startup parameter `--client.failure-points`for arangosh which enables failure points whose names are provided in an array of strings, just like in `--server.failure-point` for arangod. * BTS-1350: Fixed imprecision in index info in telemetrics object for cluster. When the collection also had an arangosearch view, telemetrics showed imprecise index values because the view was being considered as an index of type arangosearch, and its object wouldn't contain expected fields that would be found in other indexes' objects. * Add metric `arangodb_file_descriptors_current` to expose the number of file descriptors currently opened by the arangod process. This metric is available on Linux only. As counting the number of open file descriptors can be expensive, this metric is only updated in a configurable interval. The new startup option `--server.descriptors-count-interval` can be used to specify the update interval (in milliseconds). It defaults to 60,000 (i.e. once per minute). The startup option can be set to a value of `0` to disable the counting of open file descriptors for performance reasons. If counting is turned off, the metric will report a value of `0`. * ES-1566: fix an issue when trying to restrict traversals to non-existing collections with `edgeCollections` traversal option. If a non-existing collection is specified in the `edgeCollections` or `vertexCollections` options of an AQL traversal, the query will now fail with a `collection or view not found` error. Also, if wrong collection types are used (e.g. a document collection or a view for `edgeCollections`), then an error is raised about an invalid collection type being used. This is a behavior change compared to previous versions, which ignored specifying non-existing vertex collections and had undefined behavior when specifying non-existing edge collections or using a vertex collection instead of an edge collection. * Added metric `rocksdb_cache_peak_allocated` to store the peak memory allocation value for in-memory caches. * Remove leftover in-memory cache tables after dropping collections that had their `cacheEnabled` flag set to `true`. Previously some memory could remain allocated in the in-memory cache even after such collections were dropped. * BTS-1315: Fixed spurious occurring failures during Foxx Application installation in case a Load Balancer is being used in front of the coordinators. * Updated arangosync to v2.17.0. * BTS-1343: Removed an error message of endpoint being invalid when attempting fetch telemetrics from the servers by having the invalid endpoint as a starting point. As telemetrics is transparent to the end user, the error message interferes with the user experience if keeps appearing. * FE-255: add validation for collection, document, graph, view and database names. * Add metric `arangodb_file_descriptors_limit` to expose the system limit for the number of open files for the arangod process. * Fix an issue that causes followers to be dropped due to premature transaction abortions as a result of query failures. When we have a query that results in a failure this will cause the leaders to abort the transaction on the followers. However, if the followers have transactions that are shared with leaders of other shards, and if one of those leaders has not yet seen the error, then it will happily continue to replicate to that follower. But if the follower has already aborted the transaction, then it will reject the replication request. Previously this would cause the follower to be dropped, but now this should be handled gracefully. v3.11.0-beta.1 (2023-05-07) --------------------------- * Updated arangosync to v2.17.0-preview-1. * FE-243: add support for geo_s2 analyzer. * Reduce memory usage for incremental sync replication. The incremental sync protocol that was used for collections created with 3.8 or higher had memory usage issues in case the follower already had some local data and its dataset was much larger than the leader's. For example, this can happen if a follower gets disconnected, then a lot of document removals happen on the leader and afterwards the follower tries to get back into sync. In this case, the follower buffered the ids of documents it had to remove locally in a vector, which could grow arbitrarily large. The removal of the documents contained in this vector would only happen at the end, potentially even without performing intermediate commits. The change in this PR is to trigger the document removal earlier, once the vector has reached some size threshold, and also to use intermediate commits during the removals. v3.11.0-alpha.1 (2023-05-03) ---------------------------- * FE-22: Don't display collection content by default when clicking on a collection in the Web UI. * Fixed a race in a test for aborting long running operations. * FE-251: bugfix - prohibit multiple expansions of the same node in the graph viewer. * FE-162: Fix display of geodesic lines. * FE-263: Improve forceAtlas layout in the Graph Viewer. * Fixed a race in the Maintenance, impacting leaders that have just been resigned. During a MoveShard job, the old leader could sabotage the whole operation by removing the newly added follower from Current. The solution is to update Current using local collection information, instead of the potentially outdated velocypack slice. * Fix unstable test setting the desired number of dbservers. * FE-253: bugfix - validate JSON, show errors & disable save when error. * FE-254: bugfix - filter out empty storedValues for persistent index. * FE-252: bugfix - canvas image screenshot breaks graph colors. * Fix incompatibility between 3.9 and 3.10 w.r.t. to serialization of AQL array filters (i.e. `[* FILTER ...]`). The array filters were serialized in a different way in 3.9 than they are serialized in 3.10. 3.10 also expected the new serialization format when unserializing a plan. The fix now enables support for both formats. * Fixed issue #18769: Input validation allows invalid UTF-8 code points. This change enforces the validation of UTF-8 surrogate pairs in incoming JSON data. Previously, the following loopholes existed when validating UTF-8 surrogate pair data: - a high surrogate, followed by something other than a low surrogate (or the end of the string) - a low surrogate, not preceeded by a high surrogate These loopholes are now closed, which means that any JSON inputs with invalid surrogate pair data will be rejected by the server. Note that the extended validation for surrogates can be turned off along with other UTF-8 string validation by setting the server startup option `--server.validate-utf8-strings` to `false`. This is not recommended though, but should only be used in situations when a database is known to contain invalid data and must continue supporting it. * Updated rclone to v1.62.2 custom build with go1.20.3. * Changed return code of APIs that create databases from previously 1229 (`ERROR_ARANGO_DATABASE_NAME_INVALID`) to 1208 (`ERROR_ARANGO_ILLEGAL_NAME`) in case an invalid database name is used. This is a downwards-incompatible change, but unifies the behavior for database creation with the behavior of collection and view creation, which also return error 1208 in case the specified name is invalid. * FE-236: bugfix - remove unused files, use new tooltip in views UI. * FE-238: Added auto-login support in core web UI - disabled logout when auto-login is enabled, set sessionStorage "jwtUser" value when login is skipped. * FE-233: bugfix - fix query spotlight search not working. * FE-349: bugfix - filter out empty primarySort field in UI. * FE-247: bugfix - missing storedValues field in persistent index form. * FE-242, FE-244: bugfix - add support for cache fields, fix inverted index name undefined. * FE-241: bugfix - filter predefined queries based on search term. * FE-216: bugfix - make view patches async in the UI. * FE-212: bugfix: links not getting removed when copying from another view in UI. * FE-222: Fix - Allow additional properties in arangosearch, allow no fields in inverted index when 'includeAllFIelds' is true. * APM-183: Support UTF-8 on UI (collection/view/index names). * FE-199: Remove URL handling of fields on view screen. * Changed the behavior of the following JavaScript functions in arangosh and arangod (e.g. when used from a Foxx service): - `db..dropIndex(id)`: this function now throws if no index exists with the specified id. Previously the function only returned the value `false`. - `db._dropIndex(id)`: this function now throws if no index exists with the specified id. Previously the function only returned the value `false`. These changes are not downwards-compatible, but they can be easily worked around by wrapping dropIndex calls into a try ... catch. The HTTP API for dropping indexes is not affected by these changes, as it previously returned HTTP 404 already when the specified index could not be found. * Added `--dump-views` option to arangodump, to control whether arangosearch view definitions should be stored as part of the dump. The option defaults to `true`. * APM-183: optionally allow special characters and Unicode characters in collection names, view names and index names. This feature allows toggling the naming convention for collection names, view names and index names from the previous strict mode, which only allowed selected ASCII characters, to an extended, more relaxed mode. The extended mode allows additional ASCII characters as well as non-ASCII UTF-8 characters in database names, collection names, index names and view names. The extended mode can be enabled by setting the new startup option - `--database.extended-names` to true. It is turned off by default and requires an explicit opt-in, simply because some drivers and client applications may not be ready for it yet. The arangod server, the ArangoDB web interface and the following bundled client tools are prepared and ready for using the extended names: - arangobench - arangodump - arangoexport - arangoimport - arangorestore - arangosh More tools and the drivers shipped by ArangoDB may be added to the list in the future. Please note that the extended names should not be turned on during upgrades from previous versions, but only once the upgrade has been completed successfully. In addition, the extended names should not be used in environments that require extracting data into a previous version of ArangoDB, or when database dumps may be restored into a previous version of ArangoDB. This is because older versions will not be able to handle the extended names. Finally, it should not be turned on in environments in which drivers are in use that haven't been prepared to work with the extended naming convention. Warning: turning on the `--database.extended-names` option for a deployment requires it to stay enabled permanently, i.e. it can be changed from `false` to `true` but not back. When enabling it, it is also required to do this consistently on all coordinators and DB servers. The extended names for databases, collections, views and indexes will be enabled by default in one of the future releases of ArangoDB, once enough drivers and other client tools have had the chance to adapt. * FE-200: Adds smart & enterprise graph support in the UI. * Forward the `ttl` cursor option for AQL queries in the JavaScript API from the `db._query()` and the `db._createStatement()` methods to the server. * APM-407: add an optimization for inserting multiple documents at the same time via an AQL INSERT query. There is an optimizer rule `optimize-cluster-multiple-document-operations`, which fires in case an AQL query has one of the patterns - `FOR doc IN @docs INSERT doc INTO ...` (where `@docs` is a bind parameter with an array of documents to be inserted), - `FOR doc IN [...] INSERT doc INTO ...` (where the FOR loop iterates over an array of input documents known at query compile time), - `LET docs = [...] FOR doc IN docs INSERT doc INTO ...` (where the documents set up by the LET are some static documents known at query compile time) If a query has such pattern, and all the following restrictions are met, then the optimization is triggered: - there are no following RETURN nodes (including any RETURN OLD, RETURN NEW) - the FOR loop is not contained in another outer FOR loop or subquery - there are no other nodes (e.g. LET, FILTER) between the FOR and the INSERT - the INSERT is not used on a SmartGraph edge collection - the FOR loop is iterating over a constant, deterministic expression The optimization will then add a `MultipleRemoteExecutionNode` to the query execution plan, which will care about inserting all documents into the collection in one go. Further optimizer rules are skipped if the optimization is triggered. Future versions of ArangoDB may lift some of the restrictions for the query pattern, so that the optimization may be triggered in more cases in the future. * FE-200: Add canvas interactions to Graph Viewer. * FE-218: Updated WebUI dependencies. * Make REST API `/_admin/shutdown` sleep for only half a second until it initiates the server shutdown. Previously it slept for 2 seconds, but half a second should already be enough to send the server's response out. * MDS-1001: Performance improvement in AQL. If you are using a traversal like `FOR v, e, p IN <....>` and later in the query access the last vertex on the path e.g.: `FILTER p.vertices[-1].name == "ArangoDB"` it will now be transformed to `FILTER v.name == "ArangoDB"` which is an equivalent statement. The latter however is cheaper to compute, as we do not need to create an in-memory representation of the path. Furthermore we can apply additional optimizations on `v` which are not possible on `p`. The same optimization holds true for `p.edges[-1]` which is equivalent to `e`. The optimization rule for this is called "optimize-traversal-last-element-access". * FE-142: Updates indices view list & index addition to React. * A Pregel execution now stores its state during and after execution into a system collection. To read or delete entries the new API `/_api/control_pregel/history[/]` has been added. Additionally, the Pregel JavaScript module has been extended to support access as well. Read history `.history()`. Remove history `.removeHistory()`. * Marked all memory-mapping options for Pregel as obsolete. The memory mapping code was removed as it did not provide any advantages over spilling into system-provided swap space. * FE-139 adds new search view type (search-alias). * Ran automated migrations on all .scss files to remove deprecated division operator usage. * SEARCH-279: Fix consistency during update/replace operations for arangosearch links and inverted indexes. * APM-294: Added telemetrics API that gathers anonymous feature usage statistics from a deployment. The API is accessible via the endpoint `/_admin/telemetrics`. The API is enabled by default in release builds, but disabled by default in maintainer mode. It can be explicitly turned on/off with the server startup parameter `--server.telemetrics-api`. The required access privileges to access the telemetrics API can be configured via the server startup option `--server.support-info-api`. The telemetrics API is used by the arangosh: every time the arangosh is started, it will send a request to the connected server to gather telemetrics from the `/_admin/telemetrics` endpoint. The telemetrics data are then sent to an aggregation service that is run by ArangoDB. * APM-283: Use parallel gather in almost all queries. The only case where we cannot use parallel gather is when using traversals, although there are some exceptions for disjoint SmartGraphs where the traversal can run completely on the local DB-server. All other queries should now be able to parallelize the gather node. This can not only speed up queries quite significantly, but also overcomes issues with the previous serial processing within gather nodes, which could lead to high memory usage on coordinators caused by buffering of documents other shards, and timeouts on some DB-Servers because query parts were idle for too long. * Changed path were test scripts locate configuration files from `etc/relative` to `etc/testing`. These paths contain `arangosh.conf`, which we were reading from `etc/relative` in test environment. * Made the return code configurable that is delivered if a write fails because the write concern is not fulfilled (not enough in-sync replicas available). Previously (and now by default), a code of HTTP 403 is returned and the request returns immediately. If the command line option --cluster.failed-write-concern-status-code=503 is set, then HTTP 503 is returned. Note that no cluster-internal retry is happening, such that a client is informed right away about the problem. Retry loops have to be organized in the client program. * Added support for sending gzip-compressed responses from the server. Previously only deflated responses were supported. * FE-135: Add new Graph Viewer with vis.js and change the UI. * FE-19: Updated ArangoDB logo in web interface. * Make the hashed variant of AQL COLLECT support INTO clauses too. Previously only the sorted variant of AQL COLLECT supported INTO clauses. * Upgraded OpenSSL to 3.0.8. * FE-174: Change ViewsUI layout to single-page instead of tabs. * Add peak memory usage to the query object details for queries in the slow query history and in the list of currently running queries. The peak memory usage is also returned via REST APIs as `peakMemoryUsage`. * Provide options for configuring and enabling RocksDB's blob storage (BlobDB) for large documents in the documents column family. This is currently an experimental feature. The following experimental options are available: - `--rocksdb.enable-blob-files`: Enable the usage of blob files for the documents column family. This option defaults to `false`. All following options are only relevant if this option is set to `true`. - `--rocksdb.min-blob-size`: Size threshold for storing large documents in blob files (in bytes, 0 = store all documents in blob files). - `--rocksdb.blob-file-size`: Size limit for blob files in the documents column family (in bytes). - `--rocksdb.blob-compression-type`: Compression algorithm to use for blob data in the documents column family. - `--rocksdb.enable-blob-garbage-collection`: Enable blob garbage collection during compaction in the documents column family. - `--rocksdb.blob-garbage-collection-age-cutoff`: Age cutoff for garbage collecting blob files in the documents column family (percentage value from 0 to 1 determines how many blob files are garbage collected during compaction). - `--rocksdb.blob-garbage-collection-force-threshold`: Garbage ratio threshold for scheduling targeted compactions for the oldest blob files in the documents column family. * FE-132: Added query sorting (in web UI) by modified date, option to sort order. * Partial fix for PRESUPP-539: account for memory used during AQL condition transformation to disjunctive normal form (DNF). This transformation can use a lot of memory for complex filter conditions, which was previously not accounted for. Now, if the transformation uses a lot of memory and hits the configured query memory limit, the query will rather be aborted with a proper error message than overuse memory. For very complex conditions that would use massive amounts of memory when transformed into DNF, the DNF conversion is also aborted at some threshold complexity value. If the threshold is hit, the query continues with a simplified representation of the condition, which will not be usable in index lookups. However, this should still be better than overusing memory or taking a very long time to compute the DNF version. The complexity threshold value can be configured per query by setting the new `maxDNFConditionMembers` query option. There is also a new startup option `--query.max-dnf-condition-members` for coordinators and single servers to adjust the threshold value globally. * The internal Graph code is completely converted to the new graph engine. Last algorithms added to that lists are: ShortestPath, WeightedShortestPath, KShortestPaths and WeightedKShortestPaths. * FE-131: Added search input for query page. * FE-133: Alphabetical sorting for collections on user permissions page. * Removed CMake variable `ARANGODB_BITS`, which was only used in one place. * Fixed the issue that the collection view search did not support selecting everything using Ctrl + A. * APM-592: In batched query results, when executing requests for `/_api/cursor`, there might be a connection error and the user might not be able to retrieve the latest batch from the cursor. For that, a query option flag `allowRetry` was added. When set to `true`, if the latest batch response object wasn't successfully received, the user can send a retry request to receive it with a POST request to `/_api/cursor//`. Only the latest batch is cached, meaning former batches cannot be retrieved again later. * Use more compact and efficient representation for arrays and objects during AQL AST serialization and deserialization. This can help to reduce the size of messages exchanged between coordinator and database servers during query setup, and also reduce the time needed for parsing these messages. This especially helps when there are large bind parameter values that are arrays or objects. The more efficient format is used also inside an AQL query's "explain" and "profile" methods, and thus any callers that process the return values of explain and profile operations may now receive the new format. All callers inside the ArangoDB code have been adjusted, but any external callers that process the JSON response values of AQL query explain or profile operations may need to be adjusted to handle the new format. * Added new stage "instantiating executors" to the query profiling output. The time spent in "instantiating executors" is the time needed to create the query executors from the final query execution time. In cluster mode, this stage also includes the time needed for physically distributing the query snippets to the participating database servers. Previously, the time spent for instantiating executors and the physical distribution was contained in the "optimizing plan" stage, which was misleading. * Removed constant values for query variables from query plan serialization in cases they were not needed. Previously, constant values of query variables were always serialized for all occurrences of a variable in a query plan. If the constant values were large, this contributed to higher serialization and thus query setup times. Now the constant values are only serialized for relevant parts of query execution plans. * BTS-199: remove check for environment variable `GLIBCXX_FORCE_NEW` from server start, and remove setting this variable from startup scripts. The reason is that the environment variable only controls the behavior of programs linked against glibc, but our release builds are linked to libmusl. * Acquire a snapshot of the (list of) indexes when starting document insert, update/replace and remove operations, and use that snapshot throughout the operation. Previously, the list of indexes was acquired multiple times during a write operation, and it was (at least in theory) possible that the list of indexes changed between the individual acquisitions. The PR also contains an optimization to not fetch the full document from the storage engine for remove and replace operations in case the full document is not needed to process the operation. This is the case when the collection does not contain any secondary indexes and `returnOld` is not used. * Added experimental startup option `--rocksdb.block-cache-jemalloc-allocator`. This option defaults to `false`. When set to `true`, a jemalloc-based memory allocator will be used to allocate memory for the RocksDB block cache. This allocator will also mark the memory of the block cache to be excluded from coredumps, potentially reducing coredump size a lot. * Remove async mode from pregel. * Print the pid of the process which sent a SIGABRT or other fatal signal that shuts down ArangoDB ungracefully. * Avoid write-write conflicts for single document operations performed via the document REST API (i.e., no AQL, but also no streaming transactions). This is achieved by locking the key of each document before performing the actual modification. This lock acquisition effectively serializes all operations on the same document. To avoid starvation, the lock acquisition is limited to one second. This lock timeout value is currently hardcoded but will be made configurable in the future. If the lock cannot be acquired within this time, the operation fails with a write-write conflict error as before. Performing changes to a unique index entry also requires us to lock that index entry to ensure uniqueness. This lock acquisition is subject to the same lock timeout as locking the document key. We are planning to generalize this for multi-document operations as well as AQL and streaming transactions in the future. In case we cannot acquire the lock on the key of the document we want to insert/modify, the error message will be `Timeout waiting to lock key - in index primary of type primary over '_key'; conflicting key: ` where `` corresponds to the key of the document we tried to modify. In addition, the error object will contain `_key`, `_id` and `_rev` fields. The `_key` and `_id` correspond to the document we tried to insert/modify, and `_rev` will correspond to the current revision of the document from the DB if available, and otherwise empty. In case we cannot acquire the lock on a unique index entry, the error message will be `Timeout waiting to lock key - in index of type persistent over ''; document key: ; indexed values: []` where `` is the name of the index in which we tried to lock the entry, `` is the list of fields included in that index, `` corresponds to the key of the document we tried to insert/modify, and `` corresponds to the indexed values from our document. In addition, the error object will contain `_key`, `_id` and `_rev` fields. The `_key` and `_id` correspond to the document we tried to insert/modify, and `_rev` will correspond to the current revision of the document from the DB if available, and otherwise empty. This addresses GitHub issue #9702 and APM-522. * Fixed BTS-418: Suboptimal index range calculation with redundant conditions. * Added new per-operation option `refillIndexCaches` to write operations, namely: - AQL INSERT/UPDATE/REPLACE/REMOVE modification operations - single-document insert, update, replace and remove operations - multi-document insert, update, replace and remove operations If the option is set to `true` every currently running transaction will keep track of which in-memory index cache entries were invalidated by the transaction, and will try to (re-)fill them later. Currently edge indexes and velocypack-based indexes (persistent, hash, skiplist index) are supported. For velocypack-based indexes, the refilling will only happen if the index was set up with an in-memory cache (i.e. the `cacheEnabled` flag was set during index creation). Example usages: - `db..insert({ _from: ..., _to: ..., ... }, { refillIndexCaches: true });` - `db..update(key, { _from: ..., _to: ..., ... }, { refillIndexCaches: true });` - `db..replace(key, { _from: ..., _to: ..., ... }, { refillIndexCaches: true });` - `db..remove(key, { refillIndexCaches: true });` - `INSERT { ... } INTO OPTIONS { refillIndexCaches: true }` - `UPDATE { ... } WITH { ... } INTO OPTIONS { refillIndexCaches: true }` - `REPLACE { ... } WITH { ... } INTO OPTIONS { refillIndexCaches: true }` - `REMOVE { ... } IN OPTIONS { refillIndexCaches: true }` The refilling of the in-memory caches for indexes is performed by a background thread, so that the foreground write operation shouldn't be slowed down a lot. The background thread may however cause additional I/O for looking up the data in RocksDB and for repopulating the caches. The background refilling is done in a best-effort way and is not guaranteed to always succeed, e.g. if there is no memory available for the cache subsystem, or when an in-memory cache table is currently in a migration phase (grow/shrink operation). There is a new startup option `--rocksdb.auto-refill-index-caches-on-modify` for DB-Servers and single servers, which currently defaults to `false`. If it is set to `true`, the cache refilling will be turned on automatically for all insert/update/replace/remove operations, so that it doesn't need to be specified on the per-operation/per-query level. The new startup option `--rocksdb.auto-refill-index-caches-queue-capacity` can be used to limit the number of index cache entries that the background thread will queue. This is a safeguard to keep the memory usage at bay in case the background thread is slower than concurrent threads that perform ingestions. There are also new startup options to control whether or not the in-memory caches should automatically be seeded upon server restart. The option `--rocksdb.auto-fill-index-caches-on-startup` for DB-Servers and single servers enables this functionality. It currently defaults to `false`. If it is set to `true`, the in-memory caches of all eligible indexes will be automatically pre-seeded after the server startup. Note that this may cause additional CPU and I/O load. The option `--rocksdb.max-concurrent-index-fill-tasks` is available to limit the impact of the automatic index filling at startup. It controls how many full index filling operations can execute concurrently. The lower this number is, the lower the impact of cache filling, but the longer it will take. The default value for this option depends on the number of available cores, and is at least `1`. A value of `0` cannot be used. This option is only relevant if `--rocksdb.auto-fill-index-caches-on-startup` is set to `true`. The PR also adds the following metrics: - `rocksdb_cache_auto_refill_loaded_total`: Total number of queued items for in-memory index caches refilling. It will always report a value of zero on coordinators. - `rocksdb_cache_auto_refill_dropped_total`: Total number of dropped items for in-memory index caches refilling (because number of queued items would exceed the value of `--rocksdb.auto-refill-index-caches-queue-capacity`). It will always report a value of zero on coordinators. - `rocksdb_cache_full_index_refills_total`: Total number of in-memory index caches refill operations for entire indexes. The counter gets increased for every index automatically loaded (because startup option `--rocksdb.auto-fill-index-caches-on-startup` is set to `true`) or when full indexes are loaded into memory manually. In cluster deployments the counter will be increased once per eligible index per shard. It will always report a value of zero on coordinators. * BTS-128: Fixed http request not working when content-type is velocypack. * Deleted customizable Pregel (AIR) and Greenspun library. * Add support for terabyte units (t, tb, T, TB, tib, TiB, TIB) in startup options. * Make the deprecated `--server.disable-authentication-unix-sockets` and `--server.disable-authentication` startup options obsolete. They were deprecated in v3.0 and mapped to `--server.authentication` and `--server.authentication-unix-sockets`, which made them do the opposite of what their names suggest. * Log startup warnings for any experimental, deprecated, obsolete or renamed options at startup of arangod or any of the client tools. * Added option to exclude system collection from rebalance shards plan. * Improve performance and memory usage of IN list lookups for hash, skiplist and persistent indexes. * Improve memory usage tracking for IN list lookups and other RocksDB-based lookups. * Remove inactive query plan cache code (was only a stub and never enabled before). * Fixed BTS-441: Honor read only mode with disabled authentication * Obsolete startup option `--database.force-sync-properties`. This option was useful with the MMFiles storage engine, but didn't have any useful effect when used with the RocksDB engine. * BTS-483: Added restriction for usage of query cache for streaming and JS transactions when they are not read-only. * Remove map and map.gz files from repository and add them to gitignore. These files are only used for debugging and therefore should not be included in any release. This also reduces the size of release packages. * Improved help texts for the collection type and satellite collection options in the web UI. * Deprecate the startup option `--agency.pool-size`. This option was never properly supported for any values other than the value of `--agency.size`. Now any value set for `--agency.pool-size` other than the value set for `--agency.size` will now produce a fatal error on startup. * BTS-1082: Updating properties of a satellite collection breaks replicationFactor. * BTS-209: Fixed requests to `_admin/execute` treating every payload as plain text when they're in JSON or velocypack format, but will only treat the payload as velocypack if specified in the header's `content-type`. * Fixed issue #17394: Unnecessary document-lookup instead of Index-Only query. This change improves projection handling so that more projections can be served from indexes. * Change default output format of arangoexport from `json` to `jsonl`. * BTS-941: The HTTP API now delivers the correct list of the collection's shards in case a collection from an EnterpriseGraph, SmartGraph, Disjoint EnterpriseGraph, Disjoint SmartGraph or SatelliteGraph is being used. * BTS-465: Added tests for RandomGenerator and warning that other options for creating random values that are not Mersenne are deprecated. * BTS-977: Added an error message for when an unauthorized user makes an HTTP GET request to current database from a database name that exists which the user can't access and from a database name that doesn't exist, so both requests have the same error message (`_db//_api/database/current`). * Added new AQL function SHA256(value). * Added index cleanup in Supervision. If an index was not created successfully and the coordinator which initiated the creation was rebooted or is dead, then the agency Supervision will drop the index again. If it was created successfully, the agency Supervision will finalize it. * BTS-742: Added restriction for, when in smart graph, not accepting satellites in invalid format when storing a graph (like `{satellites: null}`). * BTS-477: added integration tests for covering log parameters. * Moved the handling of escaping control and unicode chars in the log to the Logger instead of LogAppenderFile. * Added authenticate header to the HTTP response when status code is 401 for HTTP/2. * Best quality spam pushed down to DEBUG. * Fixed log with json format not respecting the value of parameter `--log.shorten-filenames`. * Added "intermediateCommits" statistics return value for AQL queries, to relay the number of intermediate commits back that a write query performed. * Added message on the UI view of Logs when the user has restricted access, either because cannot access `_system`, or because is currently in another database. * Fix for the Pregel's HITS algorithm using a fixed value instead of the passed "threshold" parameter. The same applied to the new HITSKleinberg. * Now the Pregel API returns `{... algorithm: "pagerank", ...}` instead of `{... algorithm: "PageRank", ...}` when the Page Rank algorithm is run (in accordance to the documentation). * Added integration tests for `--log.escape-control-chars` and `--log.escape-unicode-chars`. * A new Pregel algorithm: the version of Hypertext-Induced Topic Search (HITS) as described in the original paper. * BTS-428: Added function DATE_ISOWEEKYEAR that retrieves the number of the week counting from when the year started in ISO calendar and also the year it's in. * Added handling of requests with Transfer-Encoding chunked, which is not implemented, so returns code HTTP code 501. * Disallowed index creation that covers fields in which the field's name starts or ends with `:` for single server or cluster when the instance is a coordinator or single server. This validation only happens for index creation, so already existing indexes that might use such field names will remain as they are. v3.10.6 (2023-04-27) -------------------- * Fixed BTS-1292: Added automatic cleanup of dangling ArangoSearch links. * Automatically repair revision trees after several failed shard synchronization attempts. This can help to get permanently out-of-sync shards back into sync. The functionality can be turned off by setting the startup option `--replication.auto-repair-revision-trees` to `false` on DB-Servers. * SEARCH-466 Fix leaking into individual link definition inherited properties from view. * Fix race condition in invalidation of token cache on coordinators. * Adjusted timeouts for cluster internal commit and abort requests to withstand network delays better. This fixes some problems when the networking infrastructure delays requests. * Added sent time accounting and some metrics to fuerte and the NetworkFeature. This can detect delays in the network infrastructure. * Added startup option `--server.ensure-whitespace-metrics-format`, which controls whether additional whitespace is used in the metrics output format. If set to `true`, then whitespace is emitted between the exported metric value and the preceeding token (metric name or labels). Using whitespace may be required to make the metrics output compatible with some processing tools, although Prometheus itself doesn't need it. The option defaults to `true`, which adds additional whitespace by default. * SEARCH-461: Added option "--arangosearch.columns-cache-only-leader". Used only on EE DBServers. Default is false. If set to true only leader shards have ArangoSearch caches enabled - this will reduce RAM usage. In case of failover happens - in background caches are populated for the new leader. Some queries that run at during a failover may still run without caches. * BTS-1148: Fix a race when aborting/finishing a currently active query on a DB-Server. This race could cause the query to remain in the server's query registry longer than intended, potentially holding some locks. Such queries were garbage collected eventually, but this could take a while, depending on the specified TTL (10min per default). This has now been fixed so that aborted/finished queries are cleaned up in a timely manner. * MDS-1098: In 3.10 we have introduced an optimization on Traversals to pull post-filter conditions into the traversal-statements, like the following: FOR v,e,p IN 10 OUTBOUND @start GRAPH "myGraph" FILTER v.isRelevant == true RETURN p If the comparison side contains a variable and the same variable is used as the start vertex e.g. like this: FOR candidate IN ["vertices/1", "vertices/2"] FOR v,e,p IN 1 OUTBOUND candidate GRAPH "myGraph" FILTER e.MostLikedNeighbor == candidate RETURN v There is a chance that we prematurely discarded this variable (candidate in the example) if it is not used later. This has lead to incorrect results. * Fixed statistics values for writes executed and writes ignored when a query is executed using the rule `optimize-cluster-single-document-operations`. It was always increasing the amount of writes executed, even if the operation wasn't successful, and also never increasing the amount of writes ignored when needed. * Updated arangosync to v2.16.1. * Fix potential thread starvation in in-memory edge cache. * SEARCH-300: Fixed a rare case when arangosearch data folders might be left on disk after database is dropped. * Fixed SEARCH-459 Fixed reporting ArangoSearch inverted index properties from ensureIndex request. * Changed path where test scripts locate configuration files from `etc/relative` to `etc/testing`. These paths contain `arangosh.conf`, which we were reading from `etc/relative` in test environment. * Fix issues with deferred database creation: When a database has made it into the Plan part of the agency with some settings, e.g. `replicationFactor` that would violate the current settings for databases (e.g. `--cluster.min-replication-factor` and `--cluster.max-replication-factor`), and then a new DB server is added, it will try to create the database locally with the settings from the Plan. As these settings however violate the min/max replication factor values, the database is not created on the new DB server and an error is written into Current instead. This can cause follow-up errors and the PlanSyncer complaining about missing databases for analyzers etc. * Fixed ES-1508: (EE only) when deleting edges in a SmartGraph via DELETE /_api/document/{collection} using _key or _id values as document selectors, the INBOUND and OUTBOUND entries of the SmartEdges could diverge. Using a document like {_key: "xxxx"} as a selector was always correct. Now _key and _id variants are supported as intended. * Fixed single-to-single replication that used HTTP authentication to authenticate requests on the leader. This could be broken if the collections on the leader were created with 3.8 or later, and thus used the Merkle tree protocol to exchange different document revisions. When using HTTP authentication, the prefetching code for document revisions did not pass on the authentication credentials, so the leader could reject requests with HTTP 401 or HTTP 403, and replication failed. Replication in the cluster and replication using JWT authentication were not affected. * Added the following metrics for WAL file tracking: - `rocksdb_live_wal_files_size`: cumulated size of alive WAL files (not archived) - `rocksdb_archived_wal_files_size`: cumulated size of archive WAL files * By default, start pruning of archived WAL files 60 seconds after server start. Previously, pruning of WAL files started 180 seconds after server startup. * Set default threshold value for automatic column flushing to 20 live WAL files (previously: 10 files), and retry flushing every 30 minutes (previous interval: every 60 minutes). * BTS-1272: Fixed metric `arangodb_connection_pool_connections_current`. In some cases where multiple connections to a server are canceled the metric could miss-count, as for now it only counted individually closed connections. The wrong counted situations are: other server crashes, restore of a HotBackup, rotation of JWT secret. * Added support to log response bodies as well as HTTP headers (incoming and outgoing), when the requests log topic is set to TRACE. v3.10.5 (2023-03-16) -------------------- * Stabilized resilience tests. The assumption that an AQL query can run without error directly after a leader has been stopped, is wrong. * Auto-flush RocksDB WAL files and in-memory column family data if the number of live WAL files exceeds a certain threshold. This is to make sure that WAL files are moved to the archive when there are a lot of live WAL files present (e.g. after a restart; in this case RocksDB does not count any previously existing WAL files when calculating the size of WAL files and comparing it `max_total_wal_size`. The feature can be configured via the following startup options: - `--rocksdb.auto-flush-min-live-wal-files`: minimum number of live WAL files that triggers an auto-flush. Defaults to `10`. - `--rocksdb.auto-flush-check-interval`: interval (in seconds) in which auto-flushes are executed. Defaults to `3600`. Note that an auto-flush is only executed if the number of live WAL files exceeds the configured threshold and the last auto-flush is longer ago than the configured auto-flush check interval. That way too frequent auto-flushes can be avoided. * Fix potential memory under-accounting on cache shutdown for in-memory caches for edge indexes. * Added the following metrics for WAL file tracking: - `rocksdb_live_wal_files`: number of alive WAL files (not archived) - `rocksdb_wal_released_tick_flush`: lower bound sequence number from which onwards WAL files will be kept (i.e. not deleted from the archive) because of external flushing needs. Candidates for these are arangosearch links and background index creation. - `rocksdb_wal_released_tick_replication`: lower bound sequence number from which onwards WAL files will be kept because they may be needed by the replication. - `arangodb_flush_subscriptions`: number of currently active flush subscriptions. * Updated internal JavaScript dependencies: - @xmldom/xmldom: 0.8.0 -> 0.8.6 - accepts: 1.3.7 -> 1.3.8 - ajv: 8.10.0 -> 8.12.0 - ansi_up: 5.0.1 -> 5.1.0 - content-disposition: 0.5.3 -> 0.5.4 - content-type: 1.0.4 -> 1.0.5 - error-stack-parser: 2.0.6 -> 2.1.4 - mime-types: 2.1.31 -> 2.1.35 - semver: 7.3.5 -> 7.3.8 * Updated transitive JS dependency hoek to @hapi/hoek@8.5.1 to resolve CVE-2020-36604 in joi. * Updated JS dependency minimatch to 3.1.2 to resolve CVE-2022-3517. * Updated JS dependency qs to 6.11.0 to resolve CVE-2022-24999. * Updated arangosync to v2.15.0. * Allow usage of projections and covering indexes in more cases. Previously, projections were not used if there were complex filter conditions on the index attribute(s) that contained the `[*]` expansion operator with inline FILTERs or RETURNs, e.g. `FILTER doc.addrs[* FILTER CURRENT.country == 'US'].zip`. * PRESUPP-546: make AQL optimizer rule `simplify-conditions` correctly report that it was triggered. Previously that rule never reported that it was triggered although even though it actually was. * Added startup option `--rocksdb.auto-refill-index-caches-on-followers` to control whether automatic refilling of in-memory caches should happen on followers or just leaders. The default value is `true`, i.e. refilling happens on followers too. * Added new geo_s2 ArangoSearch analyzer (Enterprise Only). * GORDO-1554: Fixes invalid document insertion with invalid user-specified keys (e.g. numeric values) into EnterpriseGraph related vertices. * Added metric `arangodb_replication_clients` showing the number of currently active/connected replication clients for a server. * BTS-1249: Add startup option `--foxx.enable`. This startup option determines whether access to user-defined Foxx services is possible for the instance. It defaults to `true`. If the option is set to `false`, access to Foxx services is forbidden and will be responded with an HTTP 403 Forbidden error. Access to ArangoDB's built-in web interface, which is also a Foxx service, is still possible even with the option set to `false`. When setting the option to `false`, access to the management APIs for Foxx services will also be disabled. This is the same as manually setting the option `--foxx.api false`. * Fixed a bug in the API used by `arangorestore`: On restore, a new _rev value is generated for each imported document to avoid clashes with previously present data. This must be created on the shard leader rather than the coordinator. The bug happened, when two coordinators were creating the same _rev value for two different documents concurrently. * ES-1428: make the maximum number of V8 contexts depend on the maximum number of server threads, if `--javascript.v8-contexts` is not set explicitly. Previously the maximum number of V8 contexts was hard-coded to 16 when the option `--javascript.v8-contexts` option was not set explicitly. Now the maximum number defaults to 7/8 of the value of the startup option `--server.maximal-threads`, regardless of if it is explicitly configured or the default value is used. Only 7/8 are used to leave some headroom for other important maintenance tasks. A server with default configuration should now not block waiting for V8 contexts to become available, but it may use more memory for the additional V8 contexts if there are many concurrent requests that invoke JavaScript actions (e.g. requests using the web UI or Foxx). * Improve memory usage of in-memory edge index cache if most of the edges in an index refer to a single or mostly the same collection. Previously the full edge ids, consisting of the the referred-to collection name and the referred-to key of the edge were stored in full. Now, the first edge inserted into an edge index' in-memory cache will determine the collection name for which all corresponding edges can be prefix-compressed. For example, when inserting an edge pointing to `the-collection/abc` into the empty cache, the collection name `the-collection` will be noted for that cache as a prefix. The edge will be stored in memory as only `/abc`. Further edges that are inserted into the cache and that point to the same collection will also be stored prefix-compressed. The prefix compression is transparent and does not require configuration or setup. Compression is done separately for each cache, i.e. a separate prefix can be used for each individual edge index, and separately for the `_from` and `_to` parts. Lookups from the in-memory edge cache will not return compressed values but the full-length edge ids. The compressed values will also be used in memory only and will not be persisted on disk. v3.10.4 (2023-02-19) -------------------- * Updated ArangoDB Starter to 0.15.7. * Updated OpenSSL to 1.1.1t and OpenLDAP to 2.6.4. * BTS-1184: Fixed index hint with `forceIndexHint` set to true not being used on query when geo index was present, because it would override the choice of the index hint with optimizations related to it. * Fixed EE: Concurrent batch insert/update CRUD operations into SmartEdgeCollections on conflicting edge keys could get the smart edge caching out-of-sync, which would yield different results for OUTBOUND/INBOUND search over edges. This is now fixed, however there is now a slightly higher chance to get a CONFLICT response back on those queries. * Return peak memory usage and execution time as part of query explain result. This helps finding queries that use a lot of memory to build the execution plan. * Made all transactions used by the gharial API on coordinators and a few others marked "globally managed". This fixes an issue where transaction conflicts could lead to a silent out of sync situation between a leader shard and its followers. * BTS-1219: Fix cost estimation for geo index usage and for collection enumeration with included filtering. This fixes a regression from 3.9 where a geo index was no longer used because of an optimizer rule, which gained new powers, and wrong cost estimations for execution plans. * Allow usage of document projections and traversal projections in slightly more cases, specifically when the document's or traversal's output variables were used in subqueries. Previously the usage of the document or traversal output variables in subqueries could lead to projections being disabled. * Improved optimization of functions to be covered by Traversals. Now more functions should be optimized into the traversal, and some that are not valid should not be optimized anymore. Fixes #16589. * BTS-1193: Fix for schema update. When removing a field and then inserting a new field into the schema, previously, both old and new schema would be merged, meaning it would maintain the old field and add the new one. * Fixed issue #18053: Computed Values become null when Schema is modified. * Set the cache_oblivious option of jemalloc to `false` by default. This helps to save 4096 bytes of RAM for every allocation which is at least 16384 bytes large. This is particularly beneficial for the RocksDB buffer cache. * Added startup option `--javascript.user-defined-functions`. This option controls whether JavaScript user-defined functions (UDFs) can be used in AQL queries. The option defaults to `true`. The option can be set to `false` to disallow using JavaScript UDFs from inside AQL queries. In that case, a parse error will be thrown when trying to run a query that invokes a UDF. * Allowing enabling/disabling supervision maintenance mode also via followers in active failover mode. Previously the supervision maintenance mode could only be enabled/disabled by making a call to the active failover leader. * BTS-266: When starting up a cluster without `--cluster.force-one-shard`, creating a database and then restarting the cluster with the startup option `--cluster.force-one-shard` set to true, when the formerly created database has more than one shard, but the flag is set to true, this could lead to arangosearch's analyzers to use optimizations that should not be used if not in a single shard mode. For this not to happen, the verification of the parameter being true as a condition to run optimizations was removed. * Activate RDB_CoveringIterator and use it for some geo index queries. This speeds up and simplifies geo queries with geo index which do not use GEO_DISTANCE. v3.10.3 (2023-01-23) -------------------- * Log information about follower state/apply progress in supervision job that organizes failover in active failover mode. * Updated arangosync to v2.14.0. * Updated ArangoDB Starter to 0.15.6. * Fix bug in hotbackup download/restore to make sure no data is mixed up between servers. This fixes a bug introduced in 3.10. Note that previous 3.10 versions may not correctly restore a hotbackup which was uploaded from one cluster and downloaded into another. * ES-1396: under some rare circumstances it was possible that background index creation missed some documents in case the documents were inserted after background index creation started and the corresponding WAL files with the inserts were already removed before background indexing caught up. * Web UI [FE-48]: Additional fix to the previously introduced license information usability improvement. In case the server is being started with the additional parameter `--server.harden`, the previous fix did not handle that specific edge case. * BTS-413: Added more explanatory messages for when the user cannot see the statistics for a node in the UI when in cluster mode. * Fix coordinator segfault in AQL queries in which the query is invoked from within a JavaScript context (e.g. from Foxx or from the server's console mode) **and** the query has multiple coordinator snippets of which except the outermost one invokes a JavaScript function. Instead of crashing, coordinators will now respond with the exception "no v8 context available to enter for current transaction context". For AQL queries that called one of the AQL functions `CALL` or `APPLY` with a fixed function name, e.g. `APPLY('CONCAT', ...)`, it is now also assumed correctly that no JavaScript is needed, except if the fixed function name is the name of a user-defined function. This fixes an issue described in OASIS-24962. * BTS-1192: fix a potential race during hot backup creation, which could result in error messages such as `{backup} Source file engine_rocksdb/002516.sst does not have a hash file.` during hot backup creation. However, despite the error message being logged, the hot backup was still complete. * Prevent agency configuration confusion by an agent which comes back without its data directory and thus without its UUID. * Change the request lane for replication catchup requests that leaders in active failover receive from their followers from medium to high. This will give catchup requests from followers highest priority, so that the leader will preferrably execute them compared to regular requests. * Allow cluster database servers to start even when there are existing databases that would violate the settings `--cluster.min-replication-factor` or `--cluster.max-replication-factor`. This allows upgrading from older versions in which the replication factor validation for databases was not yet present. * Remove constant values for query variables from query plan serialization in cases they were not needed. Previously, constant values of query variables were always serialized for all occurrences of a variable in a query plan. If the constant values were large, this contributed to higher serialization and thus query setup times. Now the constant values are only serialized for relevant parts of query execution plans. * Added startup option `--rocksdb.bloom-filter-bits-per-key` to configure the average number of bits to use per key in a Bloom filter. * Make the cache_oblivious option of jemalloc configurable from the environment. This helps to save 4096 bytes of RAM for every allocation which is at least 16384 bytes large. This is particularly beneficial for the RocksDB buffer cache. * Improve performance of RocksDB's transaction lock manager by using different container types for the locked keys maps. This can improve performance of write-heavy operations that are not I/O-bound by up to 10%. v3.10.2 (2022-12-16) -------------------- * Added experimental per-operation option `refillIndexCaches` to write operations, namely: - AQL INSERT/UPDATE/REPLACE/REMOVE modification operations - single-document insert, update, replace and remove operations - multi-document insert, update, replace and remove operations If the option is set to `true` every currently running transaction will keep track of which in-memory edge index cache entries were invalidated by the transaction, and will try to (re-)fill them later. Currently only edge indexes are supported. ArangoDB 3.11 will add support for velocypack-based indexes (persistent, hash, sklipist index). Example usages: - `db..insert({ _from: ..., _to: ..., ... }, { refillIndexCaches: true });` - `db..update(key, { _from: ..., _to: ..., ... }, { refillIndexCaches: true });` - `db..replace(key, { _from: ..., _to: ..., ... }, { refillIndexCaches: true });` - `db..remove(key, { refillIndexCaches: true });` - `INSERT { ... } INTO OPTIONS { refillIndexCaches: true }` - `UPDATE { ... } WITH { ... } INTO OPTIONS { refillIndexCaches: true }` - `REPLACE { ... } WITH { ... } INTO OPTIONS { refillIndexCaches: true }` - `REMOVE { ... } IN OPTIONS { refillIndexCaches: true }` The refilling of the in-memory caches for indexes is performed by a background thread, so that the foreground write operation shouldn't be slowed down a lot. The background thread may however cause additional I/O for looking up the data in RocksDB and for repopulating the caches. The background refilling is done in a best-effort way and is not guaranteed to always succeed, e.g. if there is no memory available for the cache subsystem, or when an in-memory cache table is currently in a migration phase(grow/shrink operation). There is a new experimental startup option `--rocksdb.auto-refill-index-caches-on-modify` for DB-Servers and single servers, which currently defaults to `false`. If it is set to `true`, the cache refilling will be turned on automatically for all insert/update/replace/remove operations, so that it doesn't need to be specified on the per-operation/per-query level. The experimental option `--rocksdb.auto-refill-index-caches-queue-capacity` can be used to limit the number of index cache entries that the background thread will queue. This is a safeguard to keep the memory usage at bay in case the background thread is slower than concurrent threads that perform ingestions. There are also new experimental startup options to control whether or not the in-memory caches should automatically be seeded upon server restart. The option `--rocksdb.auto-fill-index-caches-on-startup` for DB-Servers and single servers enables this functionality. It currently defaults to `false`. If it is set to `true`, the in-memory caches of all eligible indexes will be automatically pre-seeded after the server startup. Note that this may cause additional CPU and I/O load. The option `--rocksdb.max-concurrent-index-fill-tasks` is available to limit the impact of the automatic index filling at startup. It controls how many full index filling operations can execute concurrently. The lower this number is, the lower the impact of cache filling, but the longer it will take. The default value for this option depends on the number of available cores, and is at least `1`. A value of `0` cannot be used. This option is only relevant if `--rocksdb.auto-fill-index-caches-on-startup` is set to `true`. The PR also adds the following metrics: - `rocksdb_cache_auto_refill_loaded_total`: Total number of queued items for in-memory index caches refilling. It will always report a value of zero on coordinators. - `rocksdb_cache_auto_refill_dropped_total`: Total number of dropped items for in-memory index caches refilling (because number of queued items would exceed the value of `--rocksdb.auto-refill-index-caches-queue-capacity`). It will always report a value of zero on coordinators. - `rocksdb_cache_full_index_refills_total`: Total number of in-memory index caches refill operations for entire indexes. The counter gets increased for every index automatically loaded (because startup option `--rocksdb.auto-fill-index-caches-on-startup` is set to `true`) or when full indexes are loaded into memory manually. In cluster deployments the counter will be increased once per eligible index per shard. It will always report a value of zero on coordinators. * Use intermediate commits in old shard synchronization protocol. This avoids overly large RocksDB transactions when syncing large shards, which is a remedy for OOM kills during restarts. * Added a configuration option (for the agency): --agency.supervision-failed-leader-adds-follower with a default of `true` (behavior as before). If set to `false`, a `FailedLeader` job does not automatically configure a new shard follower, thereby preventing unnecessary network traffic, CPU and IO load for the case that the server comes back quickly. If the server is permanently failed, an `AddFollower` job will be created anyway eventually. * Max value of minhash_value was set to 2^53 - 1 (9007199254740991) to stay in safe integer limits for javascript. * SEARCH-433 Fix Inverted index fields presence checks for IN clauses. * BTS-1141: Changed the default value of startup option `--rocksdb.enforce-block-cache-size-limit` from `true` to `false`. This change prevents RocksDB from going into read-only mode when an internal operation tries to insert some value into the block cache, but can't do so because the block cache's capacity limit is reached. * Don't log Boost ASIO warnings such as `asio IO error: 'stream truncated'` when a peer closes an SSL/TLS connection without performing a proper connection shutdown. * Disallow creating new databases with a `replicationFactor` value set to a value lower than `--cluster.min-replication-factor` or higher than `--cluster.max-replication-factor`. Previously the `replicationFactor` settings for new databases were not bounds-checked, only for new collections. * Fixed Github issue #16451: In certain situations, a LIMIT inside a subquery could erroneously reduce the number of results of the containing (sub)query. * Added a feature to the ResignLeadership job. By default, it will now undo the leader changes automatically after the server is restarted, unless the option `undoMoves` is set to `false`. This will help to make rolling upgrades and restarts less troublesome, since the shard leaderships will not get unbalanced. * Add missing metrics for user traffic: Histograms: `arangodb_client_user_connection_statistics_bytes_received` `arangodb_client_user_connection_statistics_bytes_sent` These numbers were so far only published via the statistics API. This is needed for Oasis traffic accounting. * Added agency options --agency.supervision-delay-add-follower and --agency.supervision-delay-failed-follower to delay supervision actions for a configurable amount of seconds. This is desirable in case a DBServer fails and comes back quickly, because it gives the cluster a chance to get in sync and fully resilient without deploying additional shard replicas and thus without causing any data imbalance. * Enable "collect-in-cluster" optimizer rule for SmartGraph edge collections. * Fixed SEARCH-408 Added "cache" columns feature for ArangoSearch. * Fixed SEARCH-427 Fixed Inverted index usage of field with trackListPositions enabled. * Fix HTTP/VST traffic accounting in internal statistics / metrics. * Add serverId parameter to _admin/log/level. Allows to forward the request to other servers. * Delay a MoveShard operation for leader change, until the old leader has actually assumed its leadership and until the new leader is actually in sync. This fixes a bug which could block a shard under certain circumstances. This fixes BTS-1110. * Updated arangosync to v2.13.0. * Fixed issue #17291: Server crash on error in the PRUNE expression. Traversal PRUNE expressions containing JavaScript user-defined functions (UDFs) are now properly rejected in single server and cluster mode. PRUNE expressions that use UDFs require a V8 context for execution, which is not available on DB-servers in a cluster, and also isn't necessarily available for regular queries on single servers (a V8 context is only available if a query was executed inside Foxx or from inside a JS transaction, but not otherwise). * Removed more assertions from cluster rebalance js test that obligated the rebalance plan to always have moves, but there were cases in which all there are none. * Fix setting query memory limit to 0 for certain queries if a global memory limit is set, but overriding the memory limit is allowed. * Do not query vertex data in K_PATHS queries if vertex data is not needed. * BTS-1075: AQL: RETURN DOCUMENT ("") inconsistent - single server vs cluster. * Repair "load indexes into memory" function in the web UI. * Fixed issue #17367: FILTER fails when using negation (!) on variable whose name starts with "in". Add trailing context to NOT IN token. * Fix disk space metrics rocksdb_free_disk_space and rocksdb_total_disk_space on macOS. Previously, they seem to have reported wrong values. * Show number of HTTP requests in cluster query profiles. * Removed assertions from cluster rebalance js test that obligated the rebalance plan to always have moves, but there were cases in which all there are none. * Improved the syntax highlighter for AQL queries in the web interface with support for multi-line strings, multi-line identifiers in forward and backticks, colorization of escape sequences, separate tokens for pseudo-keywords and pseudo-variables, an updated regex for numbers, and the addition of the AT LEAST and WITH COUNT INTO constructs. v3.10.1 (2022-11-04) -------------------- * Added detailed explanations for some startup options. They are only exposed via `--dump-options` under the `longDescription` key. * Updated OpenSSL to 1.1.1s. * Solve a case of excessive memory consumption in certain AQL queries with IN filters with very long lists. Free sub-iterators as soon as they are exhausted. * BTS-1070: Fixed query explain not dealing with an aggregate function without arguments and the WINDOW node not being defined as an Ast node type name. * Updating properties of a satellite collection breaks replicationFactor. * Log the documents counts on leader and follower shards at the end of each successful shard synchronization. * Remove superfluous dash character from startup option name `--temp.-intermediate-results-encryption-hardware-acceleration`. * FE-159: When creating a database in cluster mode, there are several parameters required. However they are invisible (nothing shown) if I open DB settings after creation. Those settings should be visible in readonly mode (grey out). * Added startup option `--query.log-failed` to optionally log all failed AQL queries to the server log. The option is turned off by default. * Added startup option `--query.log-memory-usage-threshold` to optionally log all AQL queries that have a peak memory usage larger than the configured value. The default value is 4GB. * Added startup option `--query.max-artifact-log-length` to control the maximum length of logged query strings and bind parameter values. This allows truncating overly long query strings and bind parameter values to a reasonable length. Previously the cutoff length was hard-coded. * Improve cardinality estimate for AQL EnumerateCollectionNode in case a `SORT RAND() LIMIT 1` is used. Here, the estimated number of items is at most 1. * Improved shard distribution during collection creation. * Changed the encoding of revision ids returned by the following REST APIs: - GET /_api/collection//revision: the revision id was previously returned as numeric value, and now it will be returned as a string value with either numeric encoding or HLC-encoding inside. - GET /_api/collection//checksum: the revision id in the "revision" attribute was previously encoded as a numeric value in single server, and as a string in cluster. This is now unified so that the "revision" attribute always contains a string value with either numeric encoding or HLC-encoding inside. * Fixed handling of empty URL parameters in HTTP request handling. * Fixed diffing of completely non-overlapping revision trees, which could lead to out-of-bounds reads at the right end of the first (smaller) tree. * Fixed aborting the server process if an exception was thrown in C++ code that was invoked from the llhttp C code dispatcher. That dispatcher code couldn't handle C++ exceptions properly. * Fixed BTS-1073: Fix encoding and decoding of revision ids in replication incremental sync protocol. Previously, the encoding of revision ids could be ambiguous under some circumstances, which could prevent shards from getting into sync. * ES-1312: fix handling of reaching the WAL archive capacity limit. * Log better diagnosis information in case multiple servers in a cluster are configured to use the same endpoint. * BTS-908: Fixed WebUI GraphViewer not being able to create a new edge relation between two nodes in cases where only one edge definition has been defined inside the graph definition. * MDS-1019: Make user search case-insensitive and allow search by name. * MDS-1016: When creating a new collection the fields "Number of Shards" and "Replication factor" are greyed out now when the field "Distribute shards like" is not empty. * Fixed BTS-850: Fixed the removal of already deleted orphan collections out of a graph definition. The removal of an already deleted orphan collection out of a graph definition failed and has been rejected in case the collection got dropped already. * BTS-1008: Update react-autocomplete-input to fix single letter collection bug when creating a link in the views in the WebUI. * BTS-1061: ARM was not recognized on Apple M1. * BTS-325: Changed the HTTP status code from `400` to `404` of the ArangoDB error code `ERROR_GRAPH_REFERENCED_VERTEX_COLLECTION_NOT_USED` to handle this error in accordance to our edge errors. * Adjust permissions for "search-alias" views. Previously, "search-alias" views were visible to users that didn't have read permissions on the underlying referenced collections. This was inconsistent, because "arangosearch" views weren't shown to users that didn't have read permissions on the underlying links. Now, the behavior for "search-alias" views is the same as for "arangosearch" views, i.e. "search-alias" views are not shown and are not accessible for users that don't have at least read permissions on the underlying collections. * BTS-969: Added restriction for HTTP request `/cluster/rebalance`not to consider servers that have failed status as a possible target for rebalancing shards in its execution plan. * Fix an issue with replication of arangosearch view change entries in single server replication and active failover. Previously, when changing the properties of existing views, the changes were not properly picked up by followers in these setups. Cluster setups were not affected. v3.10.0 (2022-09-29) -------------------- * Convert v3.10.0-rc.1 into v3.10.0. v3.10.0-rc.1 (2022-09-25) ------------------------- * Temporary fix for BTS-1006 (hides new view types). * Fixed SEARCH-399 Rule `restrict-to-single-shard` now works properly with inverted index. * Fixed SEARCH-393 fixed analyzer setting for nested fields. * APM-517: Add tooltips with values of the displayed properties after clicking a node or an edge in the graph viewer. * Updated arangosync to v2.12.0. * Improve upload and download speed of hotbackup by changing the way we use rclone. Empty hash files are now uploaded or downloaded by pattern, and all other files are done in batches without remote directory listing, which allows rclone to parallelize and avoid a lot of unnecessary network traffic. The format of hotbackups does not change at all. * Fixed SEARCH-392 Fixed field features propagation. * Fixed SEARCH-388 Fixed handling nested subfields. * Fixed SEARCH-379 Transaction is properly set to index during optimization. * Fixed issue BTS-1018: Improve logging of binary velocypack request data. * Updated ArangoDB Starter to 0.15.5. * Fixed BTS-1017: Fixed a graph search issue, where subqueries lead to incorrect results when they have been pushed down fully onto a DBServer when they are in a Hybrid Disjoint SmartGraph context and SatelliteCollections were part of it. * Fixed issue BTS-1023: Added Linux-specific startup option `--use-splice-syscall` to control whether the Linux-specific splice() syscall should be used for copying file contents. While the syscall is generally available since Linux 2.6.x, it is also required that the underlying filesystem supports the splice operation. This is not true for some encrypted filesystems, on which splice() calls thus fail. By setting the startup option `--use-splice-syscall` to `false`, a less efficient, but more portable user-space file copying method will be used instead, which should work on all filesystems. The startup option is not available on other operating systems than Linux. * Fixed SEARCH-386 Fixed disjunction coverage in the inverted index with non-default analyzers. * Fixed SEARCH-373 Fixed indexing same field as nested and as non-nested. v3.10.0-beta.1 (2022-09-14) --------------------------- * Display progress during Arangosearch link and inverted index recovery. * Fixed SEARCH-374 and SEARCH-358 Fixed a rare case of wrong seek operation in the sparse_bitset during query ArangoSearch execution. * Updated arangosync to v2.12.0-preview-12. * Updated ArangoDB Starter to 0.15.5-preview-3. * Fixed a rare occurring issue where paths inside a DisjointSmart traversal containing only satellite relevant nodes were not returned properly (ES-1265). * Implement prefetch for revision trees, in case a batch is created with a distinguished collection as for `SynchronizeShard`. This ensures that the revision tree for the batch will be available when needed, even though the revision tree for the collection might already have advanced beyond the sequence number of the snapshot in the batch. This ensures that shards can get in sync more reliably and more quickly. * Added startup option `--rocksdb.periodic-compaction-ttl`. This option controls the TTL (in seconds) for periodic compaction of .sst files in RocksDB, based on the .sst file age. The default value from RocksDB is ~30 days. To avoid periodic auto-compaction, the option can be set to 0. * Fixed SEARCH-366: Fixed extracting sub-attributes using projections for the `_id` attribute (e.g. `RETURN doc.sub._id`). * Fixed SEARCH-368: Fixed handling of array comparison for inverted index. * Fix get snapshot on single server for search-alias view when index was removed. * Fix waitForSync options for SEARCH in non-maintainer build. * Fix SEARCH-340 and SEARCH-341: add stats and metrics for inverted index. * Fixed SEARCH-334 Added searchField option for inverted index. * Fixed SEARCH-376 Assertion fixed for non-array value in array comparison. * Fixed BTS-926: UI showing the "create index" form to non-admin users. * Added startup option `--arangosearch.skip-recovery` to skip the recovery of arangosearch view links or inverted indexes. The startup option can be specified multiple times and is expected to either contain the string `all` (will skip the recovery for all view links and inverted indexes) or a collection name + link id/name pair (e.g. `testCollection/123456`, where `123456` is a link/index id or an index name). This new startup option is an emergency means to speed up lengthy recovery procedures when there is a large WAL backlog to replay. The normal recovery will still take place even with the option set, but recovery data for links/indexes can be skipped. This can improve the recovery speed and reduce memory usage during the recovery process. All links or inverted indexes that are marked as to-be-skipped via the option, but for which there is recovery data, will be marked as "out of sync" at the end of the recovery. The recovery procedure will also print a list of links/indexes which it has marked as out-of-sync. Additionally, if committing data for a link/index fails for whatever reason, the link/index is also marked as being out-of-sync. If an out-of-sync link or index can be used in queries depends on another new startup option `--arangosearch.fail-queries-on-out-of-sync`. It defaults to `false`, meaning that out-of-sync links/indexes can still be queried. If the option is set to `true`, queries on such links/indexes will fail with error "collection/view is out of sync" (error code 1481). Links/indexes that are marked out-of-sync will keep the out-of-sync flag until they are dropped. To get rid of an out-of-sync link/index, it is recommended to manually drop and recreate it. As recreating a link/index may cause high load, this is not done automatically but requires explicit user opt-in. The number of out-of-sync links/indexes is also observable via a new metric `arangodb_search_num_out_of_sync_links`. * Moved extensive log message down to DEBUG level. * Updated Views UI with all changes necessary for the 3.10.0 launch. * Fixed SEARCH-343 Fixed iterating all documents with nested fields. * Fixed SEARCH-322 Fixed executing empty nested condition. * Do not drop follower shard after too many failed shard synchronization attempts. * Disable optimization rule to avoid crash (BTS-951). * Fixed SEARCH-369: add a compatibility check to the search-alias view for indexes from the same collection. * Fixed BTS-959: no one call shutdown and delete datastore for the inverted index in some C++ tests. * Fix SEARCH-350: Crash during consolidation. * When using `SHORTEST_PATH`, `K_SHORTEST_PATHS`, `ALL_SHORTEST_PATHS`, or `K_PATHS` in an AQL Query and the query itself produced warnings during execution, the type has been wrongly reported. It reported always with `SHORTEST_PATH` and not the specific used one. * Fixed SEARCH-368 Fixed handling of array comparison for inverted index. * SEARCH-357: Added SUBSTRING_BYTES function. * Fixed SEARCH-347 Cycle variable reference in nested search query is properly detected and rejected. * Fixed SEARCH-364 Fixed index fields match check for inverted index. * Web UI: Reduce size and initial render height of a modal (fixes BTS-940). * Fix comparison of JSON schemas on DB servers after there was a schema change via a coordinator: the schema comparison previously did not take into account that some ArangoDB versions store an internal `{"type":"json"}` attribute in the schema, and some don't. Thus two identical schemas could compare differently. The correct schema version was always applied and used, and validation of documents against the schema was also not affected. However, because two schemas could compare unequal, this could have caused unnecessary repeated work for background maintenance threads. * Removed transitive node dependencies is-wsl and media-typer. * Web UI: Now correctly handles the server error response when an error occurred during the modification of a document or an edge (BTS-934). * Fixed SEARCH-328 fixed cookie key. * Make graph search case-insensitive (fixes BTS-882). * Fixed SEARCH-329 fixed removes with nested documents. * Fixed parsing on NOT operator in nested query filter. * Fixed SEARCH-346. If index creation is aborted due to existing same index new index is properly dropped if it was already instantiated. * Add progress reporting to RocksDB WAL recovery, in case there are many WAL files to recover. * Updated arangosync to v2.12.0-preview-9. v3.10.0-alpha.1 (2022-08-17) ---------------------------- * Updated arangosync to v2.12.0-preview-6. * Updated warning messages raised for non accepted query OPTIONS, distinguishing between when the OPTIONS attribute is correct, but the value is in incorrect format, and when the OPTIONS attribute itself is incorrect. * Since ArangoDB 3.8 there was a loophole for creating duplicate keys in the same collection. The requirements were: - cluster deployment - needs at least two collections (source and target), and the target collection must have more than one shard and must use a custom shard key. - inserting documents into the target collection must have happened via an AQL query like `FOR doc IN source INSERT doc INTO target`. In this particular combination, the document keys (`_key` attribute) from the source collection were used as-is for insertion into the target collection. However, as the target collection is not sharded by `_key` and uses a custom shard key, it is actually not allowed to specify user-defined values for `_key`. That check was missing since 3.8 in this particular combination and has now been added back. AQL queries attempting to insert documents into a collection like this will now fail with the error "must not specify _key for this collection", as they used to do before 3.8. * Updated ArangoDB Starter to 0.15.5-preview-1. * Improve error handling for passing wrong transaction ids / cursor ids / pregel job ids to request forwarding. Also prevent the error "transaction id not found" in cases when request forwarding was tried to a coordinator that was recently restarted. * Fixed an invalid attribute access in AQL query optimization. Without the fix, a query such as LET data = { "a": [ ... ], } FOR d IN data["a"] RETURN d could fail with error "invalid operand to FOR loop, expecting Array". * Added support for AT LEAST quantifier for SEARCH. * Fixed BTS-335 Ranges parsing fixed for nested queries. * BTS-907: Fixed some rare SortNode related optimizer issues, when at least two or more SortNodes appeared in the AQL execution plan. * Added new AQL function `VALUE` capable of accessing object attribute by a specified path. * Fixed BTS-918 (incorrectly navigating back 1 level in history when a modal-dialog element is present). * Added OFFSET_INFO function (Enterprise Edition only) to support search results highlighting. * Updated Rclone to v1.59.0. * Fixed BTS-902 (clicking on the search icon in the analyzers filter input used to take the user to the collections view). * Fixed BTS-852 (user's saved queries used to disappear after updating user profile). * ArangoSearch nested search feature (Enterprise Edition): Added ability to index and search nested documents with ArangoSearch views. * Updated OpenSSL to 1.1.1q and OpenLDAP to 2.6.3. * Fixed handling of illegal edges in Enterprise Graphs. Adding an edge to a SmartGraph vertex collection through document API caused incorrect sharding of the edge. Now this edge is rejected as invalid. (BTS-906) * Added CSP recommended headers to Aardvaark app for better security. * Added more specific process exit codes for arangod and all client tools, and changed the executables' exit code for the following situations: - an unknown startup option name is used: previously the exit code was 1. Now the exit code when using an invalid option is 3 (symbolic exit code name EXIT_INVALID_OPTION_NAME). - an invalid value is used for a startup option (e.g. a number that is outside the allowed range for the option's underlying value type, or a string value is used for a numeric option): previously the exit code was 1. Now the exit code for these case is 4 (symbolic exit code name EXIT_INVALID_OPTION_VALUE). - a config file is specified that does not exist: previously the exit code was either 1 or 6 (symbolic exit code name EXIT_CONFIG_NOT_FOUND). Now the exit code in this case is always 6 (EXIT_CONFIG_NOT_FOUND). - a structurally invalid config file is used, e.g. the config file contains a line that cannot be parsed: previously the exit code in this situation was 1. Now it is always 6 (symbolic exit code name EXIT_CONFIG_NOT_FOUND). Note that this change can affect any custom scripts that check for startup failures using the specific exit code 1. These scripts should be adjusted so that they check for a non-zero exit code. They can opt-in to more specific error handling using the additional exit codes mentioned above, in order to distinguish between different kinds of startup errors. * BTS-913: check for proper timezone setup of the system on startup. This will then log errors that else would only occur in AQL-Functions at runtime. * Added ALL_SHORTEST_PATHS functionality to find all shortest paths between two given documents. * Fixed a potential deadlock in RocksDB compaction. For details see https://github.com/facebook/rocksdb/pull/10355. * Changed rocksdb default compression type from snappy to lz4. * arangoimport now supports the option --remove-attribute on type JSON as well. Before it was restricted to TSV and CSV only. * Fixed BTS-851: "Could not fetch the applier state of: undefined". * Removed internal JavaScript dependencies "expect.js", "media-typer" and "underscore". We recommend always bundling your own copy of third-party modules as all previously included third-party modules are now considered deprecated and may be removed in future versions of ArangoD * APM-84: Added option to spill intermediate AQL query results from RAM to disk when their size exceeds certain thresholds. Currently the only AQL operation that can make use of this is the SortExecutor (AQL SORT operation without using a LIMIT). Further AQL executor types will be supported in future releases. Spilling over query results from RAM to disk is off by default and currently in an experimental stage. In order to opt-in to the feature, it is required to set the following startup option `--temp.intermediate-results-path`. The directory specified here must not be located underneath the instance's database directory. When this startup option is specified, ArangoDB assumes ownership of that directory and will wipe its contents on startup and shutdown. The directory can be placed on ephemeral storage, as the data stored inside it is there only temporarily, while the instance is running. It does not need to be persisted across instance restarts and does not need to be backed up. When a directory is specified via the startup option, the following additional configuration options can be used to control the threshold values for spilling over data: * `--temp.intermediate-results-capacity`: maximum on-disk size (in bytes) for intermediate results. If set to 0, it means that the on-disk size is not constrained. It can be set to a value other than 0 to restrict the size of the temporary directory. Once the cumulated on-disk size of intermediate results reaches the configured maximum capacity, the query will be aborted with failure "disk capacity limit for intermediate results exceeded". * `--temp.intermediate-results-spillover-threshold-num-rows`: number of result rows from which on a spillover from RAM to disk will happen. * `--temp.intermediate-results-spillover-threshold-memory-usage`: memory usage (in bytes) after which a spillover from RAM to disk will happen. * `--temp.intermediate-results-encryption`: whether or not the on-disk data should be encrypted. This option is only available in the Enterprise Edition. * `--temp.-intermediate-results-encryption-hardware-acceleration`: whether or not to use hardware acceleration for the on-disk encryption. This option is only available in the Enterprise Edition. Please note that the feature is currently still experimental and may slightly change in future releases. As mentioned, the only Executor that can make use of spilling data to disk is the SortExecutor (SORT without LIMIT). Also note that the query results will still be built up entirely in RAM on coordinators and single servers for non-streaming queries. In order to avoid the buildup of the entire query result in RAM, a streaming query should be used. * Enterprise only: Added `MINHASH`, `MINHASH_MATCH`, `MINHASH_ERROR`, `MINHASH_COUNT` AQL functions. * Enterprise only: Added `minhash` analyzer. * BugFix in Pregel's status: When loading the graph into memory, Pregel's state is now 'loading' instead of 'running'. When loading is finished, Pregel's state changes to the 'running' state. * arangoimport now supports an additional option "--overwrite-collection-prefix". This option will only help while importing edge collections, and if it is used together with "--to-collection-prefix" or "--from-collection-prefix". If there are vertex collection prefixes in the file you want to import (e.g. you just exported an edge collection from ArangoDB) you allow arangoimport to overwrite those with the commandline prefixes. If the option is false (default value) only _from and _to values without a prefix will be prefixed by the handed in values. * Added startup option `--rocksdb.compaction-style` to configure the compaction style which is used to pick the next file(s) to be compacted. * BugFix in Pregel's Label Propagation: the union of three undirected cliques of size at least three connected by an undirected triangle now returns three communities (each clique is a community) instead of two. * Pregel now reports correct and ongoing runtimes for loading, running, and storing as well as runtimes for the separate global supersteps. * Fixed parsing of K_SHORTEST_PATHS queries to not allow ranges anymore. * Add log.time-format utc-datestring-micros to make debugging of concurrency bugs easier. * Renamed KShortestPathsNode to EnumeratePathsNote; this is visible in explain outputs for AQL queries. * Pregel SSSP now supports `resultField` as well as `_resultField` as parameter name to specify the field into which results are stored. The name `_resultField` will be deprecated in future. * Update Windows CI compiler to Visual Studio 2022. * Web UI: Fixes a GraphViewer issue related to display issues with node and edge labels. Boolean node or edge values could not be used as label values (ES-1084). * Made the SortExecutor receive its input incrementally, instead of receiving a whole matrix containing all input at once. * Optimization for index post-filtering (early pruning): in case an index is used for lookups, and the index covers the IndexNode's post-filter condition, then loading the full document from the storage engine is now deferred until the filter condition is evaluated and it is established that the document matches the filter condition. * Added a fully functional UI for Views that lets users view, modify mutable properties and delete views from the web UI. * Fix thread ids and thread names in log output for threads that are not started directly by ArangoDB code, but indirectly via library code. Previously, the ids of these threads were always reported as "1", and the thread name was "main". Now return proper thread ids and names. * Changed default Linux CI compiler to gcc-11. * Add "AT LEAST" quantifier for array filters in AQL: `RETURN [1,2,3][? AT LEAST (3) FILTER CURRENT > 42]` `RETURN [1,2,3] AT LEAST (2) IN [1,2,3,4,5]` * Changed default macOS CI compiler to LLVM clang-14. * Added an automatic cluster rebalance api. Use `GET _admin/cluster/rebalance` to receive an analysis of how imbalanced the cluster is. Calling it with `POST _admin/cluster/rebalance` computes a plan of move shard operations to rebalance the cluster. Options are passed via the request body. After reviewing the plan, one can use `POST _admin/cluster/rebalance/execute` to put that plan into action. * Introduce reading from followers in clusters. This works by offering an additional HTTP header "x-arango-allow-dirty-read" for certain read-only APIs. This header has already been used for active failover deployments to allow reading from followers. Using this header leads to the fact that coordinators are allowed to read from follower shards instead only from leader shards. This can help to spread the read load better across the cluster. Obviously, using this header can result in "dirty reads", which are read results returning stale data or even not-yet-officially committed data. Use at your own risk if performance is more important than correctness or if you know that data does not change. The responses which can contain dirty reads will have set the HTTP header "x-arango-potential-dirty-read" set to "true". There are the following new metrics showing the use of this feature: - `arangodb_dirty_read_transactions_total` - `arangodb_potentially_dirty_document_reads_total` - `arangodb_dirty_read_queries_total` * Changed HTTP response code for error number 1521 from 500 to 400. Error 1521 (query collection lock failed) is nowadays only emitted by traversals, when a collection is accessed during the traversal that has not been specified in the WITH statement of the query. Thus returning HTTP 500 is not a good idea, as it is clearly a user error that triggered the problem. * Renamed the `--frontend.*` startup options to `--web-interface.*`: - `--frontend.proxy-request.check` -> `--web-interface.proxy-request.check` - `--frontend.trusted-proxy` -> `--web-interface.trusted-proxy` - `--frontend.version-check` -> `--web-interface.version-check` The former startup options are still supported. * Added Enterprise Graph feature to enterprise version of ArangoDB. The enterprise graph is another graph sharding model that we introduced, it is less strict, and therefore easier to start with, then SmartGraphs, as it does not require a smartGraphAttribute, and allows free choice of vertex _key values. But still maintains performance gains as compared to general-graphs. For more details please check documentation. * APM-135: Added multithreading to assigning non-unique indexes to documents, in foreground or background mode. The number of index creation threads is hardcoded to 2 for now. Improvements for higher parallelism are expected for future versions. * Issue 15592: Permit `MERGE_RECURSIVE()` to be called with a single argument. * Fixed issue 16337: arangoimport with `--headers-file` and `--merge-attributes` merges column names instead of row values on the first line of a CSV file. Additionally, floating-point numbers are now merged using their standard string representation instead of with a fixed precision of 6 decimal places. * Now supporting projections on traversals. In AQL Traversal statements like FOR v,e,p IN 1..3 OUTBOUND @start GRAPH @graph RETURN v.name we will now detect attribute accesses on the data, in above example "v.name" and use it to optimize data-loading, e.g. we will only extract the "name" attribute. This optimization will help if you have large document sizes, but only access small parts of the documents. By default we will only project up to 5 attributes on each vertex, and edge. This limit can be modified by adding OPTIONS {maxProjections: 42}. To identify if your query is using projections the explain output will now contain a hint like `/* vertex (projections: `name`) */` For now only attribute accesses are detected, functions like `KEEP` will not be projected. * Change default `format_version` for RocksDB .sst files from 3 to 5. * Added support for creating autoincrement keys on cluster mode, but only for single sharded collections. * Add support for LZ4 and LZ4HC compression support for RocksDB. * Allow parallel access to the shards of smart edge collections in AQL via parallel GatherNodes. * Update RocksDB internal table checksum type to xxHash64. * Added several startup option to configure parallelism for individual Pregel jobs: - `--pregel.min-parallelism`: minimum parallelism usable in Pregel jobs. - `--pregel.max-parallelism`: maximum parallelism usable in Pregel jobs. - `--pregel.parallelism`: default parallelism to use in Pregel jobs. These parallelism options can be used by administrators to set concurrency defaults and bounds for Pregel jobs. Each individual Pregel job can set its own parallelism value using the job's `parallelism` option, but the job's parallelism value will be clamped to the bounds defined by `--pregel.min-parallelism` and `--pregel.max-parallelism`. If a job does not set its `parallelism` value, it will default to the parallelism value configured via `--pregel.parallelism`. * Added startup options to configure the usage of memory-mapped files for Pregel temporary data: - `--pregel.memory-mapped-files`: if set to `true`, Pregel jobs will by default store their temporary data in disk-backed memory-mapped files. If set to `false`, the temporary data of Pregel jobs will be buffered in RAM. The default value is `true`, meaning that memory-mapped files will be used. The option can be overridden for each Pregel job by setting the `useMemoryMaps` option of the job. - `--pregel.memory-mapped-files-location-type`: location for memory-mapped files written by Pregel. This option is only meaningful if memory-mapped files are actually used. The option can have one of the following values: - `temp-directory`: store memory-mapped files in the temporary directory, as configured via `--temp.path`. If `--temp.path` is not set, the system's temporary directory will be used. - `database-directory`: store memory-mapped files in a separate directory underneath the database directory. - `custom`: use a custom directory location for memory-mapped files. The exact location must be set via the configuration parameter `--pregel.memory-mapped-files-custom-path`. The default value for this option is `temp-directory`. - `--pregel.memory-mapped-files-custom-path`: custom directory location for Pregel's memory-mapped files. This setting can only be used if the option `--pregel.memory-mapped-files-location-type` is set to `custom`. The default location for Pregel's memory-mapped files is the temporary directory (`temp-directory`), which may not provide enough capacity for larger Pregel jobs. It may be more sensible to configure a custom directory for memory-mapped files and provide the necessary disk space there (`custom`). Such custom directory can be mounted on ephemeral storage, as the files are only needed temporarily. There is also the option to use a subdirectory of the database directory as the storage location for the memory-mapped files (`database-directory`). The database directory often provides a lot of disk space capacity, but when it is used for both the regular database data and Pregel's memory-mapped files, it has to provide enough capacity to store both. * Pregel status now reports whether memory mapped files are used in a job. * Change default value of `--rocksdb.block-cache-shard-bits` to an automatic default value that allows data blocks of at least 128MiB to be stored in each cache shard if the block cache's strict capacity limit is used. The strict capacity limit for the block cache is enabled by default in 3.10, but can be turned off by setting the option `--rocksdb.enforce-block-cache-size-limit` to `false`. Also log a startup warning if the resulting cache shard size would be smaller than is potentially safe when the strict capacity limit is set. Enforcing the block cache's capacity limit has the consequence that data reads by RocksDB must fit into the block cache or the read operation will fail with an "Incomplete" error. * The API `/_admin/status` now returns a progress attribute that shows the server's current state (starting, stopping, etc.), with details about which feature is currently started, stopped etc. During recovery, the current WAL recovery sequence number is also reported in a sub-attribute of the `progress` attribute. Clients can query this attribute to track the progress of the WAL recovery. The additional progress attribute returned by `/_admin/status` is most useful when using the `--server.early-connections true` setting. With that setting, the server will respond to incoming requests to a limited set of APIs already during server startup. When the setting is not used, the REST interface will be opened relatively late during the startup sequence, so that the progress attribute will likely not be very useful anymore. * Optionally start up HTTP interface of servers earlier, so that ping probes from tools can already be responded to when the server is not fully started. By default, the HTTP interface is opened at the same point during the startup sequence as before, but it can optionally be opened earlier by setting the new startup option `--server.early-connections` to `true`. This will open the HTTP interface early in the startup, so that the server can respond to a limited set of REST APIs even during recovery. This can be useful because the recovery procedure can take time proportional to the amount of data to recover. When the `--server.early-connections` option is set to `true`, the server will respond to requests to the following APIs during the startup already: - `/_api/version` - `/_admin/version` - `/_admin/status` All other APIs will be responded to with an HTTP response code 503, so that callers can see that the server is not fully ready. If authentication is used, then only JWT authentication can be used during the early startup phase. Incoming requests relying on other authentication mechanisms that require access to the database data will also be responded to with HTTP 503 errors, even if correct credentials are used. * Upgraded bundled version of RocksDB to 7.2. * Added `[?]` array operator to AQL, which works as follows: - `nonArray[?]`: returns `false` - `nonArray[? FILTER CURRENT ...]`: returns `false` - `array[?]`: returns `false` if array is empty, `true` otherwise - `array[? FILTER CURRENT ...]`: returns `false` if no array member satisfies the filter condition, returns `true` if at least one member satisfies it. * Upgrade jemalloc to version 5.3.0. * Set "useRevisionsAsDocumentIds" to true when restoring collection data via arangorestore in case it is not set in the collection structure input data. This allows using revision trees for restored collections. * Added new optimization rule "arangosearch-constrained-sort" to perform sorting & limiting inside ArangoSearch View enumeration node in case of using just scoring for sort. * Updated lz4 to version 1.9.3. * Added option `--custom-query-file` to arangoexport, so that a custom query string can also be read from an input file. * FE-46: UI improvement on the view UI pages as well as adding tooltips to options where necessary. The affected pages are mostly the Info and Consolidation Policy pages. * FE-44: Moved the Info page to before JSON, making the settings page the default page in the view web UI. * Refactor internal code paths responsible for `_key` generation. For collections with only a single shard, we can now always let the leader DB server generate the keys locally. For collections with multiple shards, the coordinators are now always responsible for key generation. Previously the responsibility was mixed and depended on the type of operation executed (document insert API vs. AQL query, single operation vs. batch). * Make web UI show the following information for collections: - key generator type - whether or not the document and primary index cache is enabled - if cache is enabled, show cache usage and allocation size in figures The `cacheEnabled` property of collections is now also changeable via the UI for existing collections. * FE-45: Added tooltips with helpful information to the options on the View UI settings page. * FE-43: Simplify the workflow on the web view UI (Links page): allow for users to view a single link or field with their properties at a time. * Fixed BTS-811 in which there was an incongruence between data being checksummed and data being written to `.sst` files, because checksumming should have been made after the encryption of the data, not before it. * Added command line option to arangobench to disable implicit collection creation. This allows one to run tests against a manually created and configured collection. * Fix deadlocked shard synchronizations when planned shard leader has not yet taken over leadership. * Unify the creation of normal and SmartGraph collections. This unifies the code paths for creating collections for normal collections and SmartGraph collections, so that the functionality is centralized in one place. SmartGraph-specific code for validation and collection creation has been moved to enterprise as well. * Auto-regenerate exit code and error code files in non-maintainer mode, too. * Only show slowest optimizer rules in explain output for optimizer rules that took a considerable amount of time (>= 0.0002 seconds). Previously the slowest 5 optimizer rules were shown, regardless of how long they took to execute and even if they executed sufficiently fast. * Make the StatisticsFeature start after the NetworkFeature, so that any network request issues by cluster statistics gathering can rely on the networking functionality being available until shutdown. * Rework internal queues for connection and request statistics. The previous implementation allocated a lot of memory at program start for initializing fixed-sized queues for the statistics objects. The problem with using fixed-sized queues is that they will mostly require too much memory for almost all cases, but still do not protect from the queues becoming full and not being able to hold more items. Now we go with a variable length queue instead, which only requires a small amount of memory initially, and allocate more memory only when needed. Freelists for reusing statistics items are still present to avoid lots of reallocations. The change also reduces the size of the executable's .bss section by more than 10MB. * Always open a new, working connection before HTTP request-fuzzing during testing. Otherwise the fuzzing results are not 100% comparable from run to run. * Updated bundled version of zlib library to 1.2.12. * Added AQL hint "useCache" for FOR loops, to explicitly disable the usage of in-memory caches for lookups. * When profiling an AQL query via `db._profileQuery(...)` or via the web UI, the query profile will now contain the number of index entries read from in-memory caches (usable for edge indexes and indexes of type "persistent", "hash" or "skiplist") plus the number of cache misses. * The caching subsystem now provides the following 3 additional metrics: - `rocksdb_cache_active_tables`: total number of active hash tables used for caching index values. There should be 1 table per shard per index for which the in-memory cache is enabled. The number also includes temporary tables that are built when migrating existing tables to larger equivalents. - `rocksdb_cache_unused_memory`: total amount of memory used for inactive hash tables used for caching index values. Some inactive tables can be kept around after use, so they can be recycled quickly. The overall amount of inactive tables is limited, so not much memory will be used here. - `rocksdb_cache_unused_tables`: total number of inactive hash tables used for caching index values. Some inactive tables are kept around after use, so they can be recycled quickly. The overall amount of inactive tables is limited, so not much memory will be used here. * Added optional in-memory caching for index entries when doing point lookups in indexes of type "persistent", "hash" or "skiplist". The caching is turned off by default, but can be enabled when creating an index of type "persistent", "hash" or "skiplist" by setting the "cacheEnabled" flag for the index upon index creation. The cache will be initially empty, but will be populated lazily upon querying data from the index using equality lookups on all index attributes. As the cache is hash-based and unsorted, it cannot be used for full or partial range scans, for sorting, or for lookups that do not include all index attributes. The maximum size of index entries that can be stored is currently 4 MB, i.e. the cumulated size of all index entries for any index lookup value must be less than 4 MB. This limitation is there to avoid storing the index entries of "super nodes" in the cache. The maximum combined memory usage of all in-memory caches can be controlled via the existing `--cache.size` startup option, which now not only contains the maximum memory usage for edge caches, but also for index caches added here. * Added new AQL function `KEEP_RECURSIVE` to recursively keep attributes from objects/documents, as a counterpart to `UNSET_RECURSIVE`. * Added an HTTP fuzzer to arangosh that can send fuzzed requests to the server. The amount of requests sent is provided by one of the parameters of the new arangosh function `fuzzRequests()`. The optional parameters that can be supplied are: `fuzzRequests(, , )` The parameter numIterations is the amount of times the fuzzer is going to perform its random actions on the header, and seed is for the seed that is used for randomizing. The fuzzer is available only when building with failure points. * Escape each key in attribute paths of nested attributes in the query explain output for SEARCH queries that utilize the primary sort order. * Turn off sending "Server" HTTP response header on DB servers if not explicitly requested. This saves a tiny bit of traffic on each response from a DB server. * Enable range deletions in the WAL for truncate operations in the cluster, too. This can speed up truncate operations for large collections/shards. * Set max recursion depth for VelocyPack, JSON and JavaScript arrays and objects to about 200. * Updated snowball to version 2.2.0 * Fixed: Deadlock created by high load and a follower trying to get into sync. In the final synchronization phase the follower needs to temporarily block writes on the leader so we have a reliable point in time where we can prove that the data is consistent. If the leader at this point is flooded with write requests to that shard there is a chance that all worker threads only pick up those writes, which cannot make any progress until the lock is cleared. However, the process to clear the lock was on the same priority as those writes. Hence this lock clear operations could not bypass the writes. Now we moved every follow up request after the lock to HIGH lanes, which will allow them to bypass all non-internal operations. * arangosh now uses the same header the UI uses to gain higher priority on initial connection. This will increase the chance for an arangosh to connect to a server under very high load. * Bugfix: DC2DC Disjoint-SmartGraphs and Hybrid-SmartGraphs are now replicated to the follower data-center keeping their sharding intact. * Added a log message that appears upon starting arangod that shows the number of the parent process id and, if able to acknowledge it, the name of the parent process. * Parallelize applying of revision tree changes with fetching next revision tree range in incremental collection replication for collections created with ArangoDB 3.8 and higher. * Support JSON schema objects for documenting Foxx endpoints. * Internal refactoring of IndexIterator APIs. * Sorted out various geo problems: - No more special detection of "latitude-longitude rectangles" is done, since this is in conflict with the definition of polygon boundaries to be geodesics. - Linear rings in polygons are no longer automatically "normalized", so now it is possible to have polygons which cover more than half of the Earth. - Rules for polygons and multigons have been clarified and are now properly enforced for the `GEO_POLYGON` and `GEO_MULTIPOLYGON` AQL functions. - Introduced `legacyPolygon` flag for geo indexes to continue to support the old behavior in existing geo indexes. - Added lots of additional tests, thereby fixing several bugs in geo index lookup. - Use a faster algorithm for pure `GEO_CONTAINS` and `GEO_INTERSECTS` queries. * Added back the optimization for empty document update operations (i.e. update requests in which no attributes were specified to be updated), handling them in a special way without performing any writes, also excluding such special cases of operation from replication to followers. * Changed Foxx service generator output to use static variable names. * Allow early pruning (moving a FILTER condition into an IndexNode or EnumerateCollectionNode) in more cases than before. Previously, early pruning was only possible if the FILTER condition referred to exactly one variable, which had to be the FOR loop's own variable. Now, early pruning is possible with arbitrary variables that are accessible by the FOR loop. * In an attempt to make the performance of the RocksDB throttle much more consistent and predictable the default compaction slow down trigger is lowered to 128kB. * The multi-dimensional index type `zkd` now supports an optional index hint for tweaking performance by prefetching documents: ``` FOR app IN appointments OPTIONS { lookahead: 32 } FILTER @to <= app.to FILTER app.from <= @from RETURN app ``` Specifying a lookahead value greater than zero makes the index fetch more documents that are no longer in the search box, before seeking to the next lookup position. Because the seek operation is computationally expensive, probing more documents before seeking may reduce the number of seeks, if matching documents are found. Please keep in mind that it might also affect performance negatively if documents are fetched unnecessarily. * Enabled new internal graph refactored code for depth-first, breadth-first and weighted traversals by default. * Improved performance of inner joins with dynamic lookup conditions being injected from an outer loop, for indexes of type "persistent", "hash" and "skiplist". Performance improvements can be expected if the inner join is invoked a lot of times with many different values fed in by the outer loop. The performance improvements are due to some improved handling of index lookup conditions in the internals of the VelocyPack-based index. * Improve usefulness of `storedValues` together with late materialization. * Limited module resolution in arangosh to the path from which arangosh is invoked. * Changed default value of startup option `--rocksdb.cache-index-and-filter-blocks` from `false` to `true`. This makes RocksDB track all loaded index and filter blocks in the block cache, so they are accounted for in RocksDB's block cache. Also the default value for the startup option `--rocksdb.enforce-block-cache-size-limit` was flipped from `false` to `true` to make the RocksDB block cache not temporarily exceed the configured memory limit (`--rocksdb.block-cache-size`). These default value changes will make RocksDB adhere much better to the configured memory limit. This is a trade-off between memory usage stability and performance. These change may have a small negative impact on performance because if the block cache is not large enough to hold the data plus the index and filter blocks, additional disk I/O may be performed compared to previous versions. In case there is still unused RAM capacity available, it may be sensible to increase the total size of the RocksDB block cache. * Add "filtered" column to AQL query profiling output. This column shows how many documents were filtered by the node and thus provides insights into if additional indexes could help. * Reuse ExecutorExpressionContext inside IndexExecutor, so that repeated setup and teardown of expression contexts can be avoided. * Adjust internal RocksDB setting `optimize_filters_for_hits` for Documents column family, setting it from `false` to `true`. This should reduce memory and disk space requirements for the bottom-most .sst files of the documents column family. * Upgrade VelocyPack library to latest version. * Slightly improve the explain output of SingleRemoteOperationNodes. * Added more detail to the log messages that display the total time consumption and total amount of data parsed for the client tools arangodump and arangorestore. * Upgraded boost to 1.78.0. * Fixed issue #15501: Regression when using "exclusive" query option? This fixes a regression in AQL query execution when exclusive locks are used for a query and the query also uses the DOCUMENT() AQL function. In this case, when there were more concurrent requests to the underlying collection than available scheduler threads for the low priority queue, the query start successfully acquired the exclusive lock, but could not get its follow-up requests through and starve while holding the exclusive lock. * Improve performance of `db._explain()` for very large query execution plans. Higher performance is achieved by not serializing some internal data structures when serializing execution plans. Serializing internal data is now opt-in and turned off if not needed. Apart from performance, there should be no end user visible changes. * APM-24: Log messages can be displayed together with some other useful parameters, e.g., the name of the database, username, query id, and so on. There are some predefined parameters that we consider displaying, but, for the moment, only database, username and url are being displayed. The usage upon starting the server is, for example: `arangod --log.structured-param database --log.structured-param username` * Add optional "storedValues" attribute for persistent indexes. This will add the specified extra fields to the index, so that they can be used for projections, but not for lookups or sorting. Example: db..ensureIndex({ type: "persistent", fields: ["value1"], storedValues: ["value2"] }); This will index `value1` in the traditional sense, so the index can be used for looking up by `value1` or for sorting by `value1`. The index also supports projections on `value1` as usual. In addition, due to `storedValues` being used here, the index can now also supply the values for the `value2` attribute for projections. This allows covering index scans in more cases and helps to avoid making extra document lookups in the documents column family. This can have a great positive effect on index scan performance if the number of scanned index entries is large. The maximum number of attributes to store in `storedValues` is 32. * Added agency push-queue operation. * Make per-server values "numberOfCores" and "physicalMemory" available to agency to improve quality of potential future shard rebalancing algorithms. * Unify the result structure of `db._version(true)` calls for arangosh and server console. Previously such a call in the server console would return a different structure that only consisted of the `details` subobject. This is now unified so that the result structure in the server console is consistent with arangosh, but strictly speaking this is a breaking change. * Added new option `--custom-query-bindvars` to arangoexport, so queries given via option `--custom-query` can have bind variables in them. Also changed the flag names `--query` to `--custom-query` and `--query-max-runtime` to `--custom-query-max-runtime` to be like in the other client-tools. * Improve some RocksDB-related error messages during server startup. * Raised minimal macOS supported version to 10.15 (Catalina). * Remove background thread `RocksDBShaThread` for background SHA256 checksum calculation for .sst files in the Enterprise Edition. The checksums are now calculated incrementally while writing into the .sst files, and the checksum files will be stored on disk as soon as an .sst file is made durable by RocksDB. There is no more need to periodically scan the database directory and look for any additional .sst files. * Fix BTS-580: Trimmed the password field from the payload in the client requests when displaying error messages in arangorestore, because they displayed the password as plain text. * Refactored unit tests with the `grey` keyword, which is for skipping certain tests. A test file that did not perform any test, but only had a function to sleep, was removed. Two test files were renamed so they would not be skipped. * Defer intermediate commits in the middle of a multi-document (array) operation. This is to ensure that the RocksDB key locks for all participating document keys are still held while the operations are replicating via the synchronous replication. * Changed arangobench concurrency flag name from `--concurrency` to `--threads`. * APM-217: deprecate the usage of fulltext indexes. * Changed "arangosh" directory name to "client-tools", because the directory contains the code for all client tools and not just arangosh. * Updated immer to version 0.7.0. * Made the `--version` and `--version-json` commands usable in arangobackup when no positional argument (operation type) was specified. Previously, arangobackup insisted on specifying the operation type alongside the `--version` or `--version-json` commands. * Removed the following deprecated arangobench testcases: - aqltrx - aqlv8 - counttrx - deadlocktrx - multi-collection - multitrx - random-shapes - shapes - shapes-append - skiplist - stream-cursor * Renamed arangobench testcase "hash" to "persistent-index". * Add option to content-transfer encode gzip Foxx replies. * Simplify internal request compression/decompression handling code. * Added Enterprise Sharded Graphs Simulation: Now it is possible to test SmartGraphs and SatelliteGraphs on a single server instance and then to port them to a cluster with multiple servers. All existing types of SmartGraphs are eligible to this procedure: SmartGraphs themselves, Disjoint SmartGraphs, Hybrid SmartGraphs and Hybrid Disjoint SmartGraphs. One can create a graph of any of those types in the usual way, e.g., using `arangosh`, but on a single server, then dump it, start a cluster (with multiple servers) and restore the graph in the cluster. The graph and the collections will keep all properties that are kept when the graph is already created in a cluster. This feature is only available in the Enterprise Edition. * Remove unsupported `--server.default-api-compatibility` startup option. v3.9.2 (2022-06-07) ------------------- * Enterprise only: Restricted behavior of Hybrid Disjoint Smart Graphs. Within a single traversal or path query we now restrict that you can only switch between Smart and Satellite sharding once, all queries where more than one switch is (in theory) possible will be rejected. e.g: ``` FOR v IN 2 OUTBOUND @start smartToSatEdges, satToSmartEdges ``` will be rejected (we can go smart -> sat -> smart, so two switches) ``` FOR v1 IN 1 OUTBOUND @start smartToSatEdges FOR v2 IN 1 OUTBOUND v1 satToSmartEdges ``` will still be allowed, as each statement only switches once. We have decided to take these restrictions as especially for ShortestPath queries the results are not well-defined. If you have a use case where this restriction hits you, please contact us. * Fixed issue BTS-875. * Updated arangosync to v2.10.0. * Make all requests which are needed for shard resync at least medium priority to improve getting-in-sync under load. * Fix behavior when accessing a view instead of a collection by name in a REST document operation. Now return a proper error. * Fix documentation of collection's `cacheEnabled` property default. * Added startup option `--cluster.shard-synchronization-attempt-timeout` to limit the amount of time to spend in shard synchronization attempts. The default timeout value is 20 minutes. Running into the timeout will not lead to a synchronization failure, but will continue the synchronization shortly after. Setting a timeout can help to split the synchronization of large shards into smaller chunks and release snapshots and archived WAL files on the leader earlier. This change also introduces a new metric `arangodb_sync_timeouts_total` that counts the number of timed-out shard synchronization attempts. * Make followers respond to synchronous replication requests with less data. Specifically, followers will not build detailed results with _id, _key and _rev for the inserted/modified/removed documents, which would be ignored by the leader anyway. * Very verbose warning from failing to parse GEO JSON in search. Has lead to billions of log lines on deployed services. * Fixed Github issue #16279: assertion failure/crash in AQL query optimizer when permuting adjacent FOR loops that depended on each other. * Fixed a potential hang on shutdown, when there were still document operations queued. * No good reason to fatal error in agency state, when local database entries lack local timestamp (legacy). In that situation, we will record epoch begin as local time. * Fixed BTS-860. Changed ArangoSearch index recovery procedure to remove necessity to always fully recreate index if IndexCreation marker encountered. * Removed separate FlushThread (for views syncing) and merged it with the RocksDBBackgroundThread. * Fix some issues with WAL recovery for views. Previously it was possible that changes to a view/link were already recovered and persisted, but that the lower bound WAL tick was not moved forward. This could lead to already fully recovered views/links being recovered again on the next restart. * Put hotbackup requests on the HIGH priority queue to make hotbackups work under high load (BTS-865). * Allow starting with option value `--cache.size 0`, to turn off in-memory caches for indexes entirely (for performance testing or limiting memory usage). * Updated OpenSSL to 1.1.1o and OpenLDAP to 2.6.2. * Added option `--enable-revision-trees` to arangorestore, which will add the attributes `syncByRevision` and `usesRevisionsAsDocumentIds` to the collection structure if they are missing. As a consequence, these collections created by arangorestore will be able to use revision trees and a faster getting-in-sync procedure after a restart. The option defaults to `true`, meaning the attributes will be added if they are missing. If the option is set to `false`, the attributes will not be added to the collection structure. If the attributes are already present in the dump data, they will not be modified by arangorestore irrespective of the setting of this option. * Fix agency inception when one of the gossip peers responds with HTTP 503 or another unexpected error. * Make sure that newly created TTL indexes do not use index estimates, which wouldn't be used for TTL indexes anyway. * Improve log output for WAL recovery, by providing more information and making the wording more clear. * Fix: Highly unlikely race in cluster maintenance. For every shard only one operation (change attribute, change leadership) should be performed at the same time. However if two changes are detected in the same hearbeat it could lead to both operations to be executed in parallel. In most cases this is also fine, but could lead to races on the same attribute, however the race will be sorted out in the next heartbeat interval. * Fix: for the Windows build, the new Snappy version, which was introduced in 3.9, generated code that contained BMI2 instructions which where introduced with the Intel Haswell architecture. However, our target architecture for 3.9 is actually Sandy Bridge, which predates Haswell. Running the build on these older CPUs thus resulted in illegal instruction exceptions. * Increase internal transaction lock timeout on followers during cluster write operations. Although writes to the same keys on followers should be serialized by the key locks held on the leader, it is still possible that the global transaction lock striped mutex is a source of contention and that concurrent write operations time out while waiting to acquire this global mutex. The lock timeout on followers is now significantly increased to make this very unlikely. * Improve validation for variables used in the `KEEP` part of AQL COLLECT operations. Previously referring to a variable that was introduced by the COLLECT itself from out of the KEEP part triggered an internal error. The case is detected properly now and handled with a descriptive error message. * Added startup option `--rocksdb.transaction-lock-stripes` to configure the number of lock stripes to be used by RocksDB transactions. The option defaults to the number of available cores, but is bumped to a value of 16 if the number of cores is lower. * Fix deadlocked shard synchronisations when planned shard leader has not yet taken over leadership. * Added an IO heartbeat which checks that the underlying volume is writable with reasonable performance. The test is done every 15 seconds and can be explicitly switched off. New metrics to give visibility if the test fails: - `arangodb_ioheartbeat_delays_total`: total number of delayed io heartbeats - `arangodb_ioheartbeat_duration`: histogram of execution times [us] - `arangodb_ioheartbeat_failures_total`: total number of failures These metrics are only populated, if `--database.io-heartbeat` is set to `true` (which is currently the default). * Fix lock order in Agent::advanceCommitIndex for State's _logLock and Agent's _waitForCV. * BugFix (enterprise-only): (BTS-787) In a hybrid disjoint SmartGraph, having more than one relation, if you add a new vertex collection to a Smart -> Smart edge relation this vertex collection was rejected with "has to be satellite" error. Now the collection is created as a SmartVertexCollection as desired. * Resync follower shard after a follower restart immediately and not lazily. v3.9.1 (2022-04-04) ------------------- * Updated ArangoDB Starter to 0.15.4. * Improve parallelism in arangorestore in case new data format is used. * Remove error handling fetching license information to improve user experience. To display the license informationn in the UI is only informational. It disturbs the user experience to know somthing went wrong and doesn't provide any important information for the user. * Added new server option: --icu-language. Used instead of --default-language to set pure ICU collator. For example, in Sweden language("sv") lowercase letters should precede uppercase ones. You can achieve it using following options when server starts: `--icu-language sv` * No longer put document writes from replication into the audit log by default. Same with low priority authentication like internal UI requests to .html files for the UI. This solves a performance problem for shards getting in sync with audit log switched on. * Updated OpenSSL to 1.1.1n and OpenLDAP to 2.6.1. * Fixed an assertion failure which could occur when there was an error in the HTTP header, so that the message body was not actually read. * Fixed a crash which could occur when there was an error in the HTTP header parsing. * Bug-Fix: Resolve BTS-673/Issue #15107, a spliced subquery could return too few results. * Speed up initial sync (in case there is already data present) by prefetching data from leader. * Fixed ES-1078: The REST API endpoint for handling `/_api/user/${user}/config` did not work properly. The supplied data by sending a PUT request has not been stored to the correct location. The Web UI uses this endpoint to store its graph properties for storing the visualization properties. As this endpoint did not work as expected, the graph visualization properties did not get persisted as well. This is now resolved. * Fix counts and file size sum in hotbackup META files. Do no longer count directories. * Optimize further RocksDB throttle to allow for no change on any given calculation cycle. * Fix UI to only fetch license info as admin user. * Added `disableIndex` index hint for AQL FOR loops. This index hint disables the usage of any index (except geo or full text indexes) and will cause a full scan over the collection. In some circumstances a full scan can be more efficient than an index scan, for example if the index scan produces many matches (close to the number of documents in the collection) and the index is not fully covering the query. The `disableIndex` hint can be given per FOR loop in the query, e.g.: FOR doc IN collection OPTIONS { disableIndex: true } RETURN doc.value The default value of `disableIndex` is `false`. In case a different index hint is provided, `disableIndex: true` takes precendence and produces a warning about the ambiguous settings. * Added `maxProjections` hint for AQL FOR loops. This hint can be used to set the maximum number of document attributes that are taken into account for using projections. For example, in the following query, no projections will be used because the number of potential projection attributes (`value1`, value2`, `value3`) is higher than the maximum number of projection attributes set via the `maxProjections` option: FOR doc IN collection OPTIONS { maxProjections: 2 } RETURN [ doc.value1, doc.value2, doc.value3 ] The default value for `maxProjections` is `5`, which is compatible with the previous hard-coded default value. * Avoid a deadlock in agency, when Inception::gossip() acquired a mutex, then could call Agent under the mutex, and Agent could finally could call Inception::signalConditionVar(), which would try to acquire the mutex again. * Avoid multiple parallel SIGHUP requests to be handled at the same time. Now collapse multiple incoming SIGHUP requests into a single one, which can be executed race-free. * Upgraded JavaScript "i" module from 0.3.6 to 0.3.7. * Removed internal JavaScript dependencies "mocha" and "chalk". We recommend always bundling your own copy of third-party modules, even ones listed as public. * Reduce memory usage of inner joins if they were performed by the IndexExecutor with dynamic index lookup expressions that needed to be recomputed for input from the outer loop. For example, in the query ``` FOR i IN 1..1000 FOR doc IN collection FILTER doc.indexAttribute == i RETURN doc ``` the inner loop will be executed 1000 times. The IndexExecutor in the inner loop needed to rebuild the index lookup attribute from the value of `i` 1000 times as well. The memory for index lookup attributes came from the Ast's memory allocator and was not freed until the end of the query. In this query, it would mean that up to 1000 lookup values were held in memory. With larger inputs even more memory would be used. Now the memory for index lookup values is freed when a new lookup value is computed, i.e. only a single lookup value is held in memory. This drastically reduces peak memory usage for queries that use index lookups in inner loops and that get lots of different inputs from outer loops. * Harden validator for binary VelocyPack against additional types of malicious inputs. * Bugfix: DC-2-DC Disjoint-SmartGraphs and Hybrid-SmartGraphs are now replicated to the follower data-center keeping their sharding intact. * As we are now in constant stall regime, stall onset and warnings are demoted to DEBUG. * Shorten the license grace period to the documented 3 hours. * Replaced internal JS dependency xmldom with @xmldom/xmldom. * Fixed BTS-750: Fixed the issue restricted to cluster mode in which queries containing the keywords UPDATE or REPLACE together with the keyword WITH and the same key value would result in an error. For example: `UPDATE 'key1' WITH {_key: 'key1'} IN Collection` because the same key used to update was provided in the object to update the document with. * Replaced internal JS dependency ansi-html with ansi-html-community. * In an attempt to make the performance of the RocksDB throttle much more consistent and predictable the default compaction slow down trigger is lowered to 128kB. * Bug-Fix: AQL WINDOW statement if applied within a subquery could accidentally skip over some subquery results. This did only show up if the subquery fills exactly one internal batch before it is completed, so it is rather unlikely. * Fix null pointer access when using WINDOW operation with a COUNT/LENGTH aggregate function without any arguments. * Reintroduce shard synchronization cancellation check that was disabled before. * Fixed BTS-621 Fixed rare case of segfault in cluster during database recovery if DBServer is in upgrade mode in the same time. * Fixed PRESUPP-445: Foxx queues: Some Jobs are never run in case of multiple Coordinators. * Fixed a race detected with chaos tests, where a db server could have momentarily lost leadership, just when it was about to drop a follower to a shard. * Fixed issue #15501: Regression when using "exclusive" query option? This fixes a regression in AQL query execution when exclusive locks are used for a query and the query also uses the DOCUMENT() AQL function. In this case, when there were more concurrent requests to the underlying collection than available scheduler threads for the low priority queue, the query start successfully acquired the exclusive lock, but could not get its follow-up requests through and starve while holding the exclusive lock. * Disable optimizer rule "optimize-cluster-single-document-operations" when a collection is accessed in exclusive mode, because the optimized query would use a slightly different mode of locking then. v3.9.0 (2022-02-07) ------------------- * Convert v3.9.0-rc.1 into v3.9.0 (GA). v3.9.0-rc.1 (2022-02-03) ------------------------ * Fix potential access to dangling reference in cancellation of shard synchronization. * Fixed BTS-740 (no released version infected) fixed Smart<->Sat SmartEdgeCollections determining the shard in SingleRemoteModification Nodes was incorrect. E.g. this could be triggered, by viewing the details of an edge in the UI. Only alpha/beta of 3.9.0 contained this bug * Fixed BTS-729 (no released version infected): Some conditions in a Hybrid Smart Graph could led to wrong shard location calculation and therefore to wrong graph query results. Only alpha/beta of 3.9.0 contained this bug. * Fixed minDepth handling of weighted Traversals. When using a minDepth of 3, also paths of length 2 have been returned, on all locally executed variants (SingleServer, OneShard, DisjointSmart). * Fixed BTS-728 (no released version infected) fixed: for DisjointSmartGraphs, that include a satellite vertex collection, valid disjoint path were not always followed, if one of the satellites has a connection to two (or more) vertices that have different shardValues that by chance are routed to the same shard. * Fix creation of satellite graphs with a `numberOfShards` value != 1. * Fixed BTS-712: Collation analyzer now always produces valid UTF-8 sequence. * Updated Enterprise license behavior: now there will be a 48 hour period for a new deployment and upgrade decision to provide the license. After that period, the read-only mode will be enforced. Upgrade procedure for a deployment without license will not take upgrade period into account for the read-only mode will enforcement. * Fixed a bug that hotbackup upload could miss files (fixes BTS-734). * BTS-590: When creating a new database in Web UI the value of the write concern has to be smaller or equal to the replication factor. Otherwise an error message will be displayed and no database will be created. * Fixed potentially undefined behavior for exit code of arangodump. * Ignore signals such as SIGPIPE in client tools. * Validate that selected `writeConcern` value is less or equal to the selected `replicationFactor` value when creating new databases. * Fixed issue #15476: FATAL {crash} occurs on a simple query. v3.9.0-beta.1 (2022-01-06) -------------------------- * Fixed ES-1025: fixed a performance regression caused by different hash calculation for primitive types like `uint64_t` and new the `Identifier` wrapper and derived types. * BTS-707: rename "hardened" option value for `--server.support-info-api` startup option to "admin". * APM-292: Added new AQL function SHARD_ID. * Extend timeouts for caching collection counts and index selectivity estimates on coordinators from 15s/90s to 180s. This change will cause less requests to be made from coordinators to DB servers to refresh info about collection counts and index estimates as part of AQL queries. The cached info is used in cluster query execution plans only and is not required to be fully up-to-date. * Improved performance in replication dump protocol by inserting arrays of documents instead of one document at a time and also not retrieving the document revision field when not needed. * APM-78: Added startup security option `--foxx.allow-install-from-remote` to allow installing Foxx apps from remote URLs other than Github. The option is turned off by default. * Fixed BTS-693: Sort-limit rule now always ensures proper LIMIT node placement to avoid possible invalid results in the fullCount data * Updated OpenSSL to 1.1.1m and OpenLDAP to 2.6.0. * Updated arangosync to 2.7.0. * Fixed PRESUPP-439: In arangoimport, for CSV and TSV files, it could happen that a buffer containing only the header would be sent to the server, and also batches would contain the documents equivalent to the csv rows in them, but not the header, which should be sent together with the documents. * Changed various default values for RocksDB to tune operations for different typical scenarios like gp2 type volumes and gp3 type volumes and locally attached SSDs with RAID0: - `--rocksdb.level0-slowdown-trigger` has been decreased from 20 to 16 - `--rocksdb.level0-stop-trigger` has been increased from 36 to 256 - `--rocksdb.max-background-jobs` has been increased to the number of cores and is no longer limited to 8 - `--rocksdb.enabled-pipelined-write` is now `true` by default instead of `false` - `--rocksdb.throttle-frequency` has been decreased from 60000ms down to 1000ms per iteration, which makes the RocksDB throttle react much quicker - `--rocksdb.pending-compactions-slowdown-trigger` has been decreased from 64 GB down to 8 GB - `--rocksdb.pending-compactions-stop-trigger` has been decreased from 256 GB down to 16 GB - `--rocksdb.throttle-slots` has been increased from 63 to 120 - `--rocksdb.encryption-hardware-acceleration` is now `true` by default, which helps performance and should not create any problems, since we require sandybridge anyway. Combined, these changes help ArangoDB/RocksDB to react quicker to a backlog of background jobs and thus to prevent catastrophic stops which abort data ingestion or lead to cluster internal timeouts. * Adjust default value for startup option `--rocksdb.max-subcompactions` from 1 to 2. This allows compactions jobs to be broken up into disjoint ranges which can be processed in parallel. * Added startup options to adjust previously hard-coded parameters for RocksDB's behavior: - `--rocksdb.pending-compactions-bytes-slowdown-trigger` controls RocksDB's setting `soft_pending_compaction_bytes_limit`, which controls how many pending compaction bytes RocksDB tolerates before it slows down writes. - `--rocksdb.pending-compactions-bytes-stop-trigger` controls RocksDB's setting `hard_pending_compaction_bytes_limit`, which controls how many pending compaction bytes RocksDB tolerates before it stops writes entirely. - `--rocksdb.throttle-lower-bound-bps`, which controls a lower bound for the bandwidth restriction on RocksDB writes the throttle imposes. * Allow initial, full dump shard synchronization to abort prematurely if it turns out that the follower was removed from the plan as a follower (e.g. if there are enough other in-sync followers). * Set the limit for ArangoSearch segment size to 256MB during recovery to avoid OOM kill in rare cases. * Cancel ongoing RocksDB compactions on server shutdown. * Updated Enterprise license behavior: now there will be a one hour period for a new deployment to provide the license. After that period, the read-only mode will be enforced. * Added startup options to adjust previously hard-coded parameters for the RocksDB throttle: - `--rocksdb.throttle-frequency`: frequency for write-throttle calculations (in milliseconds, default value is 60000, i.e. 60 seconds). - `--rocksdb.throttle-slots`: number of historic measures to use for throttle value calculation (default value is 63). - `--rocksdb.throttle-scaling-factor`: adaptiveness scaling factor for write- throttle calculations (default value is 17). There is normally no need to change this value. - `--rocksdb.throttle-max-write-rate`: maximum write rate enforced by the throttle (in bytes per second, default value is 0, meaning "unlimited"). The actual write rate will be the minimum of this value and the value the throttle calculation produces. - `--rocksdb.throttle-slow-down-writes-trigger`: number of level 0 files whose payload is not considered in throttle calculations when penalizing the presence of L0 files. There is normally no need to change this value. All these options will only have an effect if `--rocksdb.throttle` is enabled (which is the default). The configuration options introduced here use the previously hard-coded settings as their default values, so there should not be a change in behavior if the options are not adjusted. * Improve visibility in case of potential data corruption between primary index and actual document store in documents column family. * Fixed BTS-611: In some cases AQL queries, in particular in a cluster, reported the wrong fullCount when the optimizer rule(s)`late-document-materialization` and/or `sort-limit` were active. * Fix BTS-535: TakeoverShardLeadership waits properly for Current data in ClusterInfo. This avoids a fake warning and fake test failure. * Fix potential read inconsistency for single document operations. When reading a single document that is concurrently being updated or replaced, the read operation could erroneously return a "document not found" error although the document actually existed. This only happened for single document operations, i.e., no transactions or AQL queries. * APM-256: make arangoexport escape potential formulae in CSV exports. This addresses a potential security issue when exporting specially crafted documents to CSV, opening the CSV file in MS Excel or OpenOffice and then clicking links in any of the tainted cells. This change also adds a new option `--escape-csv-formulae` to toggle the escaping behavior for potential formulae values. The option is turned on by default. * Second step of hotbackup transfer job cleanup. Now locks are also cleaned up as well as old, seemingly unfinished jobs. * Fix GitHub issue #15084. Fixed a potential use-after-free on Windows for queries that used the NeighborsEnumerator (though other PathEnumerators might have been affected as well). * BTS-624: The `move-calculations-up` optimization rule is now also applied to subqueries, when they don't have dependencies on the outer nodes, don't have modification nodes and don't read their own writes. This fixed the execution of a query without the splicing-subqueries option being faster than the execution of a query with this option (after version 3.8, this option cannot be switched off). * Added the following metrics for revision trees: - `arangodb_revision_tree_hibernations_total`: number of times a revision tree was compressed in RAM and set to hibernation - `arangodb_revision_tree_resurrections_total`: number of times a revision tree was resurrected from hibernation and uncompressed in RAM - `arangodb_revision_tree_memory_usage`: total memory usage (in bytes) by all active revision trees * Added metric `rocksdb_wal_sequence` to track the current tip of the WAL's sequence number. v3.9.0-alpha.1 (2021-11-30) --------------------------- * Cleanup and properly fail hotbackup upload and download jobs if a dbserver fails or is restarted during the transfer. This gets rid of upload and download blockages in these circumstances. * Make `db..figures(true)` operate on the same snapshot when counting the number of documents in the documents column family and the indexes. This ensures consistency for the results of a single figures result. * Upgraded bundled version of RocksDB to 6.27. * Improved sync protocol to commit after each chunk and get rid of potentially dangerous NO_INDEXING optimization. * Removed an invalid assertion that could be triggered during chaos testing in maintainer mode. * Simplify the tagging of EnumerateCollectionNodes and IndexNodes with the "read-own-writes" flag. Previously the tagging only happened after all query optimizations were completed, making the tag unavailable to the optimizer. Now the tag is set early on, so it is accessible by the query optimizer. * APM-187: The "Rebalance Shards" button now is displayed in a new tab, and it is displayed for any database in cluster mode. There is also a new flag for arangod, `--cluster.max-number-of-move-shards` (default = 10), which limits the amount of move shards operations each time the button is clicked to rebalance shards. When the button is clicked, the number of move shards operations scheduled is shown, or that no operation was scheduled if the flag `--cluster.max-number-of-move-shards` has a value of 0. * Make the `--version` and `--version-json` commands usable in arangobackup when no positional argument (operation type) was specified. Previously, arangobackup insisted on specifying the operation type alongside the `--version` or `--version-json` commands. * Fixed an issue in old incremental sync protocol with document keys that contained special characters (`%`). These keys could be send unencoded in the incremental sync protocol, leading to wrong key ranges being transferred between leader and follower, and thus causing follow-up errors and preventing getting in sync. * APM-209: Histogram displaying is now switched off by default. For displaying it, the new flag `histogram.generate` must be set to true. Its default value is false for compatibility with other versions and also for complying with the histogram not being displayed by default. If this flag is not set to true, but other histogram flags are addressed, e.g. `--histogram.interval-size 500`, everything will still run normally, but a warning message will be displayed saying that the histogram is switched off and setting this flag would not be of use. When the flag is set to true, the histogram is displayed before the summary in the output. * In the shards overview the list of servers to move the leader shard to, now also contains the current followers. This means that from now on also active follower servers can be nominated as the leading server for that specific shard. * Extend Windows minidumps with memory regions referenced from CPU registers or the stack to provide more contextual information in case of crashes. * Fix issues during rolling upgrades from 3.8.0 to 3.8.x (x >= 1) and from 3.7.x (x <= 12) to 3.8.3. The problem was that older versions did not handle following term ids that are sent from newer versions during synchronous replication operations. * Increase default stack size on Windows from 1MB to 4MB. This should allow execution of larger queries without overflowing the stack. * Make background calculation of SHA hashes for RocksDB .sst files less intrusive. The previous implementation frequently iterated over all files in the database directory to check if it needed to ad-hoc calculate the SHA hashes for .sst files it previously missed. The procedure it used was to iterate over all files in the database directory and check if there were matching pairs of .sst files and .sha files. This was expensive, because a full directory iteration was performed and a lot of temporary strings were created for filenames and used in comparisons. This was especially expensive for larger deployments with lots of .sst files. The expensive iteration of files in the directory is now happening less frequently, and will not be as expensive as before if it runs. * Close a potential gap during shard synchronization when moving from the initial sync step to the WAL tailing step. In this small gap the leader could purge some of the WAL files that would be required by the following WAL tailing step. This was possible because at the end of the initial sync step, the snapshot on the leader is released, and there is a small window of time before the follower will issue its first WAL tailing request. * Improve Shards overview in web UI: the number of currently syncing shards is now displayed per collection. Additionally, shards on failed servers are now displayed in a different color. * Fixed BTS-637: Slow SynchronizeShard jobs which need to copy data could block quick SynchronizeShard jobs which have the data and only need to resync. * DEVSUP-899: Fixed Subquery execution in a very rare case a subquery, nested in another subquery, was not executed, which is fixed now. Technical details: If we have two subqueries: `Outer` and `Nested` the Outer will define the input for Nested. And Outer has the pattern: 1 input, subqueryDone, 1 input, subqueryDone [...] and our internal batching did cut a batch like this: [<...>, input (A)] | [subqueryDone, input (B), subqueryDone, <...>] than Nested on input (B) was not executed. As soon as we have more than 1 input per Outer, or a different cutting position, all was good. * When enabling the cluster supervision maintenance mode via the web UI, there is now the possibility to select a duration for the maintenance mode. Previous versions of ArangoDB always enabled the maintenance mode for one hour, without allowing any choice here. * Stop calling unnecessary `/_api/wal/open-transactions` REST API before starting the continuous synchronization in active fail and single server replication. This request is unnecessary with the RocksDB storage engine. * Fixed potential undefined behavior in edge cache during cache migration tasks. There was a short window of time in which an already freed Table could be used by concurrently running edge lookups. * BTS-623: The audit log messages, when written, were not showing the log level of the message, as in the example: `2021-10-21T02:28:42Z | hostname | audit-authentication | n/a | _system | 127.0.0.1:52490 | n/a | credentials missing | /_admin/aardvark/favicon.ico` With the new flag `--audit.display-log-level`, the level of the audit log message can be displayed in the log text. When set to true, this behavior is expected, as in the example: `2021-10-21T02:28:42Z | DEBUG | hostname | audit-authentication | n/a | _system | 127.0.0.1:52490 | n/a | credentials missing | /_admin/aardvark/favicon.ico` The default value for the flag is false for compatibility with former versions. When this flag is not used, it is considered to have the default behavior (that is, set to false). * Fixed SEARCH-261: Fix possible race between file creation and directory cleaner (ArangoSearch). * Fixed SEARCH-260: Fix invalid sorting order of stored features in presence of primary sort (ArangoSearch). * Change error message for queries that use too much memory from "resource limit exceeded" to "query would use more memory than allowed". * When using Indexes within traversals (e.g. [_from, date]) and filter based on a function (e.g. FILTER path.edges[0].date <= DATE_ADD(@now, 5, "day")) this function was passed through to the index. The index cannot evaluate this function and returned incorrect results. Now all functions are evaluted before looking into the index. (Fixes BTS-407) * Old license mechanism for docker containers removed. * arangorestore: Fix the order (regarding distributeShardsLike) in which collections are being created during restore, which could result in an error and make manual intervention necessary. * Single server license output checking fixed. * Updated ArangoDB Starter to 0.15.3. * Fix caching of collection counts and index selectivity estimates in cluster. The cache values expired too early in previous versions, making the cache ineffective. * Add better error message for replication request failures in case requests are retried. * Make background statistics gathering more efficient by avoiding one AQL query every 10 seconds that fetched the most recent stats entry. Instead, buffer the entry in value after we have written it. Also spread out the statistics calls by different servers more randomly, so that request spikes are avoided for cluster with many coordinators that used to run their statistics queries at about the same time when the instances were started simultaneously. * Fix a potential overwhelm situation on DB servers that can lead to no further tasks being pulled from a DB servers queue even though there would still be processing capacity and idle threads available. * Fixed compilation and linking when using glibc 2.34. * Fuerte: don't fall back to identity encoding in case of unknown encoding. * Fixed ES-881: Fixed LDAP global options. This needs to use the first active provider, not just the first provider and it should be globally disabled. * Web UI: Fixes the loading of map tiles which are being used to display the query output based on a world map when using SSL encryption. This lead to not displaying some world map tiles correctly (OASIS-590). * Web UI - Added missing HTML escaping inside the file upload plugin used in the section of deploying a new Foxx application when uploading a zip file. * Now, arangoimport supports merging of attributes. When importing data from a file into a collection, a document attribute can be comprised of merging attributes from the file into it, with separators and other literal strings. The new document attribute will result in the concatenation of the literal strings, the values of the attributes and the separators, as in the example: arangoimport --merge-attributes fullName=[firstName]:[lastName] * Do not use an edge index for range queries, i.e. with the comparison operators `>`, `>=`, `<` or `<=`, but only for equality lookups using the `==` and `IN` comparison operators. The edge index is not fully ordered, so while using it for range queries may produce _some_ documents, it is possible that other documents from the range would be skipped. * Do not rename the arangod process to "arangod [shutting down]" during the server shutdown. The renaming can cause issues with tools that look for the exact process name "arangod". * Remove internal AQL query option `readCompleteInput` that controled if all input for a modification operation (UPDATE / REPLACE / REMOVE) are read into memory first. This was a necessity with the MMFiles storage engine in cases when a query read from a collection and wrote into it in the same query afterwards. With the RocksDB engine and its snapshots, we never need to read the entire input into memory first. * Fix windows installer PATH manipulation issue by replacing the NSIS plugin (BTS-176). * Fixed counting of all read transaction as aborted. Added a new metric to count read transactions. * Fixed potential issues with revision trees and document counters getting out of sync with the underlying collection data. * Fix race in RocksDB throttle listener, when it was getting started lazily during server shutdown. * Extended the Views web UI by letting it capture View properties that are immutable once created. * Fixed BTS-602 by not starting license feature is upgrade mode. * APM-173: Now, arangobench, arangodump and arangorestore support multiple coordinators, so the flag `--server.endpoint` can be used multiple times, as in the example below: arangobench \ --server.endpoint tcp://[::1]::8529 \ --server.endpoint tcp://[::1]::8530 \ --server.endpoint tcp://[::1]::8531 This does not compromise the use of the other client tools, which preserve the behavior of having one coordinator. * The server now has two flags to control the escaping control and Unicode characters in the log. The flag `--log.escape` is now deprecated and, instead, the new flags `--log.escape-control-chars` and `--log.escape-unicode-chars` should be used. - `--log.escape-control-chars`: this flag applies to the control characters, which have hex code below `\x20`, and also the character DEL, with hex code of `\x7f`. When its value is set to false, the control character will be retained, and its actual value will be displayed when it is a visible character, or a space ` ` character will be displayed if it is not a visible character. The same will happen to `DEL` character (code `\xF7`), even though it is not a control character, because it is not visible. For example, control character `\n` is visible, so a `\n` will be displayed in the log, and control character `BEL` is not visible, so a space ` ` would be displayed. When its value is set to true, the hex code for the character is displayed, for example, `BEL` character would be displayed as its hex code, `\x07`. The default value for this flag is `true` for compatibility with previous versions. - `--log.escape-unicode-chars`: when its value is set to false, the unicode character will be retained, and its actual value will be displayed. For example, `犬` will be displayed as `犬`. When its value is set to true, the character is escaped, and the hex code for the character is displayed. For example, `犬` would be displayed as its hex code, `\u72AC`. The default value for this flag is `false` for compatibility with previous versions. * Fixed BTS-582: ArangoDB client EXE package for Windows has incorrect metadata. * Fixed BTS-575: Windows EXE installer doesn't replace service during upgrade in silent (non-UI) mode. * APM-121: allow the UPSERT query to have indexHint as an extra parameter for OPTIONS. It will be used as a hint by the inner FOR loop that is performed as part of the UPSERT query, and would help in cases such as UPSERT not picking the best index automatically for lookup. * Fix issue #14819: Query: AQL: missing variable # for node #... location RestCursorHandler.cpp. * Added enterprise licensing support including (only for Enterprise version): - additional API endpoint `_admin/license(GET/PUT)?force=true ` - arangosh functions: `setLicense()`, `getLicense()` - new error codes and metrics support * Fix issue #14807: Fix crash during optimization of certain AQL queries during the remove-collect-variables optimizer rule, when a COLLECT node without output variables (this includes RETURN DISTINCT) occurred in the plan. * Update iresearch library to the upstream. Fixed TSan/ASan detected issues. * Added new ArangoSearch analyzer type 'collation'. * Add basic overload control to arangod. This change adds the `x-arango-queue-time-seconds` header to all responses sent by arangod. This header contains the most recent request dequeuing time (in seconds) as tracked by the scheduler. This value can be used by client applications and drivers to detect server overload and react on it. The new startup option `--http.return-queue-time-header` can be set to `false` to suppress these headers in responses sent by arangod. In addition, client applications and drivers can optionally augment their requests sent to arangod with a header of the same name. If set, the value of the header should contain the maximum queuing time (in seconds) that the client is willing to accept. If the header is set in an incoming request, arangod will compare the current dequeuing time from its scheduler with the maximum queue time value contained in the request. If the current dequeuing time exceeds the value set in the header, arangod will reject the request and return HTTP 412 (precondition failed) with the new error code 21004 (queue time violated). There is also a new metric `arangodb_scheduler_queue_time_violations_total` that is increased whenever a request is dropped because of the requested queue time not satisfiable. * Fixed a bug for array indexes on update of documents (BTS-548). * Prevent some possible deadlocks under high load regarding transactions and document operations, and also improve performance slightly. * Hide help text fragment about VST connection strings in client tools that do not support VST. * Added REST API endpoint `/_admin/debug/failat/all` to retrieve the list of currently enabled failure points. This API is available only if failure testing is enabled, but not in production. * APM-60: optionally allow special characters and Unicode characters in database names. This feature allows toggling the naming convention for database names from the previous strict mode, which only allowed selected ASCII characters in database names, to an extended, more relaxed mode. The extended mode allows additional ASCII characters in database names as well as non-ASCII UTF-8 characters. The extended mode can be enabled by setting the new startup option `--database.extended-names-databases` to true. It is turned off by default and requires an explicit opt-in, simply because some drivers and client applications may not be ready for it yet. The arangod server, the ArangoDB web interface and the following bundled client tools are prepared and ready for using the extended database names: - arangobench - arangodump - arangoexport - arangoimport - arangorestore - arangosh More tools and the drivers shipped by ArangoDB will be added to the list in the future. Please note that the extended names for databases should not be turned on during upgrades from previous versions, but only once the upgrade has been completed successfully. In addition, the extended names should not be used in environments that require extracting data into a previous version of ArangoDB, or when database dumps may be restored into a previous version of ArangoDB. This is because older versions may not be able to handle the extended database names. Finally, it should not be turned on in environments in which drivers are in use that haven't been prepared to work with the extended naming convention. Warning: turning on the `--database.extended-names-databases` option for a deployment requires it to stay enabled permanently, i.e. it can be changed from `false` to `true` but not back. When enabling it, it is also required to do this consistently on all coordinators and DB servers. The extended names for databases will be enabled by default in one of the future releases of ArangoDB, once enough drivers and other client tools have had the chance to adapt. Naming conventions for collections, views, analyzers, and document keys (`_key` values) are not affected by this feature and will remain as in previous versions of ArangoDB. * Prevent stealing of values from AQL const value registers. This fixes an issue for queries that produce constant results (known at query compile time) when he queries are executed directly on a DB server in a cluster (which is not supported, but may happen for troubleshooting). * Fixed BTS-562: reduce-extraction-to-projection optimization returns null for one attribute if nested attributes are named the same. * Add `--datatype` startup option to arangoimport, in order to hard-code the datatype (null/boolean/number/string) for certain attributes in the CSV/TSV import. For example, given the following input file: key,price,weight,fk 123456,200,5,585852 864924,120,10,9998242 9949,70,11.5,499494 6939926,2130,5,96962612 When invoking arangoimport with the startup options --datatype key=string --datatype price=number --datatype weight=number --datatype fk=string it will turn the numeric-looking values in "key" into strings (so that they can be used in the `_key` attribute), but treat the attributes "price" and "weight" as numbers. The values in attribute "fk" finally will be treated as strings again (potentially because they are used for linking to other "_key" values). * Avoid the acquisition of a recursive read lock on server shutdown, which could in theory lead to shutdown hangs at least if a concurrent thread is trying to modify the list of collections (very unlikely and never observed until now). * Fixed display of unicode characters in Windows console. * Fixed issue BTS-531 "Error happens during EXE package installation if non-ASCII characters are present in target path". * Fix active failover, so that the new host actually has working Foxx services (BTS-558). * Fixed issue #14720: Bulk import ignores onDuplicate in 3.8.0. The "onDuplicate" attribute was ignored by the `/_api/import` REST API when not specifying the "type" URL parameter. * Updated OpenSSL to 1.1.1l and OpenLDAP to 2.4.59. * APM-70: allow PRUNE condition to be stored in a variable. This feature allows the PRUNE condition to be stored in a variable, and this variable can be used as a condition for some other statement, such as FILTER. * Allow startup of arangod with an existing database directory that was missing the ZkdIndex column family. * Truncate must not trigger intermediate commits while in a streaming transaction, because that would be against the assumption that streaming transactions never do intermediate commits. * Added ArangoSearch condition optimization: STARTS_WITH is merged with LEVENSHTEIN_MATCH if used in the same AND node and field name and prefix matches. * Hybrid (Disjoint) SmartGraphs (Enterprise Edition): SmartGraphs have been extended with a new option to create Hybrid SmartGraphs. Hybrid SmartGraphs are capable of using SatelliteCollections within their graph definition. You can now select some VertexCollections to be satellites, and therefore available on all DBServers. The SmartGraph can make use of those to collections to increase the traversal performance by larger local components. * Added multidimensional indexes which can be used to efficiently intersect multiple range queries. They are currently limited to IEEE-754 double values. Given documents of the form {x: 12.9, y: -284.0, z: 0.02} one can define a multidimensional index using the new type 'zkd' on the fields ["x", "y", "z"]. The AQL optimizer will then consider this index when doing queries on multiple ranges, for example: FOR p IN points FILTER x0 <= p.x && p.x <= x1 FILTER y0 <= p.y && p.y <= y1 FILTER z0 <= p.z && p.z <= z1 RETURN p The index implements the relation <=, == and >= natively. Strict relations are emulated using post filtering. Ranges can be unbounded on one or both sides. * No runtime limits for shard move and server cleanout jobs, instead possibility to cancel them. * Fix cluster-internal network protocol to HTTP/1 for now. Any other protocol selected via the startup option `--network.protocol` will automatically be switched to HTTP/1. The startup option `--network.protocol` is now deprecated and hidden by default. It will be removed in a future version of arangod. The rationale for this change is to move towards a single protocol for cluster-internal communication instead of 3 different ones. * Disable RTTI when compiling Snappy. RTTI used to be disabled previously, up until some Merkle tree improvement PR was merged about one month ago, which turned on RTTI for compiling Snappy. * (EE only) Bug-fix: If you created a ArangoSearch view on satellite collections only and then join with a collection only having a single shard the cluster-one-shard-rule was falsely applied and could lead to empty view results. The Rule will now detect the situation properly, and not trigger. * (EE only) If you have a query using only satellite collections, now the cluster-one-shard-rule can be applied to improve query performance. * (Enterprise Edition only): added query option `forceOneShardAttributeValue` to explicitly set a shard key value that will be used during query snippet distribution to limit the query to a specific server in the cluster. This query option can be used in complex queries in case the query optimizer cannot automatically detect that the query can be limited to only a single server (e.g. in a disjoint smart graph case). When the option is set to the correct shard key value, the query will be limited to the target server determined by the shard key value. It thus requires that all collections in the query use the same distribution (i.e. `distributeShardsLike` attribute via disjoint SmartGraphs). Limiting the query to a single DB server is a performance optimization and may make complex queries run a lot faster because of the reduced setup and teardown costs and the reduced cluster-internal traffic during query execution. If the option is set incorrectly, i.e. to a wrong shard key value, then the query may be shipped to a wrong DB server and may not return results (i.e. empty result set). It is thus the caller's responsibility to set the `forceOneShardAttributeValue` correctly or not use it. The `forceOneShardAttributeValue` option will only honor string values. All other values as well as the empty string will be ignored and treated as if the option is not set. If the option is set and the query satisfies the requirements for using the option, the query's execution plan will contain the "cluster-one-shard" optimizer rule. * SEARCH-238: Improved SortNodes placement optimization in cluster so late materialization could cover more cases * Fix some memory leaks after adding optimization rule for AqlAnalyzer. * Fix internal iterator states after intermediate commits in write transactions. Iterators could point to invalid data after an intermediate commit, producing undefined behavior. * Fix read-own-write behavior in different scenarios: - in some cases writes performed by an AQL query could be observed within the same query. This was not intended and is fixed now. - AQL queries in streaming transactions could observe their own writes in even more cases, which could potentially result in an endless loop when the query iterates over the same collection that it is inserting documents into. - UPSERT did not find documents inserted by a previous iteration if the subquery relied on a non-unique secondary index. - disabled intermediate commits for queries with UPSERTs, because intermediate commits can invalidate the internal read-own-write iterator required by UPSERT. Previously, UPSERTs that triggered intermediate commits could have produced unexpected results (e.g., previous inserts that have been committed might not be visible) or even crashes. To achieve the correct read-own-write behavior in streaming transactions, we sometimes have to copy the internal WriteBatch from the underlying RocksDB transaction. In particular, the copy is created whenever an AQL query with modification operations (INSERT/REMOVE/UPDATE/UPSERT/REPLACE) is executed in the streaming transaction. If there have not been any other modifications so far (queries/document operations), then the WriteBatch is empty and creating the copy is essentially a no-op. However, if the transaction already contains a lot of modifications, creating the WriteBatch copy might incur some overhead that can now lead to decreased performance. * Fix rare case of invalid data that could be inserted into the ArangoSearch index if several clients concurrently insert data and use custom analyzer with non-string return type. * Fix a rare shutdown race in RocksDBShaCalculatorThread. * Added "Analyzers" view to web UI to let manage ArangoSearch analyzers creation. * Add pseudo log topic "all" to set the log levels for all log topics at once. For example, this can be used when starting a server with trace or debug logging enabled for all log topics, e.g. `--log.level all=debug` `--log.level all=trace` This is very coarse and should only be used for such use cases. * Change the default value for the `--threads` startup parameter of the following client tools from previously 2 to the maximum of 2 and the number of available CPU cores: - arangodump - arangoimport - arangorestore * Remove old fixPrototypeChain agency migration, which was introduced in 3.2 and is no longer necessary. This will make it impossible to upgrade directly from a version < 3.2 to a version >= 3.9, provided one has a chain of `distributeShardsLike` collections. * Added metrics for the number of errors and warnings logged: - `arangodb_logger_warnings_total`: total number of warnings (WARN messages) logged since server start - `arangodb_logger_errors_total`: total number of errors (ERR messages) logged since server start * Added REST API `/_admin/support-info` to retrieve deployment information. As this API may reveal sensitive data about the deployment, it can only be accessed from inside the system database. In addition, there is a policy control startup option `--server.support-info-api` that determines if and to whom the API is made available. This option can have the following values: - `disabled`: support info API is disabled. - `jwt`: support info API can only be accessed via superuser JWT. - `hardened`: if `--server.harden` is set, the support info API can only be accessed via superuser JWT. Otherwise it can be accessed by admin users only. - `public`: everyone with access to `_system` database can access the support info API. * Send a keystroke to arangod's stdin when a shutdown command is received via the REST API `/_admin/shutdown` and the server is started with the `--console` argument. The keystroke will exit the blocking read loop that is waiting on console input and that otherwise blocks the shutdown. The implementation is based on ioctl and is thus only present on Linux and macOS. * Some AQL queries erroneously reported the "access after data-modification" error for queries in which there was a read attempt from a collection _before_ a data-modification operation. Such access is legal and should not trigger said error anymore. Accessing a collection _after_ in a query a data-modification in the same query is still disallowed. * Make AQL modification operations in a cluster asynchronous. This allows to free the thread for other work until both the write and synchronous replication are complete. * Fixed: /_api/transaction/begin called on edge collections of disjoint SmartGraphs falsely returned CollectionNotFound errors. * Bugfix: In more complex queries there was a code-path where a (Disjoint-)Smart graph access was not properly optimized. * Add ReplicatedLogs column family. * Add optimization rule for AqlAnalyzer. * Change optimization level for debug builds back to `-O0` (from `-Og`) because `-Og` seems to cause debuggability issues in some environments. * Automatically extend web UI sessions while they are still active. The web UI can now call a backend route to renew its JWT, so there will not be any rude logouts in the middle of an active session. Active web UI sessions (here: sessions with user activity within the last 90 minutes) will automatically renew their JWT if they get close to the JWT expiry date. * Reduce memory usage for in-memory revision trees. Previously, a revision tree instance for a non-empty collection/shard was using 4 MB of memory when uncompressed. Trees that were unused for a while were compressed on the fly to use less memory, and later uncompressed again when needed. Now the uncompressed in-memory version of the revision tree will dynamically allocate memory as needed. This allows the initial version of the trees to get away with just 64 KB of memory. Memory usage will grow lazily when more parts of the trees get populated. The compression of unused in-memory tree data is still in place. * Refactored arangobench: - Updated testcases to show description of them when beginning execution - Fixed testcase histogram with time measures when batch size > 0 - Integrated testcases with Velocypack for simplification - Deprecated some testcases - Internal changes for performance optimization * Add 3 AQL functions: COSINE_SIMILARITY, L1_DISTANCE and L2_DISTANCE. * Honor the value of startup option `--rocksdb.sync-interval` on Windows, too. Previously, the value was ignored and WAL syncing on Windows was using a different code paths than on the other supported platforms. Now syncing is unified across all platforms, and they all call RocksDB's `SyncWAL()`. * APM-132: Clean up collection statuses. Removes collection statuses "new born", "loading", "unloading" and "unloaded". These statuses were last relevant with the MMFiles storage engine, when it was important to differentiate which collections are present in main memory and which aren't. With the RocksDB storage engine, all that was automatically handled anyway, and the statuses were not important anymore. The change removes the "Load" and "Unload" buttons for collections from the web interface. All collections in the web interface will be marked as "loaded" permanently. This change also obsoletes the `load()` and `unload()` calls for collections as well as their HTTP API equivalents. The APIs will remain in place for now but are changed to no-ops. They will removed eventually in a future version of ArangoDB. This will be announced separately. * Reduce default value for max-nodes-per-callstack to 200 for macOS, because on macOS worker threads have a stack size of only 512kb. * Slightly increase internal AQL query and transaction timeout on DB servers from 3 to 5 minutes. Previously, queries and transactions on DB servers could expire quicker, which led to spurious "query ID not found" or "transaction ID not found" errors on DB servers for multi-server queries/transactions with unbalanced access patterns for the different participating DB servers. The timeouts on coordinators remain unchanged, so any queries/transactions that are abandoned will be aborted there, which will also be propagated to DB servers. In addition, if a participating server in an AQL query becomes unavailable, the coordinator is now notified of that and will terminate the query more eagerly. * Add hard-coded complexity limits for AQL queries, in order to prevent programmatically generated large queries from causing trouble (too deep recursion, enormous memory usage, long query optimization and distribution passes etc.). This change introduces 2 limits: - a recursion limit for AQL query expressions. An expression can now be up to 500 levels deep. An example expression is `1 + 2 + 3 + 4`, which is 3 levels deep `1 + (2 + (3 + 4))`. The expression recursion is limited to 500 levels. - a limit for the number of execution nodes in the initial query execution plan. The number of execution nodes is limited to 4,000. * Remove _msg/please-upgrade handler. * Adapt various places related to handling of execution plans non-recursive in order to avoid stack overflows. This allows us now to execute much larger queries. * Fix locking of AQL queries write queries on DB servers. * APM-112: invalid use of OPTIONS in AQL queries will now raise a warning in the query. The feature is useful to detect misspelled attribute names in OPTIONS, e.g. INSERT ... INTO collection OPTIONS { overwrightMode: 'ignore' } /* should be 'overwriteMode' */ It is also useful to detect the usage of valid OPTIONS attribute names that are used for a wrong query part, e.g. FOR doc IN collection FILTER doc.value == 1234 INSERT doc INTO other OPTIONS { indexHint: 'myIndex' } /* should be used above for FOR */ In case a wrong option attribute is used, a warning with code 1575 will be raised. By default, warnings are reported but do not lead to the query being aborted. This can be toggled by the startup option `--query.fail-on-warnings` or the per-query runtime option `failOnWarnings`. * Added new command line-option `--version-json`. This will return the version information as json object. * Fix ArangoAgency::version(), which always returned an empty string instead of the agency's correctly reported version. This also fixes the agency version in the startup log messages of the cluster. * Fix potential memleak in Pregel conductor garbage collection. * Added garbage collection for finished and failed Pregel conductors. Previously, Pregel executions that finished successfully or unsuccessfully remained in memory until being explicitly canceled. This prevented a cleanup of abandoned jobs. Such jobs are now automatically cleaned about 10 minutes after finalization. The time-to-live values can be overriden per Pregel job by passing a "ttl" value. * Revive startup parameter `--server.session-timeout` to control the timeout for web interface sessions and other sessions that are based on JWTs created by the `/_open/auth` API. This PR also changes the default session timeout for web interface sessions to one hour. Older versions of ArangoDB had longer session timeouts. * Removed redirects from /_admin/cluster* to /_admin/cluster/*. Adjusted internal requests to use the new url. * Fix potential stack overflow when executing large queries. This is achieved by splitting the callstack and moving part of the execution to a separate thread. The number of execution nodes after which such a callstack split should be performed can be configured via the query option `maxNodesPerCallstack` and the command line option `--query.max-nodes-per-callstack`; the default is 250. * Fixed invalid shard synchronization for documents not added via INSERT with `overwriteMode` set to `ignore`. In this case, if a document with the given key already exists, it is not changed on the leader (i.e. no write happens on the leader). However, a write was replicated to the follower, which was wrong. This write is now suppressed, which can only make such insert operations faster. * Web UI: Disables the hover tooltip within the statistics view of the memory consumption chart. * Add 3 AQL functions: DECAY_GAUSS, DECAY_EXP and DECAY_LINEAR. * Implemented an optimization for Traversals. If you apply a POST filter on the vertex and/or edge result this filter will now be applied during the traversal to avoid generating the full output for AQL. This will have positive effect if you filter on the vertex/edge but return the path, this way the system does only need to produce a path that is allowed to be passed through. E.g. FOR v,e,p IN 10 OUTBOUND @start GRAPH "myGraph" FILTER v.isRelevant == true RETURN p can now be optimized, and the traversal statement will only produce paths where the last vertex has `isRelevant == true`. * Fix BTS-446: When finding a not yet fully initialized agency, do not immediately fatal exit. Keep trying for (very generous) 5 minutes. * Reduced the agency store public members, for simpler support long-term. * Added a number of tests for the Agency Store public members. * Updated bundled version of Snappy library to 1.1.9. * Introduce a new internal error code for cases where a call cannot succeed because the server startup phase is still in progress. This error will be mapped to the HTTP status code 503 (service unavailable). One example where this can happen is when trying to authenticate a request, but the _users collection is not yet available in the cluster. * Fix DEVSUP-749: Fix potential deadlock when executing concurrent view/link DDL operations and index DDL operations on the same collection. * Fixed issue #14122: when the optimizer rule "inline-subqueries" is applied, it may rename some variables in the query. The variable renaming was however not carried out for traversal PRUNE conditions, so the PRUNE conditions could still refer to obsolete variables, which would make the query fail with errors such as Query: AQL: missing variable ... for node ... while planning registers * Fixed the error response if the HTTP version is not 1.0 or 1.1 and if the Content-Length is too large (> 1 GB). * Add a connection cache for internal replication requests. * Improve legibility of size values (by adding KB, MB, GB, TB suffixes) to output generated by client tools. * Timely updates of rebootId / cluster membership of DB servers and coordinators in ClusterInfo. Fixes BTS-368 detected in chaos tests. * Guarded access only to ActionBase::_result. * Fixed proper return value in sendRequestRetry if server is shutting down. * Fixed internal issue #798: In rare case when remove request completely cleans just consolidated segment commit could be cancelled and documents removed from collection may be left dangling in the ArangoSearch index. Also fixes ES-810 and BTS-279. * Retry if an ex-leader can no longer drop a follower because it is no longer leading. * Fixed a small problem in fuerte which could lead to an assertion failure. * Upgrade jemalloc version to latest stable dev. * Fixed issue BTS-373: ASan detected possible heap-buffer-overflow at arangodb::transaction::V8Context::exitV8Context(). * Allow to specify a fail-over LDAP server. Instead of "--ldap.OPTION" you need to specify "--ldap2.OPTION". Authentication / Authorization will first check the primary LDAP server. If this server cannot authenticate a user, it will try the secondary one. It is possible to specify a file containing all users that the primary (or secondary) LDAP server is handling by specifying the option "--ldap.responsible-for". This file must contain the usernames line-by-line. * Make the time-to-live (TTL) value of a streaming cursor only count after the response has been sent to the client. * Improve performance of batch CRUD operations (insert, update, replace, remove) if some of the documents in the batch run into write-write conflicts. Rolling back partial operations in case of a failure is very expensive because it requires rebuilding RocksDB write batches for the transaction from scratch. Rebuilding write batches takes time proportional to the number of operations in the batch, and for larger batches the cost can be prohibitive. Now we are not rolling back write batches in some situations when this is not required, so that in many cases running into a conflict does not have that high overhead. There can still be issues when conflicts happen for index entries, but a lot of previously problematic cases should now work better. * Allow AQL variable names starting with an underscore, as stated in the docs. * Fix crashes during arangorestore operations due to usage of wrong pointer value for updating user permissions. * Added option `--query-max-runtime` to arangoexport, in order to control maximum query runtime. * Fix BTS-340: AQL expressions similar to `x < 3 || x` are no longer erroneously be reduced to `x < 3` by the optimizer rule remove-redundant-or. * Changed default value of arangodump's `--envelope` option from `true` to `false`. This allows using higher parallelism in arangorestore when restoring large collection dumps. As a side-effect, this will also decrease the size of dumps taken with arangodump, and should slightly improve dump speed. * Improve parallelism capabilities of arangorestore. arangorestore can now dispatch restoring data chunks of a collection to idle background threads, so that multiple restore requests can be in flight for the same collection concurrently. This can improve restore speed in situations when there are idle threads left (number of threads can be configured via arangorestore's `--threads` option) and the dump file for the collection is large. The improved parallelism is only used when restoring dumps that are in the non-enveloped format. This format has been introduced with ArangoDB 3.8. The reason is that dumps in the non-enveloped format only contain the raw documents, which can be restored independent of each other, i.e. in any order. However, the enveloped format may contain documents and remove operations, which need to be restored in the original order. * Fix BTS-374: thread race between ArangoSearch link unloading and storage engine WAL flushing. * Fix thread race between ArangoSearch link unloading and storage engine WAL flushing. * Add value of `_key` to more insert/update/replace/remove error messages so it is easier to figure out which document caused unique constraint violations and/or write-write conflict during a multi-document write operation. * Don't display obsoleted startup options and sections in `--help` and `--help-.` commands. Also rename "global" to "general" options. * Removed assertion for success of a RocksDB function. Throw a proper exception instead. * Experimentally switch to wyhash (from xxhash) for velocypack. This is an experiment in devel to check if it produces any observable speedups. * Remove deprecated HTTP REST API `/_api/export`. This API was deprecated in a previous version because it was not supported in clusters and was also covered completely by streaming AQL queries for the RocksDB storage engine. * Added enterprise-build-repository and oskar-build-repository to `--version` as `enterprise-build-repository` and `oskar-build-repository`. * Clean up replication code and remove a 3.2-compatibility mode that was only useful when replicating from a leader < ArangoDB version 3.3. * Obsolete option `--database.old-system-collections`. This option has no meaning in ArangoDB 3.9, as old system collections will not be created anymore in this version. The option was deprecated in 3.8 and announced to be obsoleted. * Upgrade velocypack to latest, C++17-only version. * Make arangovpack more powerful, by supporting different input and output formats (json and vpack, plain or hex-encoded). The arangovpack options `--json` and `--pretty` have been removed and have been replaced with separate options for specifying the input and output types: - `--input-type` ("json", "json-hex", "vpack", "vpack-hex") - `--output-type` ("json", "json-pretty", "vpack", "vpack-hex") The previous option `--print-non-json` has been replaced with the option `--fail-on-non-json` which makes arangovpack fail when trying to emit non-JSON types to JSON output. * Remove obsolete API endpoint /_admin/repair/distributeShardsLike`. This API was intended to correct some bad state introduced before 3.2.12 or 3.3.4, respectively. It had to be invoked manually by callers and there was never any driver support for it. * Remove now-unused SubqueryExecutor. This is an internal change only and should not have any effect on queries, as from 3.8 onwards only spliced subqueries should be used in query execution plans and during query execution. * Switched to GCC 10 as the default compiler and use Sandy Bridge as the default required architecture (Linux, macOS binaries). * Removed obsolete metrics in new v2 metric API. Those metrics' values were identical to the sum value of histograms. * Fix potentially undefined behavior when creating a CalculationTransactionContext for an ArangoSearch analyzer. An uninitialized struct member was passed as an argument to its base class. This potentially had no observable effects, but should be fixed. v3.8.1 (2021-08-27) ------------------- * Reduce internal priority of AQL execution. This prevents possible deadlocks with modification operations in a cluster and replicationFactor >= 2, and can also improve responsiveness under high load of AQL queries. * Updated arangosync to 2.6.0. * Added protocol specific metrics: histogram about request body size, total number of HTTP/2 connections and total number of VST connections. * Fix a potential multi-threading issue in index creation on coordinators, when an agency callback was triggered at the same time the method `ensureIndexCoordinatorInner` was left. * Append physical compaction of log collection to every Raft log compaction (BTS-542). * Preselect "create index in background" option when creating indexes in the web UI. The "create index in background" option can be less intrusive because it allows other write operations on the collection to proceed. * Do not block a scheduler thread on the coordinator while an index is being created. Instead, start a background thread for the actual index fill-up work. The original thread can then be relinquished until the index is completely filled or index creation has failed. The default index creation timeout on coordinators has also been extended from 1 hour to 4 days, but it is still configurable via the startup parameter `--cluster.index-create-timeout` in case this is necessary. * Fixed: getResponsibleShard call on disjoint Smart Graphs if you asked for the responsible shard on a disjoint edge collection where the _from and _to differ (invalid), the server would respond with "DATASOURCE_NOT_FOUND". This is now fixed to "BAD_PARAMETER" to emphasize that the collection is fine but the input is invalid. * Fixed: _api/transaction/begin called on edge collections of disjoint SmartGraphs falsely returned CollectionNotFound errors. * Bug-Fix: In more complex queries there was a code-path where a (Disjoint-) Smart graph access was not properly optimized. * Fix wrong assertion in fuerte and move it to where the TLA+ model says i should be. This fixes a unit test failure occurring on newer Macs with a certain clang version. * When creating Pregel memory-mapped files, create them with O_TMPFILE attribute on Linux so that files are guaranteed to vanish even if a process dies. * Improve log messages for Pregel runs by giving them more context. * Fixed issue BTS-536 "Upgrading without rest-server is aborted by error". Now stating `--server.rest-server false` does not require the additional `--console` argument for upgrading a server. * Fixed issue #14592: IS_NULL(@x) isn't recognized as a constant expression. * Fixed issue BTS-539 "Unsynchronized query kill while it's being finalized in another thread was uncovered through `test-kill.js` of `communication_ssl` suite". Fixed possible (but unlikely) crash when killing an AQL query. * Fixed various problems in GEO_INTERSECTS: wrong results, not implemented cases and numerically unstable behaviour. In particular, the case of the intersection of two polygons in which one is an S2LngLatRect is fixed (BTS-475). * Fixed ES-867 and ES-922: removed eslint from NPM packages descriptions and updated netmask package to non-vulnerable version. * Web UI: Fixes the loading of map tiles which are being used to display the query output based on a world map when using SSL encryption. This lead to not displaying some world map tiles correctly (OASIS-590). * Timely update of database server list on health check fixes BTS-505. * Updated JavaScript dependencies, including breaking changes to non-public modules. We recommend always bundling your own copy of third-party modules, even ones listed as public. - accepts: 1.3.5 -> 1.3.7 - ansi_up: 4.0.3 -> 5.0.1 - content-type: (added) -> 1.0.4 - error-stack-parser: 2.0.2 -> 2.0.6 - highlight.js: 9.15.6 -> 10.7.3 - http-errors: 1.7.2 -> 1.8.0 - iconv-lite: 0.4.24 -> 0.6.3 - js-yaml: 3.13.1 -> 3.14.1 - lodash: 4.17.13 -> 4.17.21 - marked: 0.6.2 -> removed - mime-types: 2.1.22 -> 2.1.31 - mocha: 6.1.3 -> 6.2.3 - netmask: 1.0.6 -> 2.0.2 - qs: 6.7.0 -> 6.10.1 - range-parser: 1.2.0 -> 1.2.1 - semver: 6.0.0 -> 7.3.5 - sinon: 1.17.6 -> 1.17.7 - timezone: 1.0.22 -> 1.0.23 - type-is: 1.6.16 -> 1.6.18 - underscore: 1.9.1 -> 1.13.1 - xmldom: 0.1.27 -> 0.6.0 * Updated ArangoDB Starter to 0.15.1. * Fix BTS-453: Download of a HotBackup from remote source doesn't work on macOS. * Web UI: Fixes a logical error which occured after re-visiting the logs view which lead to not displaying the logs view and its entries correctly (BTS-507). * Raised the versions of the node modules `node-sass` and `sass-loader` to be able to build the Web UI with Node v16+. * Suppress repeated warnings when settings LDAP options which turn out to be unsupported on the target system. This avoids logging the same warnings repeatedly. * Make `--javascript.copy-installation` also copy the `node_modules` sub directory. This is required so we have a full copy of the JavaScript dependencies and not one that excludes some infrequently changed modules. In addition, file copying now intentionally excludes .map files as they are not needed. * Fixed a bug, where Coordinators handled plan changes for databases in heartbeat thread in wrong order. Databases could be listed, but not used. * Include K_SHORTEST_PATHS and SHORTEST_PATH execution nodes in AQL query memory usage accounting. The memory used by these execution node types was previously not tracked against the configured query memory limit (BTS-411). * Lower log level to warning, when take over shard leadership finds an agency Current entry is missing the server taking over. * Fixed BTS-408: treat positive or negative signed numbers as constants immediately during AQL query parsing. Previously, a value of `-1` was parsed initially as `unary minus(value(1))`, which was not treated in the same way as a constant value `value(-1)`. The value was later optimized to just `value(-1)`, but this only happened during constant-folding after parsing. Any operations that referred to the unfolded values during parsing thus did not treat such values as constants. * Fix startup issues with encryption-at-rest enabled when there were empty (0 byte) RocksDB WAL files present. Such empty files caused RocksDB to abort the startup, reporting corruption. However, empty WAL files are possible in case of server crashes etc. Now, if a WAL file is completely empty, there will be no attempt to read the encryption meta data from it, so the startup succeeds (BTS-392). * Fixes a bug in the maintenance's error-handling code. A shard error would result in log messages like ``` WARNING [ceb1a] {maintenance} caught exception in Maintenance shards error reporting: Expecting Object ERROR [c9a75] {maintenance} Error reporting in current: Expecting Object ``` and also prevent the maintenance from reporting the current state to the agency, which in turn can prevent cluster-wide progress of various actions. * APM-107: Added metric "rocksdb_read_only" to determine whether RocksDB is currently in read-only mode due to a background error. The metric will have a value of "1" if RocksDB is in read-only mode and "0" if RocksDB is in normal operations mode. If the metric value is "1" it means all writes into RocksDB will fail, so inspecting the logfiles and acting on the actual error situation is required. * Fix numeric overflow in AQL WINDOW node cost estimation if the number of preceding rows was set to `unbounded`. * Added a retry loop for arangorestore during the initial connection phase. The number of retries defaults to 3 and can be configured using --initial-connect-retries (BTS-491). * Add following term ids, which prevents old synchronous replication requests to be accepted after a follower was dropped and has gotten in sync again. This makes the chaos tests which delay synchronous replication requests more reliable and prevent inconsistent shard replicas under bad network conditions. * Enable process metrics on agent instances by default. Previously, some metrics (including the metrics starting with `arangodb_process` prefix) were not returned by agent instances. * Add prefix parameter to LEVENSHTEIN_MATCH function in ArangoSearch (DEVSUPP-753). * Bug-fix: Pregel WCC algorithm could yield incorrect results if a part of the connected component was only attached via OUTBOUND edges. The underlying algorithm is now modified to properly retain INBOUND edges for the runtime of the execution. This uses more RAM for the algorithm but guarantees correctness. * Fix serialization of query shutdown error code when sending it to DB servers. The error code is numeric, but it was sent to the `/_api/aql/finish` API as a string. This led to the DB servers always assuming the default error code TRI_ERROR_INTERAL (error code 4). This was not a problem for normal query operations, but it could have led to warnings being logged stating "please contact ArangoDB support". Now the actual error code is using during query shutdown. * Fix display of running and slow queries in web UI when there are multiple coordinators. Previously, the display order of queries was undefined, which could lead to queries from one coordinator being display on top once and then the queries from another. That made using this UI harder than necessary. Now queries are sorted for display, according to their query IDs. * Fixed an issue in index selection, when the selectivty estimate of another prefix index was used without checking if the other index covered the FILTER condition. For example, given the following indexes: - index 1: ["e", "a", "b", "c"] - index 2: ["e", "a", "b"] - index 3: ["d", "e", "f", "g"] and the FILTER condition `d == 1 && e == 2 && f == 3`, then the best index to pick would be index 3. However, the optimizer may have picked index 1 here. All indexes are valid candidates for this FILTER condition, but none of the indexes covered all attributes of the FILTER condition. So the index selectivity estimates were (correctly) not used directly to determine the best index. The actual bug happened when comparing the usefulness of the candidate indexes, when figuring out that even though the selectivity estimate for index 1 could not be used, but that there existed a prefix index of index 1 (index 2). The selecivity estimate of this index was taken _without_ checking that prefix index actually satisfied the FILTER condition fully. The prefix index' selectivity estimate must only be used if it fully satisfies the FILTER condition, which was not the case here. * Fixed DEVSUP-799: unique vertex getter may point to invalid memory after being resetted, resulting in undefined behavior for traversals returning unique vertices from inner FOR loops. * Improve usability of hidden options: `--help` mentions that these exist and how to display them. * Fixed ES-863: reloading of users within the Cluster. If a Coordinator is asked to reload its users (e.g. by the UserManager in Foxx, it is also possible to do via API, but this is internal and on purpose not documented, so unlikely that it is used), in concurrency with user management updates there is a chance that the reload is not correctly performed on this coordinator. It may have missed the last update locally, causing one user to have an older state. It will be fixed on the next modification of any other users/permissions. Unfortunately this bug can cascade and when hit again, the coordinator can now be off by two updates. In DC2DC this situation is more likely to happen on the target datacenter, causing this datacenter to have other users/permissions than the source one. * Fix BTS-446: When finding a not yet fully initialized agency, do not immediately fatal exit. Keep trying for (very generous) 5 minutes. * Only build actually used subattributes of traversal paths, i.e. "vertices", "edges" or "weights". If any of the paths subcomponents is not used, the optimizer will try to save these components from being built for each result item. * Backport bugfix from upstream rocksdb repository for calculating the free disk space for the database directory. Before the bugfix, rocksdb could overestimate the amount of free space when the arangod process was run as non-privileged users. * Fixed a problem with active failover, where a failover could take 5 mins because the follower was caught in a bad state during replication. This fixes BTS-425. * Add soft coordinator shutdown: This is a new option `soft=true` for the DELETE /_admin/shutdown API. Has only meaning for coordinators, otherwise ignored. A number of things are allowed to finish but no new things are allowed when in soft coordinator shutdown: - AQL cursors - transactions - asynchronous operations - Pregel runs Once all of the ongoing operations of these have finished and all requests on the low priority queue have been executed, the coordinator shuts down the normal way. This is supposed to make a coordinator restart less intrusive for clients. * Fix BTS-398: Cannot force index hint for primary index if FILTER has multiple OR conditions that require different indexes. * Bug-fix (macOs): in macOs there is an upper bound for descriptors defined by the system, which is independend of the settings in `ulimit -n`. If the hard limit is set above this upper bound value ArangoDB tries to raise the soft limit to the hard limit on boot. This will fail due to the system limit. This could cause ArangoDB to not start, asking you to lower the minimum of required file descriptors. The system set upper bound is now honored and the soft limit will be set to either hard limit or system limit whichever is lower. * Implemented APM-86: add query option `fillBlockCache` to control population of RocksDB block cache with data read by the query. The default value for this per-query option is `true`, which mimics the previous behavior. Setting the option to off allows not storing data in RocksDB's block cache for queries that are known to read only semi-relevant or unimportant data. v3.8.0 (2021-07-14) ------------------- * Always remove blocker object for revision trees in case of replication failures. * Fix invalid assertion for insert/removal buffers positioning and internals of `hasBlockerUpTo` function. * Updated ArangoDB Starter to 0.15.0-1. * Updated arangosync to 2.4.0. * For cluster AQL queries, let the coordinator determine the query id to be used on DB servers. This allows the coordinator to roll back AQL query setup requests via the query id. Previously, the DB servers each generated a local query id and returned it to the coordinator, who would then keep track of them for later use. The problem with this was that if an AQL query setup request timed out, the coordinator had no way to roll it back. In addition, if setting up a query takes long on a DB server so that the coordinator sends a rollback request, there are some measures in place for the unlikely case in which the rollback request overtakes the setup request. In this case, the rollback request will not find a query yet, but will register a tombstone for it. Once the query gets registered by the delayed request, it will (correctly) fail because of the tombstone. * Removed a special case for empty document update operations (i.e. update requests in which no attributes were specified to be updated) were handled in a special way without performing any writes. The problem was that such updates did not update the local state, but could have been replicated to followers. This special empty update case is now removed and update operations that do not update any attributes are treated as normal write operations both locally and in the replication. * Fix partial cleanup of internal write batches for multi-document operations of which one or multiple failed. The previous implementation had an unreleased performance optimization that wouldn't clean up the write batch completely. That could have led to a wrong sequence of events being accumulated in the write batch, which may have confused the WAL tailing API later. This bug was only present in 3.8 RCs. * Fix some occurrences in which Merkle trees could silently apply the same change multiple times, which led to data drift between the Merkle tree and the underlying collection's data. This bug was only present in 3.8 but no earlier versions. * On a failure during synchronous replication, do not remove the failed follower from the list of known servers in the transaction. If we do, we would not be able to send the commit/abort to the follower later. However, we still need to send the commit/abort to the follower at transaction end, because the follower may be responsible for _other_ shards as well. This change also removes dangling transactions that could stay around on followers until they expired after the transaction idle timeout (180 seconds), and that could prevent a follower from getting back in sync during this period. * Added more context to "dropping follower" messages, so it is easier to analyze what exactly went wrong. * Fixed invalid shard synchronization for documents not added via INSERT with `overwriteMode` set to `ignore`. In this case, if a document with the given key already exists, it is not changed on the leader (i.e. no write happens on the leader). However, a write was replicated to the follower, which was wrong. This write is now suppressed, which can only make such insert operations faster. * Fix DEVSUP-753: now it is safe to call visit on exhausted disjunction iterator. * Slightly improve specific warning messages for better readability. * Fix URL request parsing in case data is handed in in small chunks. Previously the URL could be cut off if the chunk size was smaller than the URL size. * Fix BTS-430: Added missing explain output about indexes for SHORTEST_PATH, K_SHORTEST_PATHS and K_PATHS. * Added check to utils/generateAllMetricsDocumentation.py to check that the file name and the value of the name attribute are the same in the metrics documentation snippets. Correct a few such names. v3.8.0-rc.2 (2021-06-07) ------------------------ * Updated arangosync to 2.3.0. * Fix BTS-456, BTS-457: Make geo intersection between point and rectangle symmetrical. * Fix BTS-450: RandomGenerator caught assertion during a value generation within `dump_maskings` testsuite. Ensure correct conversion between 64 and 32bit. * Added check for data type compatibility between members of pipeline ArangoSearch analyzer. v3.8.0-rc.1 (2021-05-26) ------------------------ * Fix BTS-442: a query with fullCount on a sharded collection hangs indefinitely when LIMIT is less then number of available documents. * Removed unused documentation snippets (non-Rest DocuBlocks) as well as the documentation about the long deprecated features Simple Queries and JavaScript-based graph traversal. Also removed the descriptions of the JS API methods `collection.range()`, `collection.closedRange()`, `cursor.setBatchSize()` and `cursor.getBatchSize()`. All the functionality is superseded by AQL. * Fixed ES-881: ensure that LDAP options for async, referrals and restart set the off value correctly. Otherwise, this can result in an "operations error". * Improve Merkle tree memory usage and allow left-growth of trees, too. This can help with insertions of arbitrarily old data. * Added metric `arangodb_sync_rebuilds_total` to track the full rebuild of a shard follower after too many subsequent shard synchronization failures. This metric should always have a value of 0. Everything else indicates a serious problem. * Fixed BTS-422: SingleRemoteModification in AQL behaves different. This disables the optimizer rule `optimize-cluster-single-document-operations` for array inputs, e.g. INSERT [...] INTO collection REMOVE [...] IN collection For the cases, the optimization is not pulled off, and the normal insert/ update/replace/remove behavior is executed, which will fail because of an array being used as input. * Fix BTS-409: return error 1948 when a negative edge was detected during or was used as default weight in a SHORTEST_PATH or a K_SHOTRTEST_PAHS traversal. * Fixed issue BTS-424: fix invalid input row handling in WINDOW execution. * Added 2 options to allow HTTP redirection customization for root ("/") call of HTTP API: `--http.permanently-redirect-root`: if true (default), use a permanent redirection (use HTTP 301 code), if false fall back to temporary redirection (use HTTP 302 code); `--http.redirect-root-to`: redirect of root URL to a specified path (redirects to "/_admin/aardvark/index.html" if not set (default)). * Fixed DEVSUP-764 (SEARCH-7): inconsistent BM25 scoring for LEVENSHTEIN_MATCH function. * Rename two metrics with previously Prometheus-incompatible names: - `arangodb_aql_global_query_memory_limit_reached` was renamed to `arangodb_aql_global_query_memory_limit_reached_total` - `arangodb_aql_local_query_memory_limit_reached` was renamed to `arangodb_aql_local_query_memory_limit_reached_total` These metrics were introduced in 3.8 so there is no migration for these metrics. * Return error 1948 when a negative edge was detected during a weighted traversal or was used as default weight. * Fixes BTS-417. In some cases an index did not consider both bounds (lower and upper) for a close range scan if both bounds are expressed using the same operator, e.g., `FILTER doc.beginDate >= lb AND ub >= doc.beginDate`. * Fix various issues related to the new WINDOW operation (see BTS-402) - Improved explain output for ISO 8601 duration strings and fixed missing week component. - Improved validation of input data and error messages. - Prevent FILTERs from being moved beyond a WINDOW. * Fixes BTS-416. During shutdown, a Shard leader wrongly reported that it could not drop a shard follower instead of correctly indicating the shutdown as the reason. * Fixes pregel lifetime management. Previously shutting down the server while a pregel job was still running could result in a segfault or a shutdown hanger. * Improve error reporting for Merkle tree operations and improve memory usage for unused trees by hibernating them. In addition, add some backoff to shard synchronization in case there are repeated sync failures for the same shard. * Fixed BTS-403: Hot restores must also clear relevant `Current` keys. The overriding of the `Plan` entries needs to be reflected in `Current` to avoid conflicts in maintenance jobs. * Log a proper message if an unexpected state is encountered when taking over shard leadership. In addition, make the change to the internal followerinfo state atomic so that it cannot be semi-changed. * Fixed two bugs in fuerte with HTTP/2 and VST connections. One could lead to ordered timeouts not being honoured. The other could lead to an ordered callback be called multiple times. * Improve "Shards" view in web UI so that the shards of individual collections can be expanded and collapsed without affecting the display of any other shards. Also added a "Toggle all" button the web UI to expand/collapse the shards for all collections. * Improve exception safety for maintenance thread and shard unlock operations. * Fixed issue #14122: when the optimizer rule "inline-subqueries" is applied, it may rename some variables in the query. The variable renaming was however not carried out for traversal PRUNE conditions, so the PRUNE conditions could still refer to obsolete variables, which would make the query fail with errors such as Query: AQL: missing variable ... for node ... while planning registers * Improve performance of batch CRUD operations (insert, update, replace, remove) if some of the documents in the batch run into write-write conflicts. Rolling back partial operations in case of a failure is very expensive because it requires rebuilding RocksDB write batches for the transaction from scratch. Rebuilding write batches takes time proportional to the number of operations in the batch, and for larger batches the cost can be prohibitive. Now we are not rolling back write batches in some situations when this is not required, so that in many cases running into a conflict does not have that high overhead. There can still be issues when conflicts happen for index entries, but a lot of previously problematic cases should now work better. This change also reduces the RocksDB-internal lock timeout for writing to keys locked by another transaction from 1s to 1ms. This will mean that operations that ran into a write-write conflict may fail quicker than before, and not wait and retry to acquire the locked key(s). * Fix response when isBuilding could not be removed from newly created collection, when agency precondition fails. This can happen, when own rebootId increment has triggered plan entry to be removed. * Fixed issue BTS-354: Assertion related to getCollection. * Fix DEVSUP-749: Fix potential deadlock when executing concurrent view/link DDL operations and index DDL operations on the same collection. * When writing to starting shard leader respond with specific 503. Fixes BTS-390. * Fixed a use after free bug in the connection pool. * Show peak memory usage in AQL query profiling output. * Fixed various issues (mainly data races) reported by ThreadSanitizer. * Fixed bug in error reporting when a database create did not work, which lead to a busy loop reporting this error to the agency. * Fixed the error response if the HTTP version is not 1.0 or 1.1 and if the Content-Length is too large (> 1 GB). * Guarded access only to ActionBase::_result. * Updated arangosync to 2.2.0. * Fixed proper return value in sendRequestRetry if server is shutting down. * Fixed internal issue #798: In rare case when remove request completely cleans just consolidated segment commit could be cancelled and documents removed from collection may be left dangling in the ArangoSearch index. Also fixes ES-810 and BTS-279. * Fixed a small problem in fuerte which could lead to an assertion failure. * Retry if an ex-leader can no longer drop a follower because it is no longer leading. * Fixed issue BTS-373: ASan detected possible heap-buffer-overflow at arangodb::transaction::V8Context::exitV8Context(). * Fix a potential buffer overflow in RestReplicationHandler. * Make the time-to-live (TTL) value of a streaming cursor only count after the response has been sent to the client. * Added option `--query-max-runtime` to arangoexport, in order to control maximum query runtime. * Fix BTS-340: AQL expressions similar to `x < 3 || x` are no longer erroneously be reduced to `x < 3` by the optimizer rule remove-redundant-or. * Change arangosh client behavior: - *_RAW methods will never add a `body` to HEAD responses - *_RAW methods will now always return velocypack-typed responses in Buffers - `--server.force-json` will now be applied as default, overrideable by user code v3.8.0-beta.1 (2021-04-20) -------------------------- * Fix BTS-374: thread race between ArangoSearch link unloading and storage engine WAL flushing. * Improve parallelism capabilities of arangorestore. arangorestore can now dispatch restoring data chunks of a collection to idle background threads, so that multiple restore requests can be in flight for the same collection concurrently. This can improve restore speed in situations when there are idle threads left (number of threads can be configured via arangorestore's `--threads` option) and the dump file for the collection is large. The improved parallelism is only used when restoring dumps that are in the non-enveloped format. This format has been introduced with ArangoDB 3.8. The reason is that dumps in the non-enveloped format only contain the raw documents, which can be restored independent of each other, i.e. in any order. However, the enveloped format may contain documents and remove operations, which need to be restored in the original order. * Fix crashes during arangorestore operations due to usage of wrong pointer value for updating user permissions. * Fixed BTS-360 and ES-826: sporadic ArangoSearch error `Invalid RL encoding in 'dense_fixed_offset_column_key'`. * Add HTTP REST API endpoint POST `/_api/cursor/` as a drop-in replacement for PUT `/_api/cursor/`. The POST API is functionally equivalent to the existing PUT API. The benefit of using the POST API is that HTTP POST requests will not be considered as idempotent, so proxies may not retry them if they fail. This was the case with the existing PUT API, as HTTP PUT requests can be considered idempotent according to the HTTP specification. The POST API is not used internally by ArangoDB's own requests in this version. This means that compatibility to older versions of ArangoDB that do not provide the new API is ensured. * Timely updates of rebootId / cluster membership of DB servers and coordinators in ClusterInfo. Fixes BTS-368 detected in chaos tests. * Fix cluster internal retry behavior for network communications. In particular retry on 421 (leader refuses operation). This leads to the cluster letting less internal errors out to clients. * Fixed CPPCHECK warning or added suppression. * Web UI - Added missing HTML escaping inside the file upload plugin used in the section of deploying a new Foxx application when uploading a zip file. * Allow to specify a fail-over LDAP server. Instead of "--ldap.OPTION" you need to specify "--ldap2.OPTION". Authentication / Authorization will first check the primary LDAP server. If this server cannot authenticate a user, it will try the secondary one. It is possible to specify a file containing all users that the primary (or secondary) LDAP server is handling by specifying the option "--ldap.responsible-for". This file must contain the usernames line-by-line. * Fix BTS-352: removed assertion for success of a RocksDB function and throw a proper exception instead. * Added option `--query.require-with` to make AQL in single server mode also require `WITH` clauses where the cluster would need them. The option is turned off by default, but can be turned on in single servers to remove this behavior difference between single servers and clusters, making later a transition from single server to cluster easier. * Fixed a problem in document batch operations, where errors from one shard were reported multiple times, if the shard is completely off line. * Fixed issue #13169: arangoimport tsv conversion of bools and null, although switched off by `--convert false`. Importing unquoted `null`, `false` and `true` literals from delimited files get imported as strings now if `convert` is explicitly turned off. It previously affected unquoted numbers only. * Web UI: Highlight binary and hexadecimal integer literals in AQL queries. * Prevent arangod from terminating with "terminate called without an active exception" (SIGABRT) in case an out-of-memory exception occurs during creating an ASIO socket connection. * Micro improvements for Pregel job API and documentation: - Added a few useful attributes to Pregel HTTP API docs. - Added "parallelism" attribute to the result of Pregel job status responses, so that the effective parallelism is reported back. - Make sure "computationTime" in Pregel job status response does not underflow in case of errors. * Fix BTS-350, BTS-358: Fixed potential startup errors due to global replication applier being started before end of database recovery procedure. Also fixed potential shutdown errors due to global replication applier being shut down in parallel to a concurrent shut down attempt. * Fix BTS-357: Fix processing of analyzer with return type by TOKENS function. * Fix BTS-346: Improved handling of AQL query kill command in unlikely places, before the query starts to execute and after the query is done but the result is still being written. Now the cleanup of queries works more reliably. This unreliable kill time windows were very short and unlikely to hit, although if one was hit transactions were not aborted, and collection locks could be lingering until query timeout. * Updated ArangoDB Starter to 0.15.0. * Fix BTS-357: Fix processing of analyzer with return type by TOKENS function. * Added error handling for figures command in cluster. Previously errors returned by shards were ignored when aggregating the individual responses. * Remove CMake control variable `UNCONDITIONALLY_BUILD_LOG_MESSAGES`. * Fix undefined behavior in dynarray constructor when running into an out-of-memory exception during construction. In arangod, this can only happen during metrics objects construction at program start. * Added option `--headers-file` to arangoimport, to optionally read CSV/TSV headers from a separate file. * Fixed issue BTS-353: memleak when running into an out-of-memory situation while repurposing an existing AqlItemBlock. * Change metrics' internal `low()` and `high()` methods so that they return by value, not by reference. * Fix logging of urls when using `--log.level requests=debug`. There was an issue since v3.7.7 with the wrong URL being logged in request logging if multiple requests were sent over the same connection. In this case, the request logging only reported the first URL requested in the connection, even for all subsequent requests. * Added startup option `--query.allow-collections-in-expressions` to control whether collection names can be used in arbitrary places in AQL expressions, e.g. `collection + 1`. This was allowed before, as a collection can be seen as an array of documents. However, referring to a collection like this in a query would materialize all the collection's documents in RAM, making such constructs prohibitively expensive for medium-size to large-size collections. The option can now be set to `false` to prohibit accidental usage of collection names in AQL expressions. With that setting, using a collection inside an arbitrary expression will trigger the error `collection used as expression operand` and make the query fail. Even with the option being set to `false`, it is still possible to use collection names in AQL queries where they are expected, e.g. `FOR doc IN collection RETURN doc`. The option `--query.allow-collections-in-expressions` is introduced with a default value of `true` in 3.8 to ensure downwards-compatibility, but the default value will change to `false` in 3.9. Furthermore, the option will be deprecated in 3.9 and removed in later versions, in addition to making unintended usage of collection names always an error. * Deprecate option `--rocksdb.exclusive-writes`, which was meant to serve only as a stopgap measure while porting applications from the MMFiles storage engine to RocksDB. * Fix errors caused by creating some log messages in log level DEBUG in log topics PREGEL and GRAPHS. Setting the log level to DEBUG for these topics could lead to errors when running some Pregel jobs or SmartGraph traversals. v3.8.0-alpha.1 (2021-03-29) --------------------------- * Updated ArangoDB Starter to 0.15.0-preview-2. * Updated OpenSSL to 1.1.1k and OpenLDAP to 2.4.58. * Updated arangosync to 2.0.1. * Fix connectionTime statistic. This statistic should provide the distribution of the connection lifetimes, but in previous versions the tracking was broken and no values were reported. * When using connections to multiple endpoints and switching between them, arangosh can now reuse existing connections by referring to an internal connection cache. This helps for arangosh scripts that repeatedly connect to multiple endpoints, and avoids wasting lots of ephemeral TCP ports remaining in CLOSE_WAIT state. This change is transparent to any arangosh scripts or commands that do not reconnect to other endpoints than the one specified at arangosh start. * Add an option for locking down all endpoints in the `/_admin/cluster` REST API for callers without a proper JWT set in the request. There is a new startup option `--cluster.api-jwt-policy` that allows *additional* checks for a valid JWT in requests to sub-routes of `/_admin/cluster`. The possible values for the startup option are: - "jwt-all": requires a valid JWT for all accesses to `/_admin/cluster` and its sub-routes. If this configuration is used, the "Cluster" and "Nodes" sections of the web interface will be disabled, as they are relying on the ability to read data from several cluster APIs. - "jwt-write": requires a valid JWT for write accesses (all HTTP methods except HTTP GET) to `/_admin/cluster`. This setting can be used to allow privileged users to read data from the cluster APIs, but not to do any modifications. All existing permissions checks for the cluster API routes are still in effect with this setting, meaning that read operations without a valid JWT may still require dedicated other permissions (as in 3.7). - "jwt-compat": no *additional* access checks are in place for the cluster APIs. However, all existing permissions checks for the cluster API routes are still in effect with this setting, meaning that all operations may still require dedicated other permissions (as in 3.7). The default value for the option is `jwt-compat`, which means this option will not cause any extra JWT checks compared to 3.7. * UI builds are now using the yarn package manager instead of the previously used node package manager. * Fix shortName labels in metrics, in particular for agents. * The old metrics api contains the following gauges which should actually be counters: - arangodb_scheduler_jobs_dequeued - arangodb_scheduler_jobs_submitted - arangodb_scheduler_jobs_done Therefore the new v2 metric api adds the following counters: - arangodb_scheduler_jobs_dequeued_total - arangodb_scheduler_jobs_submitted_total - arangodb_scheduler_jobs_done_total These counters are only visible in the new v2 metric and replace the old metrics which are suppressed for v2. * Fix implicit capture of views in a context of JS transaction. * Introduce metrics for AQL query memory limit violations: - `arangodb_aql_global_query_memory_limit_reached`: Total number of times the global query memory limit was violated. - `arangodb_aql_local_query_memory_limit_reached`: Total number of times a local query memory limit was violated. * Set the default value for `--query.global-memory-limit` to around 90% of RAM, so that a global memory limit is now effective by default. The default global memory limit value is calculated by a formula depending on the amount of available RAM and will result in the following values for common RAM sizes: RAM: 0 (0MiB) Limit: 0 unlimited, %mem: n/a RAM: 134217728 (128MiB) Limit: 33554432 (32MiB), %mem: 25.0 RAM: 268435456 (256MiB) Limit: 67108864 (64MiB), %mem: 25.0 RAM: 536870912 (512MiB) Limit: 255013683 (243MiB), %mem: 47.5 RAM: 805306368 (768MiB) Limit: 510027366 (486MiB), %mem: 63.3 RAM: 1073741824 (1024MiB) Limit: 765041049 (729MiB), %mem: 71.2 RAM: 2147483648 (2048MiB) Limit: 1785095782 (1702MiB), %mem: 83.1 RAM: 4294967296 (4096MiB) Limit: 3825205248 (3648MiB), %mem: 89.0 RAM: 8589934592 (8192MiB) Limit: 7752415969 (7393MiB), %mem: 90.2 RAM: 17179869184 (16384MiB) Limit: 15504831938 (14786MiB), %mem: 90.2 RAM: 25769803776 (24576MiB) Limit: 23257247908 (22179MiB), %mem: 90.2 RAM: 34359738368 (32768MiB) Limit: 31009663877 (29573MiB), %mem: 90.2 RAM: 42949672960 (40960MiB) Limit: 38762079846 (36966MiB), %mem: 90.2 RAM: 68719476736 (65536MiB) Limit: 62019327755 (59146MiB), %mem: 90.2 RAM: 103079215104 (98304MiB) Limit: 93028991631 (88719MiB), %mem: 90.2 RAM: 137438953472 (131072MiB) Limit: 124038655509 (118292MiB), %mem: 90.2 RAM: 274877906944 (262144MiB) Limit: 248077311017 (236584MiB), %mem: 90.2 RAM: 549755813888 (524288MiB) Limit: 496154622034 (473169MiB), %mem: 90.2 * Increase default idle timeout in streaming transactions from 10 seconds to 60 seconds, and make the timeout configurable via a startup parameter `--transaction.streaming-idle-timeout`. * Use RebootTracker to abort cluster transactions on DB servers should the originating coordinator die or be rebooted. The previous implementation left the coordinator's transactions open on DB servers until they timed out there. Now, the coordinator's unavailability or reboot will be detected as early as it is reported by the agency, and all open transactions from that coordinator will be auto-aborted on DB servers. * Update the Web UI's list of built-in AQL functions for proper syntax highlighting in the query editor. * Fix a crash caused by returning a result produced by ANALYZER function. * Fix a race in LogAppender::haveAppenders. `haveAppenders` is called as part of audit logging. It accesses internal maps but previously did not hold a lock while doing so. * Bug-fix in the case of very rare network issues there was a chance that an AQL query could get stuck during a cleanup and after a commit. This would cause the client to receive a timeout, and the Coordinator blocking a Scheduler thread. This situation is sorted out and the thread will not be blocked anymore. We also added logs in case the query could not successfully be cleaned up, which would leave locks on shards behind. * Fix an assertion failure that occurred when restoring view definitions from a cluster into a single server. * Added new ArangoSearch analyzer type "stopwords". * Fix error message in case of index unique constraint violations. They were lacking the actual error message (i.e. "unique constraint violated") and only showed the index details. The issue was introduced only in devel in Feb. * Removed old metrics in new v2 metric api. Those metric endpoints were identical to the sum value of histograms. * Allow process-specific logfile names. This change allows replacing '$PID' with the current process id in the `--log.output` and `--audit.output` startup parameters. This way it is easier to write process-specific logfiles. * Backport a bugfix from upstream RocksDB for opening encrypted files with small sizes. Without the bugfix, the server may run into assertion failures during recovery. * Fix duplicate leaving of V8 contexts when returning streaming cursors. The `exitContext` call done on query shutdown could previously try to exit the V8 context multiple times, which would cause undefined behavior. Now we are tracking if we already left the context to prevent duplicate invocation. * In a cluster, do not create the collections `_statistics`, `_statistics15` and `statisticsRaw` on DB servers. These collections should only be created by the coordinator, and should translate into 2 shards each on DB servers. But there shouldn't be shards named `_statistics*` on DB servers. * Fixed two bogus messages about hotbackup restore: - Coordinators unconditionally logged the message "Got a hotbackup restore event, getting new cluster-wide unique IDs..." on shutdown. This was not necessarily related to a hotbackup restore. - DB servers unconditionally logged the message "Strange, we could not unregister the hotbackup restore callback." on shutdown, although this was meaningless. * Rename "save" return attribute to "dst" in AQL functions `DATE_UTCTOLOCAL` and `DATE_LOCALTOUTC`. * Fix potentially undefined behavior when creating a CalculationTransactionContext for an arangosearch analyzer. An uninitialized struct member was passed as an argument to its base class. This potentially had no observable effects, but should be fixed. * Retry a cluster internal network request if the connection comes from the pool and turns out to be stale (connection immediately closed). This fixes some spurious errors after a hotbackup restore. * Fix progress reporting for arangoimport with large files on Windows. Previously, progress was only reported for the first 2GB of data due to an int overflow. * Log the actual signal instead of "control-c" and also include the process id of the process that sent the signal. * Fixed GitHub issue #13665: Improve index selection when there are multiple candidate indexes. * When dropping a collection or an index with a larger amount of documents, the key range for the collection/index in RocksDB gets compacted. Previously, the compaction was running in foreground and thus would block the deletion operations. Now, the compaction is running in background, so that the deletion operations can return earlier. The maximum number of compaction jobs that are executed in background can be configured using the new startup parameter `--rocksdb.max-parallel-compactions`, which defaults to 2. * Put Sync/LatestID into hotbackup and restore it on hotbackup restore if it is in the backup. This helps with unique key generation after a hotbackup is restored to a young cluster. * Fixed a bug in the index count optimization that doubled counted documents when using array expansions in the fields definition. * Don't store selectivity estimate values for newly created system collections. Not storing the estimates has a benefit especially for the `_statistics` system collections, which are written to periodically even on otherwise idle servers. In this particular case, the actual statistics data was way smaller than the writes caused by the index estimate values, causing a disproportional overhead just for maintaining the selectivity estimates. The change now turns off the selectivity estimates for indexes in all newly created system collections, and for new user-defined indexes of type "persistent", "hash" or "skiplist", there is now an attribute "estimates" which can be set to `false` to disable the selectivity estimates for the index. The attribute is optional. Not setting it will lead to the index being created with selectivity estimates, so this is a downwards-compatible change for user-defined indexes. * Added startup option `--query.global-memory-limit` to set a limit on the combined estimated memory usage of all AQL queries (in bytes). If this option has a value of `0`, then no memory limit is in place. This is also the default value and the same behavior as in previous versions of ArangoDB. Setting the option to a value greater than zero will mean that the total memory usage of all AQL queries will be limited approximately to the configured value. The limit is enforced by each server in a cluster independently, i.e. it can be set separately for coordinators, DB servers etc. The memory usage of a query that runs on multiple servers in parallel is not summed up, but tracked separately on each server. If a memory allocation in a query would lead to the violation of the configured global memory limit, then the query is aborted with error code 32 ("resource limit exceeded"). The global memory limit is approximate, in the same fashion as the per-query limit provided by the option `--query.memory-limit` is. Some operations, namely calls to AQL functions and their intermediate results, are currently not properly tracked. If both `--query.global-memory-limit` and `--query.memory-limit` are set, the former must be set at least as high as the latter. To reduce the cost of globally tracking the memory usage of AQL queries, the global memory usage counter is only updated in steps of 32 kb, making this also the minimum granularity of the global memory usage figure. In the same fashion, the granularity of the peak memory usage counter inside each query was also adjusted to steps of 32 kb. * Added startup option `--query.memory-limit-override` to control whether individual AQL queries can increase their memory limit via the `memoryLimit` query option. This is the default, so a query that increases its memory limit is allowed to use more memory. The new option `--query.memory-limit-override` allows turning this behavior off, so that individual queries can only lower their maximum allowed memory usage. * Added metric `arangodb_aql_global_memory_usage` to expose the total amount of memory (in steps of 32 kb) that is currently in use by all AQL queries. * Added metric `arangodb_aql_global_memory_limit` to expose the memory limit from startup option `--query.global-memory-limit`. * Allow setting path to the timezone information via the `TZ_DATA` environment variable, in the same fashion as the currently existing `ICU_DATA` environment variable. The `TZ_DATA` variable is useful in environments` that start arangod from some unusual locations, when it can't find its `tzdata` directory automatically. * Fixed a bug in query cost estimation when a NoResults node occurred in a spliced subquery. This could lead to a server crash. * Fix slower-than-necessary arangoimport behavior: arangoimport has a built-in rate limiter, which can be useful for importing data with a somewhat constant rate. However, it is enabled by default and limits imports to 1MB per second. These settings are not useful. This change turns the rate limiting off by default, and sets the default chunk size to 8MB (up from 1MB) as well. This means that arangoimport will send larger batches to the server by default. The already existing `--batch-size` option can be used to control the maximum size of each batch. The new parameter `--auto-rate-limit` can now be used to toggle rate limiting. It defaults to off, whereas previously rate limiting was enabled by default unless `--batch-size` was specified when arangoimport was invoked. * The cluster dashboard charts in the web UI are now more readable during the initialization phase. Additionally, the amount of agents are now displayed there as well. An agent failure will also appear here in case it exists. * Added more useful information during the SmartGraph creation in the web UI in case the current database is a OneShard database. * Add support for building with Zen 3 CPU when optimizing for the local architecture. * The web UI's node overview now displays also agent information (cluster only). * The statistics view in the web UI does now provide more system specific information in case the Metrics API is enabled. Different statistics may be visible depending on the operating system. * Added metrics documentation snippets and infrastructure for that. * Added a new cluster distribution view to the web UI. The view includes general details about cluster-wide distribution in general as well as more detailed shard distribution specific information. * Reasonably harden MoveShard against invalid VelocyPack input. * Removed older reference to VelocyPackDumper. * Added `--documents-per-batch` option to arangoexport. This option allows to control the number of documents to be returned by each server-side batch. It can be used to limit the number of documents per batch when exporting collections with large documents. * Added a new metrics view to the web UI. This view can be used in a clustered environment as well as in a single instance. Metrics are displayed either in a tabular format or as plain text (Prometheus Text-based format). Additionally, the metrics can be downloaded there. * Added a new maintenance mode tab to the web UI in cluster mode. The new tab shows the current state of the cluster supervision maintenance and allows to enable/disable the maintenance mode from there. The tab will only be visible in the `_system` database. The required privileges for displaying the maintenance mode status and/or changing it are the as for using the REST APIs for the maintenance mode. * Added ability to display Coordinator and DBServer logs from inside the Web UI in a clustered environment when privileges are sufficient. Additionally, displayed log entries can now be downloaded from the web UI in single server and in cluster mode. * The Web UI's info view of a collection now displays additional properties and statistics (e.g. RocksDB related figures, sharding information and more). * Improve progress reporting for shard synchronization in the web UI. The UI will now show how many shards are actively syncing data, and will provide a better progress indicator, especially if there is more than one follower for a shard. * Added `--shard` option to arangodump, so that dumps can be restricted to one or multiple shards only. * Add optional hostname logging to log messages. Whether or not the hostname is added to each log message can be controlled via the new startup option `--log.hostname`. Its default value is the empty string, meaning no hostname will be added to log messages. Setting the option to an arbitrary string value will make this string be logged in front of each regular log message, and inside the `hostname` attribute in case of JSON-based logging. Setting the option to a value of `auto` will use the hostname as returned by `gethostbyname`. * Added list-repeat AIR primitive that creates a list containing n copies of the input value. * Prevent arangosh from trying to connect after every executed command. This fixes the case when arangosh is started with default options, but no server is running on localhost:8529. In this particular case, arangosh will try to connect on startup and after every executed shell command. The connect attempts all fail and time out after 300ms. In this case we now don't try to reconnect after every command. * Added 'custom-query' testcase to arangobench to allow execution of custom queries. This also adds the options `--custom-query` and `--custom-query-file` for arangobench. * Addition to the internal Refactoring of K_PATHS feature: K_PATHS queries are now being executed on the new refactored graph engine in a clustered environment. This change should not have any visible effect on users. * Reduce memory footprint of agency Store in Node class. * On Windows create a minidump in case of an unhandled SEH exception for post-mortem debugging. * Add JWT secret support for arangodump and arangorestore, i.e. they now also provide the command-line options `--server.ask-jwt-secret` and `--server.jwt-secret-keyfile` with the same meanings as in arangosh. * Add optional hyperlink to program option sections for information purposes, and add optional sub-headlines to program options for better grouping. These changes will be visible only when using `--help`. * For Windows builds, remove the defines `_SILENCE_ALL_CXX17_DEPRECATION_WARNINGS` and `_ENABLE_ATOMIC_ALIGNMENT_FIX` that were needed to build Boost components with MSVC in older versions of Boost and MSVC. Both of these defines are obsolete nowadays. * Database initial sync considers document count on leader for estimating timeouts when over 1 million docs on leader. * Fixed issue #13117: Aardvark: Weird cursor offsets in query editor. Disabled font ligatures for Ace editor in Web UI to avoid rare display issue. * Make all AQL cursors return compact result arrays. As a side-effect of this change, this makes profiling (i.e. using `db._profileQuery(...)` work for streaming queries as well. Previously, profiling a streaming query could have led to some internal errors, and even query results being returned, even though profiling a query should not return any query results. * Try to raise file descriptors limit in local start scripts (in `scripts/` directory - used for development only). * Fixed replication bug in MerkleTree sync protocol, which could lead to data corruption. The visible effect was that shards could no longer get in sync since the counts would not match after sync, even after a recount. This corruption only happened if there were large amounts of differences (at least 65537) and the destination side had newer revisions for some keys than the source side. * Simplify the DistributeExecutor and avoid implicit modification of its input variable. Previously the DistributeExecutor could update the input variable in-place, leading to unexpected results (see #13509). The modification logic has now been moved into three new _internal_ AQL functions (MAKE_DISTRIBUTE_INPUT, MAKE_DISTRIBUTE_INPUT_WITH_KEY_CREATION, and MAKE_DISTRIBUTE_GRAPH_INPUT) and an additional calculation node with an according function call will be introduced if we need to prepare the input data for the distribute node. * Added new REST APIs for retrieving the sharding distribution: - GET `/_api/database/shardDistribution` will return the number of collections, shards, leaders and followers for the database it is run inside. The request can optionally be restricted to include data from only a single DB server, by passing the `DBserver` URL parameter. This API can only be used on coordinators. - GET `/_admin/cluster/shardDistribution` will return global statistics on the current shard distribution, showing the total number of databases, collections, shards, leaders and followers for the entire cluster. The results can optionally be restricted to include data from only a single DB server, by passing the `DBserver` URL parameter. By setting the `details` URL parameter, the response will not contain aggregates, but instead one entry per available database will be returned. This API can only be used in the `_system` database of coordinators, and requires admin user privileges. * Decrease the size of serialized index estimates, by introducing a compressed serialization format. The compressed format uses the previous uncompressed format internally, compresses it, and stores the compressed data instead. This makes serialized index estimates a lot smaller, which in turn decreases the size of I/O operations for index maintenance. * More improvements for logging: - Added new REST API endpoint GET `/_admin/log/entries` to return log entries in a more intuitive format, putting each log entry with all its properties into an object. The API response is an array with all log message objects that match the search criteria. This is an extension to the already existing API endpoint GET `/_admin/log`, which returned log messages fragmented into 5 separate arrays. The already existing API endpoint GET `/_admin/log` for retrieving log messages is now deprecated, although it will stay available for some time. - Truncation of log messages now takes JSON format into account, so that the truncation of oversized JSON log messages still keeps a valid JSON structure even after the truncation. - The maximum size of in-memory log messages was doubled from 256 to 512 chars, so that longer parts of each log message can be preserved now. * Fix `/_admin/cluster/removeServer` API. This often returned HTTP 500 with an error message "Need open Array" due to an internal error when setting up agency preconditions. * Remove logging startup options `--log.api-enabled` and `--log.keep-logrotate` for all client tools (arangosh, arangodump, arangorestore etc.), as these options are only meaningful for arangod. * Extend the "move-calculations-up" optimizer rule so that it can move calculations out of subqueries into the outer query. * Don't allocate ahead-of-time memory for striped PRNG array in arangod, but instead use thread-local PRNG instances. Not only does this save a few megabytes of memory, but it also avoids potential (but unlikely) sharing of the same PRNG instance by multiple threads. * Remove undocumented CMake variable `USE_BACKTRACE`, and remove define `ARANGODB_ENABLE_BACKTRACE`. Both were turned off by default before, and when turned on allow to produce backtraces from within the executable in case debug symbols were available, working and the build was also compiled with `USE_MAINTAINER_MODE=On`. Some code in this context was obviously unreachable, so now it has all been removed. To log a backtrace from within arangod, it is now possible to call `CrashHandler::logBacktrace()`, which will log a backtrace of the calling thread to the arangod log. This is restricted to Linux builds only. * Fix warnings about suggest-override which can break builds when warnings aret reated as errors. * Turn off option `--server.export-read-write-metrics` for now, until there is certainty about the runtime overhead it introduces. * Remove unsafe query option `inspectSimplePlans`. This option previously defaulted to `true`, and turning it off could make particular queries fail. The option was ignored in the cluster previously, and turning it off only had an effect in single server, there making very simple queries (queries not containing any FOR loops) not going through the optimizer's complete pipeline as a performance optimization. However, the optimization was only possible for a very small number of queries and even had adverse effects, so it is now removed entirely. * On Linux and MacOS, require at least 8192 usable file descriptors at startup. If less file descriptors are available to the arangod process, then the startup is automatically aborted. Even the chosen minimum value of 8192 will often not be high enough to store considerable amounts of data. However, no higher value was chosen in order to not make too many existing small installations fail at startup after upgrading. The required number of file descriptors can be configured using the startup option `--server.descriptors-minimum`. It defaults to 8192, but it can be increased to ensure that arangod can make use of a sufficiently high number of files. Setting `--server.descriptors-minimum` to a value of `0` will make the startup require only an absolute minimum limit of 1024 file descriptors, effectively disabling the change. Such low values should only be used to bypass the file descriptors check in case of an emergency, but this is not recommended for production. * Added metric `arangodb_transactions_expired` to track the total number of expired and then garbage-collected transactions. * Allow toggling the document read/write counters and histograms via the new startup option `--server.export-read-write-metrics false`. This option defaults to `true`, so these metrics will be exposed by default. * Upgraded bundled version of libunwind to v1.5. * Added startup option `--javascript.tasks` to allow turning off JavaScript tasks if not needed. The default value for this option is `true`, meaning JavaScript tasks are available as before. However, with this option they can be turned off by admins to limit the amount of JavaScript user code that is executed. * Only instantiate a striped PRNG instance for the arangod server, but not for any of the client tools (e.g. arangosh, arangodump, arangorestore). The client tools do not use the striped PRNG, so we can save a few MBs of memory for allocating the striped PRNG instance there, plus some CPU time for initializing it. * Improve shard synchronization protocol by only transferring the required parts of the inventory from leader to follower. Previously, for each shard the entire inventory was exchanged, which included all shards of the respective database with all their details. In addition, save 3 cluster-internal requests per shard in the initial shard synchronization protocol by reusing already existing information in the different steps of the replication process. * Added metric `arangodb_scheduler_low_prio_queue_last_dequeue_time` that provides the time (in milliseconds) it took for the most recent low priority scheduler queue item to bubble up to the queue's head. This metric can be used to estimate the queuing time for incoming requests. The metric will be updated probabilistically when a request is pulled from the scheduler queue, and may remain at its previous value for a while if only few requests are coming in or remain permanently at its previous value if no further requests are incoming at all. * Allow {USER} placeholder string also in `--ldap.search-filter`. * Fixed some wrong behavior in single document updates. If the option ignoreRevs=false was given and the precondition _rev was given in the body but the _key was given in the URL path, then the rev was wrongly taken as 0, rather than using the one from the document body. * Improved logging for error 1489 ("a shard leader refuses to perform a replication operation"). The log message will now provide the database and shard name plus the differing information about the shard leader. * Add shard-parallelism to arangodump when dumping collections with multiple shards. Previously, arangodump could execute a dump concurrently on different collections, but it did not parallelize the dump for multiple shards of the same collection. This change should speed up dumping of collections with multiple shards. When dumping multiple shards of the same collection concurrently, parallelism is still limited by all these threads needing to serialize their chunks into the same (shared) output file. * Add option `--envelope` for arangodump, to control if each dumped document should be wrapped into a small JSON envelope (e.g. `{"type":2300,"data":{...}}`). This JSON envelope is not necessary anymore since ArangoDB 3.8, so omitting it can produce smaller (and slightly faster) dumps. Restoring a dump without these JSON envelopers is handled automatically by ArangoDB 3.8 and higher. Restoring a dump without these JSON envelopes into previous versions (pre 3.8) however is not supported. Thus the option should only be used if the client tools (arangodump, arangorestore) and the arangod server are all using v3.8 or higher, and the dumps will never be stored into earlier versions. The default value for this option is `true`, meaning the JSON wrappers will be stored as part of the dump. This is compatible with all previous versions. * Make AQL optimizer rule "splice-subqueries" mandatory, in the sense that it cannot be disabled anymore. As a side effect of this change, there will no query execution plans created by 3.8 that contain execution nodes of type `SubqueryNode`. `SubqueryNode`s will only be used during query planning and optimization, but at the end of the query optimization phase will all have been replaced with nodes of types `SubqueryStartNode` and `SubqueryEndNode`. The code to execute non-spliced subqueries remains in place so that 3.8 can still execute queries planned on a 3.7 instance with the "splice-subqueries" optimizer rule intentionally turned off. The code for executing non-spliced subqueries can be removed in 3.9. * AQL query execution plan register usage optimization. This is a performance optimization that may positively affect some AQL queries that use a lot of variables that are only needed in certain parts of the query. The positive effect will come from saving registers, which directly translates to saving columns in AqlItemBlocks. Previously, the number of registers that were planned for each depth level of the query never decreased when going from one level to the next. Even though unused registers were recycled since 3.7, this did not lead to unused registers being completely dismantled. Now there is an extra step at the end of the register planning that keeps track of the actually used registers on each depth, and that will shrink the number of registers for the depth to the id of the maximum register. This is done for each depth separately. Unneeded registers on the right hand side of the maximum used register are now discarded. Unused registers on the left hand side of the maximum used register id are not discarded, because we still need to guarantee that registers from depths above stay in the same slot when starting a new depth. * Added metric `arangodb_aql_current_query` to track the number of currently executing AQL queries. * Internal refactoring of K_PATH feature, with the goal to have all graph algorithms on the same framework. This change should not have any visible effect on users. * Removed server-side JavaScript object `ArangoClusterComm`, so it cannot be used from inside JavaScript operations or Foxx. The `ArangoClusterComm` object was previously used inside a few internal JavaScript operations, but was not part of the public APIs. * Restrict access to functions inside JavaScript objects `ArangoAgency` and `ArangoAgent` to JavaScript code that is running in privileged mode, i.e. via the server's emergency console, the `/_admin/execute` API (if turned on) or internal bootstrap scripts. * Added startup option `--javascript.transactions` to allow turning off JavaScript transactions if not needed. The default value for this option is `true`, meaning JavaScript transactions are available as before. However, with this option they can be turned off by admins to limit the amount of JavaScript user code that is executed. * Introduce a default memory limit for AQL queries, to prevent rogue queries from consuming the entire memory available to an arangod instance. The limit is introduced via changing the default value of the option `--query.memory-limit` from previously `0` (meaning: no limit) to a dynamically calculated value. The per-query memory limits defaults are now: Available memory: 0 (0MiB) Limit: 0 unlimited, %mem: n/a Available memory: 134217728 (128MiB) Limit: 33554432 (32MiB), %mem: 25.0 Available memory: 268435456 (256MiB) Limit: 67108864 (64MiB), %mem: 25.0 Available memory: 536870912 (512MiB) Limit: 201326592 (192MiB), %mem: 37.5 Available memory: 805306368 (768MiB) Limit: 402653184 (384MiB), %mem: 50.0 Available memory: 1073741824 (1024MiB) Limit: 603979776 (576MiB), %mem: 56.2 Available memory: 2147483648 (2048MiB) Limit: 1288490189 (1228MiB), %mem: 60.0 Available memory: 4294967296 (4096MiB) Limit: 2576980377 (2457MiB), %mem: 60.0 Available memory: 8589934592 (8192MiB) Limit: 5153960755 (4915MiB), %mem: 60.0 Available memory: 17179869184 (16384MiB) Limit: 10307921511 (9830MiB), %mem: 60.0 Available memory: 25769803776 (24576MiB) Limit: 15461882265 (14745MiB), %mem: 60.0 Available memory: 34359738368 (32768MiB) Limit: 20615843021 (19660MiB), %mem: 60.0 Available memory: 42949672960 (40960MiB) Limit: 25769803776 (24576MiB), %mem: 60.0 Available memory: 68719476736 (65536MiB) Limit: 41231686041 (39321MiB), %mem: 60.0 Available memory: 103079215104 (98304MiB) Limit: 61847529063 (58982MiB), %mem: 60.0 Available memory: 137438953472 (131072MiB) Limit: 82463372083 (78643MiB), %mem: 60.0 Available memory: 274877906944 (262144MiB) Limit: 164926744167 (157286MiB), %mem: 60.0 Available memory: 549755813888 (524288MiB) Limit: 329853488333 (314572MiB), %mem: 60.0 As previously, a memory limit value of `0` means no limitation. The limit values are per AQL query, so they may still be too high in case queries run in parallel. The defaults are intentionally high in order to not stop any valid, previously working queries from succeeding. * Added startup option `--audit.queue` to control audit logging queuing behavior (Enterprise Edition only): The option controls whether audit log messages are submitted to a queue and written to disk in batches or if they should be written to disk directly without being queued. Queueing audit log entries may be beneficial for latency, but can lead to unqueued messages being lost in case of a power loss or crash. Setting this option to `false` mimics the behavior from 3.7 and before, where audit log messages were not queued but written in a blocking fashion. * Added metric `arangodb_server_statistics_cpu_cores` to provide the number of CPU cores visible to the arangod process. This is the number of CPU cores reported by the operating system to the process. If the environment variable `ARANGODB_OVERRIDE_DETECTED_NUMBER_OF_CORES` is set to a positive value at instance startup, this value will be returned instead. * `COLLECT WITH COUNT INTO x` and `COLLECT var = expr WITH COUNT INTO x` are now internally transformed into `COLLECT AGGREGATE x = LENGTH()` and `COLLECT var = expr AGGREGATE x = LENGTH()` respectively. In addition, any argument passed to the `COUNT`/`LENGTH` aggregator functions are now optimized away. This not only simplified the code, but also allows more query optimizations: - If the variable in `COLLECT WITH COUNT INTO var` is not used, the implicit aggregator is now removed. - All queries of the form `COLLECT AGGREGATE x = LENGTH()` are now executed using the count executor, which can result in significantly improved performance. * Added AQL timezone functions `DATE_TIMEZONE` and `DATE_TIMEZONES`. * Make DB servers report storage engine health to the agency, via a new "health" attribute in requests sent to Sync/ServerStates/. The supervision can in the future check this attribute if it is posted, and mark servers as BAD or FAILED in case an unhealthy status is reported. DB server health is currently determined by whether or not the storage engine (RocksDB) has reported a background error, and by whether or not the free disk space has reached a critical low amount. The current threshold for free disk space is set at 1% of the disk capacity (only the disk is considered that contains the RocksDB database directory). The minimum required free disk space percentage can be configured using the new startup option `--rocksdb.minimum-disk-free-percent`, which needs to be between 0 and 1 (including). A value of 0 disables the check. The minimum required free disk space can also be configured in bytes using the new startup option `--rocksdb.minimum-disk-free-bytes`. A value of 0 disables this check, too. * Failed servers are now reported consistently in the web interface, at approximately the same time in the navigation bar and in the nodes view. Previously these two places had their own, independent poll mechanism for the nodes' health, and they were updated independently, which could cause an inconsistent view of the nodes' availability. Using only one poll mechanism instead also saves some period background requests for the second availability check. * Stabilize a Foxx cleanup test. * Drop a pair of braces {} in /_admin/metrics in case of empty labels, which makes the API adhere better to the official Prometheus syntax. * Add some more metrics to the ConnectionPool. * Reduce overhead of audit logging functionality if audit logging is turned off. * Add several more attributes to audit-logged queries, namely query execution time and exit code (0 = no error, other values correspond to general ArangoDB error codes). * Fixed a bug in maintainer mode sorting followerinfo lists the wrong way. * Limit value of `--rocksdb.block-cache-size` to 1 GB for agent instances to reduce agency RAM usage, unless configured otherwise. In addition, limit the value of `--rocksdb.total-write-buffer-size` to 512 MB on agent instances for the same reason. * Added new `rocksdb_write_stalls` and `rocksdb_write_stops` counter metrics, which should be more accurate than existing metrics related to the underlying conditions. * Increased the default value of `--rocksdb.min-write-buffer-number-to-merge` in some cases when we have allocated a sufficient amount of memory to the write buffers for this to make sense. The increased value should help prevent compaction-induced write stalls/stops, and should only be enabled when under conditions such that it shouldn't greatly increase the chance of flush-induced write stalls/stops. * Changed the default values for `--rocksdb.cache-index-and-filter-blocks` and `--rocksdb.cache-index-and-filter-blocks-with-high-priority` to true to improve control over memory usage. * Lowered the minimum allowed value for `--rocksdb.max-write-buffer-number` from 9 to 4 to allow more fine-grained memory usage control. * Added new ArangoSearch view option 'countApproximate' for customizing view count strategy. * Views on SmartGraph Edge collections do not contain some documents twice. * Fixed issue #12248: Web UI - Added missing HTML escaping in the setup script section of a Foxx app. * The scheduler will now run a minimum of 4 threads at all times, and the default and minimal value for `--server.maximal-threads` has been lowered from 64 to the greater of 32 and twice the number of detected cores. * Throttle work coming from low priority queue, according to a constant and to an estimate taking into account fanout for multi-shard operations. * Move to 4 priority levels "low", "medium", "high" and "maintenance" in scheduler to ensure that maintenance work and diagnostics is always possible, even in the case of RocksDB throttles. Do not allow any RocksDB work on "maintenance". * Commit replications on high priority queue. * Essentially get rid of timeout in replication to drop followers. This is now entirely handled via reboot and failure tracking. The timeout has now a default minimum of 15 minutes but can still be configured via options. * Additional metrics for all queue lengths and low prio ongoing work. * New metric for number and total time of replication operations. * New metrics for number of internal requests in flight, internal request duration, and internal request timeouts * Fix `Gauge` assignment operators. * Add cluster support for collection.checksum() method to calculate CRC checksums for collections. * Make all Pregel HTTP and JavaScript APIs also accept stringified execution number values, in addition to numeric ones. This allows passing larger execution numbers as strings, so that any data loss due to numeric data type conversion (uint32 => double) can be avoided. The change also makes the Pregel HTTP and JavaScript APIs for starting a run return a stringified execution number, e.g. "12345" instead of 12345. * Turn off `StatisticsWorker` thread on DB servers. This thread was previously only running queries on the local RocksDB instance, but using the cluster-wide collection names. So effectively it did nothing except use a bit of background CPU. In this case it is better to turn off the background thread entirely on the DB servers. * Avoid the usage of std::regex when constructing date/time string values for log messages. This is a performance optimization only. * Increase background garbage-collection interval for cluster transactions from 1 second to 2 seconds. This change should reduce the amount of background task activity a tiny bit (though hardly measurable on an otherwise idle server). * Make the audit log honor the configured logging date/time output format (i.e. `--log.time-format` option). Previously the audit logging always created a time value in the server's local time, and logged it in format YYYY-MM-DDTHH:MM:SS. From 3.8 onwards, the audit logger will honor the date/time format specified via the `--log.time-format` option, which defaults to `utc-datestring`. This means the audit logging will by default log all dates/times in UTC time. To restore the pre-3.8 behavior, please set the option `--log.time-format` to `local-datestring`, which will make the audit logger (and all other server log messages) use the server's local time. * Added metrics for the system CPU usage: - `arangodb_server_statistics_user_percent`: Percentage of time that the system CPUs have spent in user mode - `arangodb_server_statistics_system_percent`: Percentage of time that the system CPUs have spent in kernel mode - `arangodb_server_statistics_idle_percent`: Percentage of time that the system CPUs have been idle - `arangodb_server_statistics_iowait_percent`: Percentage of time that the system CPUs have been waiting for I/O These metrics resemble the overall CPU usage metrics in `top`. They are available on Linux only. * Fix log topic of general shutdown message from "cluster" to general. * Automatically add "www-authenticate" headers to server HTTP 401 responses, as required by the HTTP specification. * Enable HTTP request statistics and provide metrics even in case `--server.statistics-history` is set to `false` (this option will set itself to off automatically on agency instances on startup if not explicitly set). This change provides more metrics on all server instances, without the need to persist them in the instance's RocksDB storage engine. * Remove extra CMake option `DEBUG_SYNC_REPLICATION` and use the already existing `USE_FAILURE_TESTS` options for its purpose. * Updated bundled version of Snappy compression/decompression library to 1.1.8. * Added support of `GEO_DISTANCE`, `GEO_CONTAINS`, `GEO_INTERSECTS`, `GEO_IN_RANGE` to ArangoSearch. * Added new `GeoJSON` ArangoSearch analyzer. * Added new `GeoPoint` ArangoSearch analyzer. * Added new `GEO_IN_RANGE` AQL function. * Added new 'aql' type for ArangoSearch analyzers. * Obsoleted the startup options `--database.throw-collection-not-loaded-error` and `--ttl.only-loaded-collection`. These options were meaningful for the MMFiles storage engine only, but for the RocksDB storage engine they did not make any difference. Using these startup options is still possible, but will have no effect other than generating a warning at server startup. * Added CMake option `USE_MINIMAL_DEBUGINFO`. This option is turned off by default. If turned on, the created binaries will contain only a minimum amount of debug symbols, reducing the size of the executables. If turned off (which is the default), the binaries will contain full debug information, which will make them bigger in size unless the debug information is later stripped again. * Modified the returned error code for calling the `shards()` function on a collection in single-server from "internal error" (error number 4) to "shards API is only available in cluster" and error number 9, HTTP status code 501. * Added WINDOW keyword to AQL to allow aggregations on related rows. * Added new graph method K_PATHS to AQL. This will enumerate all paths between a source and a target vertex that match the given length. For example, the query ``` FOR path IN 2..4 OUTBOUND K_PATHS "v/source" TO "v/target" GRAPH "g" RETURN path ``` will yield all paths in format { vertices: [v/source, ... , v/target], edges: [v/source -> v/1, ..., v/n -> v/target } that have length exactly 2 or 3 or 4, start at v/source and end at v/target. The order of those paths in the result set is not guaranteed. * Fixed issue BTS-195: AQL update queries using the `keepNull` option set to false had an inconsistent behavior. For example, given a collection `test` with an empty document with just key `testDoc`, the following query would return different results when running for the first time or the second time: UPDATE 'testDoc' WITH {test: {sub1: true, sub2: null}} IN test OPTIONS { keepNull: false, mergeObjects: true } For its first run, the query would return { "_key": "testDoc", "test": { "sub1": true, "sub2": null } } (with the `null` attribute value not being removed). For all subsequent runs, the same query would return { "_key": "testDoc", "test": { "sub1": true, } } (with the `null` value removed as requested). This inconsistency was due to how the `keepNull` attribute was handled if the attribute already existed in the to-be-updated document or not. The behavior is now consistent, so `null` values are now properly removed from sub-attributes even if in the to-be-updated document the target attribute did not yet exist. This makes such updates idempotent again. This a behavior change compared previous versions, but it will only have effect when `keepNull` is set to `false` (the default value is `true` however), and only when just-inserted object sub-attributes contained `null` values. * Optimization of empty append entries. * Remove any special handling for obsoleted collection attributes `indexBuckets`, `journalSize`, `doCompact` and `isVolatile`. These attributes were meaningful only with the MMFiles storage engine and have no meaning with the RocksDB storage engine. Thus any special handling for these attributes can be removed in the internal code. Client applications and tests that rely on the behavior that setting any of these attributes produces an error when using the RocksDB engine may need adjustment now. * Added a --continue option to arangorestore. arangorestore now keeps track of the progress and can continue the restore operation when some error occured. * Don't respond with misleading error in smart vertex collections. When inserting a document with a non-conforming key pattern into a smart vertex collection, the response error code and message are 1466 (ERROR_CLUSTER_MUST_NOT_SPECIFY_KEY) and "must not specify _key for this collection". This is misleading, because it is actually allowed to specify a key value for documents in such collection. However, there are some restrictions for valid key values (e.g. the key must be a string and contain the smart graph attribute value at the front, followed by a colon. If any of these restrictions are not met, the server currently responds with "must not specify key for this collection", which is misleading. This change rectifies it so that the server responds with error 4003 (ERROR_KEY_MUST_BE_PREFIXED_WITH_SMART_GRAPH_ATTRIBUTE) and message "in smart vertex collections _key must be a string and prefixed with the value of the smart graph attribute". This should make it a lot easier to understand what the actual problem is. * Fix an issue in arangoimport improperly handling filenames with less than 3 characters. The specified input filename was checked for a potential ".gz" ending, but the check required the filename to have at least 3 characters. This is now fixed. * Fix for BTS-191: Made transaction API database-aware. * Minor clean up of and less verbosity in agent callbacks. * Speed up initial replication of collections/shards data by not wrapping each document in a separate `{"type":2300,"data":...}` envelope. In addition, the follower side of the replication will request data from leaders in VelocyPack format if the leader is running at least version 3.8. Stripping the envelopes and using VelocyPack for transfer allows for smaller data sizes when exchanging the documents and faster processing, and thus can lead to time savings in document packing and unpacking as well as reduce the number of required HTTP requests. * Added metric `arangodb_agency_callback_registered counter` for tracking the total number of agency callbacks that were registered. * Added weighted traversal. Use `mode: "weighted"` as option to enumerate paths by increasing weights. The cost of an edge can be read from an attribute which can be specified using `weightAttribute` option. * Fixed issue ES-696: SEARCH vs FILTER lookup performance. Consolidation functionality for ArangoSearch view links was able to hit non- mergable enormous amount of segments due to improper scheduling logic. * Make scheduler react and start new threads slightly faster in case a lot of new work arrives. * Added new ArangoSearch "pipeline" analyzer type. * Added replication metrics `arangodb_replication_initial_sync_bytes_received` for the number of bytes received during replication initial sync operations and `arangodb_replication_tailing_bytes_received` for the number of bytes received for replication tailing requests. Also added `arangodb_replication_failed_connects` to track the number of connection failures or non-OK response during replication. * Added metrics `rocksdb_free_inodes` and `rocksdb_total_inodes` to track the number of free inodes and the total/maximum number of inodes for the file system the RocksDB database directory is located in. These metrics will always be 0 on Windows. * Fixed slightly wrong log level for authentication and also added login event to the standard log. * Added new metrics for the total and the free disk space for the mount used for the RocksDB database directory: - `arangodb_rocksdb_free_disk_space`: provides the free disk space for the mount, in bytes - `arangodb_rocksdb_total_disk_space`: provides the total disk space of the mount, in bytes * Apply user-defined idle connection timeouts for HTTP/2 and VST connections. The timeout value for idle HTTP/2 and VST connections can now be configured via the configuration option `--http.keep-alive-timeout` in the same way as for HTTP/1 connections. HTTP/2 and VST connections that are sending data back to the client are now closed after 300 seconds or the configured idle timeout (the higher of both values is used here). Before this change, the timeouts for HTTP/2 and VST connections were hardcoded to 120 seconds, and even non-idle connections were closed after this timeout. * Added new metrics for replication: - `arangodb_replication_dump_requests`: number of replication dump requests made. - `arangodb_replication_dump_bytes_received`: number of bytes received in replication dump requests. - `arangodb_replication_dump_documents`: number of documents received in replication dump requests. - `arangodb_replication_dump_request_time`: wait time for replication dump requests. - `arangodb_replication_dump_apply_time`: time required for applying data from replication dump responses. - `arangodb_replication_initial_sync_keys_requests`: number of replication initial sync keys requests made. - `arangodb_replication_initial_sync_docs_requests`: number of replication initial sync docs requests made. - `arangodb_replication_initial_sync_docs_requested`: number of documents requested via replication initial sync requests. - `arangodb_replication_initial_sync_docs_inserted`: number of documents inserted by replication initial sync. - `arangodb_replication_initial_sync_docs_removed`: number of documents inserted by replication initial sync. - `arangodb_replication_initial_chunks_requests_time`: wait time histogram for replication key chunks determination requests. - `arangodb_replication_initial_keys_requests_time`: wait time for replication keys requests. - `arangodb_replication_initial_docs_requests_time`: time needed to apply replication docs data. - `arangodb_replication_initial_insert_apply_time`: time needed to apply replication initial sync insertions. - `arangodb_replication_initial_remove_apply_time`: time needed to apply replication initial sync removals. - `arangodb_replication_initial_lookup_time`: time needed for replication initial sync key lookups. - `arangodb_replication_tailing_requests`: number of replication tailing requests. - `arangodb_replication_tailing_follow_tick_failures`: number of replication tailing failures due to missing tick on leader. - `arangodb_replication_tailing_markers`: number of replication tailing markers processed. - `arangodb_replication_tailing_documents`: number of replication tailing document inserts/replaces processed. - `arangodb_replication_tailing_removals`: number of replication tailing document removals processed. - `arangodb_replication_tailing_bytes_received`: number of bytes received for replication tailing requests. - `arangodb_replication_tailing_request_time`: wait time for replication tailing requests. - `arangodb_replication_tailing_apply_time`: time needed to apply replication tailing markers. * Allow calling of REST APIs `/_api/engine/stats`, GET `/_api/collection`, GET `/_api/database/current` and GET `/_admin/metrics` on followers in active failover deployments. This can help debugging and inspecting the follower. * Support projections on sub-attributes (e.g. `a.b.c`). In previous versions of ArangoDB, projections were only supported on top-level attributes. For example, in the query FOR doc IN collection RETURN doc.a.b the projection that was used was just `a`. Now the projection will be `a.b`, which can help reduce the amount of data to be extracted from documents, when only some sub-attributes are accessed. In addition, indexes can now be used to extract the data of sub-attributes for projections. If for the above example query an index on `a.b` exists, it will be used now. Previously, no index could be used for this projection. Projections now can also be fed by any attribute in a combined index. For example, in the query FOR doc IN collection RETURN doc.b the projection can be satisfied by a single-attribute index on attribute `b`, but now also by a combined index on attributes `a` and `b` (or `b` and `a`). * Remove some JavaScript files containing testsuites and test utilities from our official release packages. * Show optimizer rules with highest execution times in explain output. * Renamed "master" to "leader" and "slave" to "follower" in replication messages. This will change the contents of replication log messages as well the string contents of replication-related error messages. The messages of the error codes 1402, 1403 and 1404 were also changed accordingly, as well as the identifiers: - `TRI_ERROR_REPLICATION_MASTER_ERROR` renamed to `TRI_ERROR_REPLICATION_LEADER_ERROR` - `TRI_ERROR_REPLICATION_MASTER_INCOMPATIBLE` renamed to `TRI_ERROR_REPLICATION_LEADER_INCOMPATIBLE` - `TRI_ERROR_REPLICATION_MASTER_CHANGE` renamed to `TRI_ERROR_REPLICATION_LEADER_CHANGE` This change also renames the API endpoint `/_api/replication/make-slave` to `/_api/replication/make-follower`. The API is still available under the old name, but using it is deprecated. * Make optimizer rule "remove-filters-covered-by-index" remove FILTERs that were referring to aliases of the collection variable, e.g. FOR doc IN collection LET value = doc.indexedAttribute FILTER value == ... Previously, FILTERs that were using aliases were not removed by that optimizer rule. In addition, the optimizer rule "remove-unnecessary-calculations" will now run again in case it successfully removed variables. This can unlock further removal of unused variables in sequences such as FOR doc IN collection LET value = doc.indexedAttribute LET tmp1 = value > ... LET tmp2 = value < ... when the removal of `tmp1` and `tmp2` makes it possible to also remove the calculation of `value`. * Fixed issue BTS-168: Fixed undefined behavior that did trigger segfaults on cluster startups. It is only witnessed for macOS based builds. The issue could be triggered by any network connection. This behavior is not part of any released version. * Hard-code returned "planVersion" attribute of collections to a value of 1. Before 3.7, the most recent Plan version from the agency was returned inside "planVersion". In 3.7, the attribute contained the Plan version that was in use when the in-memory LogicalCollection object was last constructed. The object was always reconstructed in case the underlying Plan data for the collection changed or when a collection contained links to arangosearch views. This made the attribute relatively useless for any real-world use cases, and so we are now hard-coding it to simplify the internal code. Using the attribute in client applications is also deprecated. * Don't prevent concurrent synchronization of different shards from the same database. Previously only one shard was synchronized at a time per database. * Wait until restore task queue is idle before shutting down. * Fix a race problem in the unit tests w.r.t. PlanSyncer. * Errors with error code 1200 (Arango conflict) will now get the HTTP response code 409 (Conflict) instead of 412 (Precondition failed), unless "if-match" header was used in `_api/document` or `_api/gharial`. * Keep the list of last-acknowledged entries in Agency more consistent. During leadership take-over it was possible to get into a situation that the new leader does not successfully report the agency config, which was eventually fixed by the Agent itself. Now this situation is impossible. * Added support `db._engineStats()` API in coordinator. Previously calling this API always produced an empty result. Now it will return the engine statistics as an object, with an entry for each individual DB-Server. * Added option `--log.use-json-format` to switch log output to JSON format. Each log message then produces a seperate line with JSON-encoded log data, which can be consumed by applications. * Added option `--log.process` to toggle the logging of the process id (pid) in log messages. Logging the process id is useless when running arangod in Docker containers, as the pid will always be 1. So one may as well turn it off in these contexts. * Added option `--log.in-memory` to toggle storing log messages in memory, from which they can be consumed via the `/_admin/log` and by the web UI. By default, this option is turned on, so log messages are consumable via API and the web UI. Turning this option off will disable that functionality and save a tiny bit of memory for the in-memory log buffers. v3.7.10 (2021-03-14) -------------------- * Reasonably harden MoveShard against unexpected VelocyPack input. * Follower DB servers will now respond with error code `TRI_ERROR_CLUSTER_SHARD_FOLLOWER_REFUSES_OPERATION` to any read request. This fixes inadequate HTTP 404 responses from followers, e.g. during chaos tests. * Fixed Github issue #13632: Query Fails on Upsert with Replace_nth. * Updated arangosync to 1.2.3. * Backported AQL sort performance improvements from devel. This change can improve the performance of local sorts operations, e.g. Baseline (3.7.9): Query String (94 chars, cacheable: false): FOR i IN 1..500000 LET value = CONCAT('testvalue-to-be-sorted', i) SORT value ASC RETURN value Execution plan: Id NodeType Calls Items Runtime [s] Comment 1 SingletonNode 1 1 0.00003 * ROOT 2 CalculationNode 1 1 0.00003 - LET #2 = 1 .. 500000 /* range */ /* simple expression */ 3 EnumerateListNode 500 500000 0.08725 - FOR i IN #2 /* list iteration */ 4 CalculationNode 500 500000 0.22722 - LET value = CONCAT("testvalue-to-be-sorted", i) /* simple expression */ 5 SortNode 500 500000 2.05180 - SORT value ASC /* sorting strategy: standard */ 6 ReturnNode 500 500000 0.02911 - RETURN value Query Statistics: Writes Exec Writes Ign Scan Full Scan Index Filtered Exec Time [s] 0 0 0 0 0 2.39644 With sort optimization (3.7.10): Query String (94 chars, cacheable: false): FOR i IN 1..500000 LET value = CONCAT('testvalue-to-be-sorted', i) SORT value ASC RETURN value Execution plan: Id NodeType Calls Items Runtime [s] Comment 1 SingletonNode 1 1 0.00002 * ROOT 2 CalculationNode 1 1 0.00003 - LET #2 = 1 .. 500000 /* range */ /* simple expression */ 3 EnumerateListNode 500 500000 0.08755 - FOR i IN #2 /* list iteration */ 4 CalculationNode 500 500000 0.26161 - LET value = CONCAT("testvalue-to-be-sorted", i) /* simple expression */ 5 SortNode 500 500000 1.36070 - SORT value ASC /* sorting strategy: standard */ 6 ReturnNode 500 500000 0.02864 - RETURN value Query Statistics: Writes Exec Writes Ign Scan Full Scan Index Filtered Exec Time [s] 0 0 0 0 0 1.73940 * Fixed a problem that coordinators would vanish from the UI and the Health API if one switched the agency Supervision into maintenance mode and kept left that maintenance mode on for more than 24h. * Fixed a bug in the web interface that displayed the error "Not authorized to execute this request" when trying to create an index in the web interface in a database other than `_system` with a user that does not have any access permissions for the `_system` database. The error message previously displayed error actually came from an internal request made by the web interface, but it did not affect the actual index creation. * Fixed issue BTS-309: The Graph API (Gharial) did not respond with the correct HTTP status code when validating edges. It now responds with 400 (Bad Request) as documented and a new, more precise error code (1947) and message if a vertex collection referenced in the _from or _to attribute is not part of the graph. v3.7.9 (2021-03-01) ------------------- * Fix issue #13476: The Java driver v6.9.0 (and older) has bad performance when iterating over AQL cursor results in certain cases. This works around this. This workaround will no longer be available in 3.8. * Enable statistics in web UI in non-`_system` databases in cluster mode. In cluster mode, the web UI dashboard did not display statistics properly when not being logged in to the `_system` database. For all other databases than `_system`, no statistics were displayed but just some "No data..." placeholders. Statistics for non-`_system` databases were not properly displayed since 3.7.6 due to an internal change in the statistics processing. In addition, a new startup option `--server.statistics-all-databases` controls whether cluster statistics are displayed in the web interface for all databases (if the option is set to `true`) or just for the system database (if the option is set to `false`). The default value for the option is `true`, meaning statistics will be displayed in the web interface for all databases. * Updated OpenSSL to 1.1.1j and OpenLDAP to 2.4.57. * Cleanup old HotBackup transfer jobs in agency. * Added logging of elapsed time of ArangoSearch commit/consolidation/cleanup jobs. * Fix too early stop of replication, when waiting for keys in large collections/shards. * Fixed issue BTS-268: fix a flaky Foxx self-heal procedure. * Fixed issue DEVSUP-720: Within an AQL query, the "COLLECT WITH COUNT INTO" statement could lead to a wrong count output when used in combination with an index which has been created with an array index attribute. * Fix profiling of AQL queries with the `silent` and `stream` options sets in combination. Using the `silent` option makes a query execute, but discard all its results instantly. This led to some confusion in streaming queries, which can return the first query results once they are available, but don't necessarily execute the full query. Now, `silent` correctly discards all results even in streaming queries, but this has the effect that a streaming query will likely be executed completely when the `silent` option is set. This is not the default however, and the `silent` option is normally not set. There is no change for streaming queries if the `silent` option is not set. As a side-effect of this change, this makes profiling (i.e. using `db._profileQuery(...)` work for streaming queries as well. Previously, profiling a streaming query could have led to some internal errors, and even query results being returned, even though profiling a query should not return any query results. * Improved the wording for sharding options displayed in the web interface. Instead of offering `flexible` and `single`, now use the more intuitive `Sharded` and `OneShard` options, and update the help text for them. * EE only bugfix: On DisjointSmartGraphs that are used in anonymous way, there was a chance that the query could fail, if non-disjoint collections were part of the query. Named DisjointSmartGraphs have been save to this bug. Example: DisjointSmartGraph (graph) on vertices -edges-> vertices Query: WITH vertices, unrelated FOR out IN 1 OUTBOUND "v/1:1" edges FOR u IN unrelated RETURN [out, u] The "unrelated" collection was pulled into the DisjointSmartGraph, causing the AQL setup to create erroneous state. This is now fixed and the above query works. This query: WITH vertices, unrelated FOR out IN 1 OUTBOUND "v/1:1" GRAPH "graph" FOR u IN unrelated RETURN [out, u] was not affected by this bug. * Avoid a potential deadlock when dropping indexes. A deadlock could theoretically happen for a thread that is attempting to drop an index in case there was another thread that tried to create or drop an index in the very same collection at the very same time. We haven't managed to trigger the deadlock with concurrency tests, so it may have been a theoretical issue only. The underlying code was changed anyway to make sure this will not cause problems in reality. * Make dropping of indexes in cluster retry in case of precondition failed. When dropping an indexes of a collection in the cluster, the operation could fail with a "precondition failed" error in case there were simultaneous index creation or drop actions running for the same collection. The error was returned properly internally, but got lost at the point when `.dropIndex()` simply converted any error to just `false`. We can't make `dropIndex()` throw an exception for any error, because that would affect downwards-compatibility. But in case there is a simultaneous change to the collection indexes, we can just retry our own operation and check if it succeeds then. This is what `dropIndex()` will do now. * Improve incremental sync replication for single server and cluster to cope with multiple secondary index unique constraint violations (before this was limited to a failure in a single unique secondary index). This allows replicating the leader state to the follower in basically any order, as any *other* conflicting documents in unique secondary indexes will be detected and removed on the follower. * Fix potential undefined behavior when iterating over connected nodes in an execution plan and calling callbacks for each of the nodes: if the callbacks modified the list of connected nodes of the current that they were called from, this could lead to potentially undefined behavior due to iterator invalidation. The issue occurred when using a debug STL via `_GLIBCXX_DEBUG`. * Fixed a RocksDB bug which could lead to an assertion failure when compiling with STL debug mode -D_GLIBCXX_DEBUG. * Fixed a rare internal buffer overflow around ridBuffers. * Issue #13141: The `move-filters-into-enumerate` optimization, when applied to an EnumerateCollectionNode (i.e. full collection scan), did not do regular checks for the query being killed during the filtering of documents, resulting in the maxRuntime option and manual kill of a query not working timely. * Do not create index estimator objects for proxy collection objects on coordinators and DB servers. Proxy objects are created on coordinators and DB servers for all shards, and they also make index objects available. In order to reduce the memory usage by these objects, we don't create any index estimator objects for indexes in those proxy objects. Index estimators usually take several KB of memory each, so not creating them will pay out for higher numbers of collections/shards. * Improvements for logging. This adds the following startup options to arangod: - `--log.max-entry-length`: controls the maximum line length for individual log messages that are written into normal logfiles by arangod (note: this does not include audit log messages). Any log messages longer than the specified value will be truncated and the suffix '...' will be added to them. The purpose of this parameter is to shorten long log messages in case there is not a lot of space for logfiles, and to keep rogue log messages from overusing resources. The default value is 128 MB, which is very high and should effectively mean downwards-compatiblity with previous arangod versions, which did not restrict the maximum size of log messages. - `--audit.max-entry-length`: controls the maximum line length for individual audit log messages that are written into audit logs by arangod. Any audit log messages longer than the specified value will be truncated and the suffix '...' will be added to them. The default value is 128 MB, which is very high and should effectively mean downwards-compatiblity with previous arangod versions, which did not restrict the maximum size of log messages. - `--log.in-memory-level`: controls which log messages are preserved in memory. The default value is `info`, meaning all log messages of types `info`, `warning`, `error` and `fatal` will be stored by an instance in memory (this was also the behavior in previous versions of ArangoDB). By setting this option to `warning`, only `warning` log messages will be preserved in memory, and by setting the option to `error` only error messages will be kept. This option is useful because the number of in-memory log messages is limited to the latest 2048 messages, and these slots are by default shared between informational, warning and error messages. * Honor the value of startup option `--log.api-enabled` when set to `false`. The desired behavior in this case is to turn off the REST API for logging, but was not implemented. The default value for the option is `true`, so the REST API is enabled. This behavior did not change, and neither did the behavior when setting the option to a value of `jwt` (meaning the REST API for logging is only available for superusers with a valid JWT token). * Fix error reporting in the reloadTLS route. * Split the update operations for the _fishbowl system collection with Foxx apps into separate insert/replace and remove operations. This makes the overall update not atomic, but as removes are unlikely here, we can now get away with a simple multi-document insert-replace operation instead of a truncate and an exclusive transaction, which was used before. v3.7.8 (2021-02-16) ------------------- * Fixed ES-784 regression related to encryption cipher propagation to ArangoSearch data. v3.7.7 (2021-02-05) ------------------- * Added metrics for document read and write operations: - `arangodb_document_writes: Total number of document write operations (successful and failed) not performed by synchronous replication. - `arangodb_document_writes_replication`: Total number of document write operations (successful and failed) by cluster synchronous replication. - `arangodb_collection_truncates`: Total number of collection truncate operations (successful and failed) not performed by cluster synchronous replication. - `arangodb_collection_truncates_replication`: Total number of collection truncate operations (successful and failed) by synchronous replication. - `arangodb_document_read_time`: Execution time histogram of all document primary key read operations (successful and failed) [s]. Note: this does not include secondary index lookups, range scans and full collection scans. - `arangodb_document_insert_time`: Execution time histogram of all document insert operations (successful and failed) [s]. - `arangodb_document_replace_time`: Execution time histogram of all document replace operations (successful and failed) [s]. - `arangodb_document_remove_time`: Execution time histogram of all document remove operations (successful and failed) [s]. - `arangodb_document_update_time`: Execution time histogram of all document update operations (successful and failed) [s]. - `arangodb_collection_truncate_time`: Execution time histogram of all collection truncate operations (successful and failed) [s]. The timer metrics are turned off by default, and can be enabled by setting the startup option `--server.export-read-write-metrics true`. * Fixed issue #12543: Unused Foxx service config can not be discarded. * Fixed issue #12363: Foxx HTTP API upgrade/replace always enables development mode. * Fixed BTS-284: upgrading from 3.6 to 3.7 in cluster enviroment. Moved upgrade ArangoSearch links task to later step as it needs cluster connection. Removed misleading error log records for failed ArangoSearch index creation during upgrade phase. * Normalize user-provided input/output directory names in arangoimport, arangoexport, arangodump and arangorestore before splitting them into path components, in the sense that now both forward and backward slashes can be used on Windows, even interchangingly. * Fixed some wrong behaviour in single document updates. If the option ignoreRevs=false was given and the precondition _rev was given in the body but the _key was given in the URL path, then the rev was wrongly taken as 0, rather than using the one from the document body. * Allow {USER} paceholder string also in `--ldap.search-filter`. * Make `padded` and `autoincrement` key generators export their `lastValue` values, so that they are available in dumps and can be restored elsewhere from a dump. * Fix decoding of values in `padded` key generator when restoring from a dump. * Fixed error reporting for hotbackup restore from dbservers back to coordinators. This could for example swallow out of disk errors during hotbackup restore. * Fix decoding of values in `padded` key generator when restoring from a dump. * Fixed some situations of [...] SUBQUERY FILTER LIMIT [...] in AQL queries, yielding incorrect responses. A distributed state within the subquery was not resetted correctly. This could also lead into "shrink" errors of AQL item blocks, or much higher query runtimes. Fixes: - BTS-252 - ES-687 - github issue: #13099 - github issue: #13124 - github issue: #13147 - github issue: #13305 - DEVSUP-665 * Fix a bug in the agency Supervision which could lead to removeFollower jobs constantly being created and immediately stopped again. * Limit additional replicas in failover cases to +2. * Prepare register planning for rolling upgrades. Previously, changes in register planning from 3.7 to a minor future version (i.e. 3.8) could cause queries executed by a 3.7 coordinator in combination with a minor future version (i.e. 3.8) DBServer to fail during a rolling upgrade. * Fixed rare objectId conflict for indexes. * Fix for OASIS-409: fixed indexing _id attribute at recovery. * Fix some issues with key generators not properly taking into account the `allowUserKeys` attribute when in a cluster. * Added the following bit handling functions to AQL: - BIT_AND(array): and-combined result - BIT_OR(array): or-combined result - BIT_XOR(array): xor-combined result - BIT_NEGATE(value, bits): bitwise negation of `value`, with a mask of `bits` length - BIT_TEST(value, index): test if bit at position `index` is set in `value` (indexes are 0-based) - BIT_POPCOUNT(value): return number of bits set in `value` - BIT_SHIFT_LEFT(value, shift, bits): bitwise shift-left of `value` by `shift` bits, with a mask of `bits` length - BIT_SHIFT_RIGHT(value, shift, bits): bitwise shift-right of `value` by `shift` bits, with a mask of `bits` length - BIT_CONSTRUCT(array): construct a number with bits set at the positions given in the array - BIT_DECONSTRUCT(value): deconstruct a number into an array of its individual set bits - BIT_TO_STRING(value): create a bitstring representation from numeric `value` - BIT_FROM_STRING(value): parse a bitstring representation into a number `BIT_AND`, `BIT_OR` and `BIT_XOR` are also available as aggregate functions for usage inside COLLECT AGGREGATE. All above bit operations support unsigned integer values with up to 32 bits. Using values outside the supported range will make any of these bit functions return `null` and register a warning. * Add binary (base 2) and hexadecimal (base 16) integer literals to AQL. These literals can be used where regular (base 10) integer literal can used. The prefix for binary integer literals is `0b`, e.g. `0b10101110`. The prefix for hexadecimal integer literals i `0x`, e.g. `0xabcdef02`. Binary and hexadecimal integer literals can only be used for unsigned integers. The maximum supported value is `(2 ^ 32) - 1`, i.e. `0xffffffff` (hexadecimal) or `0b11111111111111111111111111111111` (binary). * Print a version mismatch (major/minor version difference) between the arangosh version and the remote arangod version at arangosh startup. * Fix a potential shutdown deadlock in AgencyCache. * Updated arangosync to 1.2.2. * Minor and rare AQL performance improvement, in nested subqueries: LET sq1 ([..] FILTER false == true LET sq2 = () [..]) where sq1 produces no data (e.g. by the above filter) for sq2, the part have been asked two times (second returns empty result), instead of one, if and only if the mainquery executes sq1 exactly one time. Now we get away with one call only. In the case sq1 has data, or sq1 is executed more often, only one call was needed (assuming the data fits in one batch). * Improve internal error reporting by cluster maintenance. * Bug-Fix: In one-shard-database setups that were created in 3.6.* and then upgraded to 3.7.5 the DOCUMENT method in AQL will now return documents again. v3.7.6 (2021-01-04) ------------------- * Updated OpenSSL to 1.1.1i and OpenLDAP to 2.4.56. * Added new metric: "arangodb_collection_lock_sequential_mode" this will count how many times we need to do a sequential locking of collections. If this metric increases this indicates lock contention in transaction setup. Most likely this is caused by exlcusive locks used on collections with more than one shard. * Fix for BTS-213 Changed the transaction locking mechanism in the cluster case. For all installations that do not use "exclusive" collection locks this change will not be noticable. In case of "exclusive" locks, and collections with more than one shard, it is now less likely to get a LOCK_TIMEOUT (ErrorNum 18). It is still possible to get into the LOCK_TIMEOUT case, especially if the "exclusive" operation(s) are long-running. * Fixed an endless busy loop which could happen if a coordinator tries to roll back a database creation, but the database has already been dropped by other means. * Make internal ClusterInfo::getPlan() wait for initial plan load from agency. * Remove HTTP "Connection" header when forwarding requests in the cluster from one coordinator to another, and let the internal network layer handle closing of connections and keep-alive. * Prevent a write to RocksDB during recovery in the case that the database already exists. The write at startup is potentially blocking, and will delay the startup for servers that were shut down while in a write-stopped state. * Fix recovery of "clientId" values in Agency when restarting an agent from persistence. * Added "startupTime", "computationTime" and "storageTime" to Pregel result statistics. * Add query execution time and query id to audit log query messages. * Fixed issue #13238 Thread naming API on Windows are now used only if available in KERNEL32.DLL * Fix for issue #772: Optimized document counting for ArangoSearch views. Added new ArangoSearch view option 'countApproximate' for customizing view count strategy. * Fix ordering of FollowerInfo lists in maintainer mode. * Fix AR-113. Disallow non-values in the AQL geo-index-optimizer rule. * Added SNI support for arangosh. * Fix agency restart with mismatching compation and log indexes. * Improve performance and memory efficiency of agency restart from persisted database directory. * Added the following agency-related metrics: - `arangodb_agency_client_lookup_table_size`: current number of entries in agency client id lookup table. This gauge is available only on agent instances. - `arangodb_agency_cache_callback_count`: current number of entries in agency cache callbacks table. This gauge will be effective on coordinators and DB servers. - `arangodb_agency_callback_count`: current number of agency callbacks registered. This gauge will be effective on coordinators and DB servers. * Fix cluster-internal replication of documents with special keys (percent character, which has a special meaning when used inside URLs). * Improvements for the Pregel distributed graph processing feature: - during the loading/startup phase, the in-memory edge cache is now intentionally bypassed. The reason for this is that any edges are looked up exactly once, so caching them is not beneficial, but would only lead to cache pollution. - the loading/startup phase can now load multiple collections in parallel, whereas previously it was only loading multiple shards of the same collection in parallel. This change helps to reduce load times in case there are many collections with few shards, and on single server. - the loading and result storage phases code has been overhauled so that it runs slightly faster. - for Pregel runs that are based on named graphs (in contrast to explicit naming of the to-be-used vertex and edge collections), only those edge collections are considered that, according to the graph definition, can have connections with the vertex. This change can reduce the loading time substantially in case the graph contains many edge definitions. - the number of executed rounds for the underlying Pregel algorithm now does not vary for different `parallelism` values. * Reimplement coordshort request handler. The new implementation only runs two DB queries without any additional requests to other coordinators, resulting in reduced load on the cluster. Previously this involved requests to all coordinators, where each of them ran two DB queries. * When querying the list of currently running or slow AQL queries, ignore not-yet created databases on other coordinators. * Fix AQL cost estimate of spliced subqueries which could lead to overly large numbers in the explain output of such queries. * Add an AQL query kill check during early pruning. Fixes issue #13141. * Fix Windows directory creation error handling. * Added new metrics for tracking AQL queries and slow queries: * `arangodb_aql_query_time`: histogram with AQL query times distribution. * `arangodb_aql_slow_query_time`: histogram with AQL slow query times distribution. * Reduce the number of dropped followers when running larger (>= 128 MB) write transactions. * Remove a case in which followers were dropped unnecessarily in streaming transactions that replicated to the same follower. * Added metrics for collection locks: - `arangodb_collection_lock_timeouts_exclusive`: Number of lock timeouts when trying to acquire collection exclusive locks - `arangodb_collection_lock_timeouts_write`: Number of lock timeouts when trying to acquire collection write locks - `arangodb_collection_lock_acquisition_micros`: Total amount of collection lock acquisition time [μs] - `arangodb_collection_lock_acquisition_time`: Total collection lock acquisition time histogram [s] * Reduce lock timeout on followers to 15 seconds. Rationale: we should not have any locking conflicts on followers, generally. Any shard locking should be performed on leaders first, which will then, eventually replicate changes to followers. replication to followers is only done once the locks have been acquired on the leader(s). * Better tracking of memory used in AQL graph traversals, COLLECT and SORT operations. From this version onwards, certain AQL queries can report a higher memory usage than in previous versions of ArangoDB. This is not because the queries use more memory than before, but because the memory usage tracking has been improved. A side effect of this change is that queries with a memory limit set may now be aborted whereas in previous versions they ran through successfully (but actually violated the limit). In this case it may be necessary to adjust (i.e. raise) query memory limits accordingly. * Added startup option `--foxx.force-update-on-startup` to toggle waiting for all Foxx services in all databases to be propagated to a coordinator before it completes the boot sequence. In case the option is set to `false` (i.e. no waiting), the coordinator will complete the boot sequence faster, and the Foxx services will be propagated lazily. Until the initialization procedure has completed for the local Foxx apps, any request to a Foxx app will be responded to with an HTTP 503 error and message waiting for initialization of Foxx services in this database This can cause an unavailability window for Foxx services on coordinator startup for the initial requests to Foxx apps until the app propagation has completed. When not using Foxx, this option should be set to `false` to benefit from a faster coordinator startup. Deployments relying on Foxx apps being available as soon as a coordinator is integrated or responding should set this option to `true` (which is the default value). The option only has an effect for cluster setups. On single servers and in active failover mode, all Foxx apps will be available from the very beginning. Note: ArangoDB 3.6 and 3.7 introduce this option with a default value of `true`. ArangoDB 3.8 changes the default value to `false`. * Changed the server-side implementation of the following internal JavaScript APIs to no-ops: * `internal.reloadAqlFunctions()`: this is a no-op function now * `@arangodb/actions.buildRouting()`: this is a no-op function now * `@arangodb/actions.routingTree`: will return an empty object * `@arangodb/actions.routingList`: will return an empty object All the above APIs were intended to be used for internal means only. These APIs are deprecated now and will be removed in ArangoDB v3.9. * Fix HTTP/1.1 status response header in fuerte responses This change makes fuerte return the full status header, including the numeric status code and the status string in the `http/1.1` header of fuerte responses. Previously, the return header lacked the numeric status code, so it looked like ``` "http/1.1" : "Ok" ``` Now, with the numeric status code, the response header will look like ``` "http/1.1" : "200 Ok" ``` This PR also adds a protocol() method for arango client connections in order to check the protocol in use. The possible return values are - "http" for HTTP/1.1 connections - "http2" for HTTP/2 connections - "vst" for VST connections - "unknown" for everyhting else This is needed during testing, but can also be used for other purposes. * Fixed bug in the connection pool which could prevent connection reusage under high load and lead to lots of new connection creations, in particular with TLS. * Added more metrics around connection pool. * Fix a potential nullptr access in AsyncAgencyComm in case there was a specific error when sending an agency request. * Clean up agency change log, cluster info caches. v3.7.5 (2020-12-09) ------------------- * Fixed ES-662 by introducing refactored thread pool to make more efficient consolidation and commit routines for links of ArangoSearch views. Added new command line options for fine-grained ArangoSearch maintenance control: - `--arangosearch.commit-threads` - max number of ArangoSearch commit threads - `--arangosearch.commit-threads-idle` - min number of ArangoSearch commit threads - `--arangosearch.consolidation-threads` - max number of ArangoSearch consolidation threads - `--arangosearch.consolidation-threads-idle` - min number of ArangoSearch consolidation threads Deprecated the following command line options: - `--arangosearch.threads` - will be used in --arangosearch.commit-threads` and `--arangosearch.consolidation-threads` as provided value divided by 2 if set to non-autodetect value > 0 - `--arangosearch.threads-limit` * Updated ArangoDB Starter to 0.14.15-1. * Fixed agency redirect in poll API. * Updated arangosync to 1.2.1. * Added support for fetching the list of currently running and slow AQL queries from all databases at once, by adding an `all` parameter to the following query APIs: * `require("@arangodb/aql/queries").current({ all: true })`: will return the currently running queries from all databases, not just the currently selected database. * HTTP GET `/_api/query/current?all=true`: same, but for the HTTP REST API. * `require("@arangodb/aql/queries").slow({ all: true })`: will return the slow query history from all databases, not just the currently selected database. * HTTP GET `/_api/query/slow?all=true`: same, but for the HTTP REST API. * `require("@arangodb/aql/queries").clearSlow({ all: true })`: will clear the slow query history for all databases, not just the currently selected database. * HTTP DELETE `/_api/query/slow?all=true`: same, but for the HTTP REST API. Using the `all` parameter is only allowed when making the call inside the `_system` database and with superuser privileges. * Fixed issue #12734: Accept HTTP headers into Foxx framework. * Fix Gauge class' assignment operators. * Clean up callback bin and empty promises in single-host-agency. * Fix an issue where a query would not return a result when the geo index was used. * Fix the activation of the agency supervision maintenance via the REST API `/_admin/cluster/maintenance`. This API stored a boolean value instead of an (expected) maintenance period end date/time string. * Make the cancel operation safe for asynchronoulsly started JavaScript transactions (via HTTP POST to `/_api/transaction` with the `x-arango-async` header set). * Fixed initial population of local AgencyCache values after a server restart. Previously the local cache was populated from the agency using a commit index value of 1, whereas it should have been 0 to get the full agency snapshot. * Updated OpenSSL to 1.1.1h. * Make the number of network I/O threads properly configurable via the startup option `--network.io-threads`. This option existed before, but its configured value was effectively clamped to a value of `1`. ArangoDB 3.7.5 thus also uses a default value of `1` for this option to remain compatible in terms of default option values. * Fix internal issue #777: Fixed memory access while substituting stored values for ArangoSearch view optimization. * Added new metric `arangodb_network_forwarded_requests` to track the number of requests forwarded from one coordinator to another in a load-balancing context. * Added new metric `arangodb_replication_cluster_inventory_requests` to track the number of requests received for cluster inventories. The cluster inventory API is called at the beginning of a dump process or by arangosync. * Added new AQL metrics: - `arangodb_aql_total_query_time_msec": Total execution time of all AQL queries (ms) - `arangodb_aql_all_query`: total number of all AQL queries * Added new metric `arangodb_aql_total_query_time_msec` to track the combined runtime of AQL queries (slow queries and non-slow queries). * Added more scheduler metrics: - `arangodb_scheduler_threads_started`: Total number of scheduler threads started - `arangodb_scheduler_threads_stopped`: Total number of scheduler threads stopped - `arangodb_scheduler_jobs_done`: Total number of scheduler queue jobs done - `arangodb_scheduler_jobs_submitted`: Total number of jobs submitted to the scheduler queue - `arangodb_scheduler_jobs_dequeued`: Total number of jobs dequeued from the scheduler queue - `arangodb_scheduler_num_working_threads`: Number of currently working scheduler threads * Added startup option `--server.unavailability-queue-fill-grade`. This option has a consequence for the `/_admin/server/availability` API only, which is often called by load-balancers and other availability probing systems. The `/_admin/server/availability` API will now return HTTP 200 if the fill grade of the scheduler's queue is below the configured value, or HTTP 503 if the fill grade is above it. This can be used to flag a server as unavailable in case it is already highly loaded. The default value for this option is `1`, which will mean that the availability API will start returning HTTP 503 responses in case the scheduler queue is completely full. This is mostly compatible with previous versions of ArangoDB. Previously the availability API still returned HTTP 200 in this situation, but this can be considered a bug, because the server was effectively totally overloaded. To restore 100% compatible behavior with previous version, it is possible to set the option to a value of `0`, which is a special value indicating that the queue fill grade will not be honored. To prevent sending more traffic to an already overloaded server, it can be sensible to reduce the default value to `0.75` or even `0.5`. This would mean that instances with a queue longer than 75% (or 50%, resp.) of their maximum queue capacity would return HTTP 503 instead of HTTP 200 when their availability API is probed. nb: the default value for the scheduler queue length is 4096. * Fixed bug with ArangoSearch views on SmartGraph edge collections which could contain some documents twice. This change removes `_to_*` local auxiliary link creation and existence within a view linked with a SmartGraph edge collection. * Fixed an AQL bug that ignored PRUNE statements in OneShard setups. * Added arangobench options: `--create-database` to create the test database on start `--duration` to run test for a duration rather than a defined count * Fixed a deadlock between AQL write transactions and hotbackup, since in AQL write transactions follower transactions did not know they are follower transactions. * Make the DOCUMENT AQL function eligible for running on DB servers in OneShard deployment mode. This allows pushing more query parts to DB servers for execution. * Fix REST API endpoint PUT `/_api/collection//recalculateCount` on coordinators. Coordinators sent a wrong message body to DB servers here, so the request could not be handled properly. * Fixed issue #12778: fails validation if additionalProperties: false. * Added missing exceptions catch clause for some parts of supervision and heartbeat threads. * Fixed potential deadlock in cluster transactions if a transaction is returned that was soft-aborted by transaction garbage collection before. This deadlock should rarely ever occur in practice, as it can only be triggered once during the server shutdown sequence. * Fix a memory leak because server internal connections were not cleaned up for agency communication. * Added compile option USE_JEMALLOC_PROF to enable memory profiling. * Fixed BTS-233 issue: Fixed invalid IndexId comparator. * Fixed very spurious errors if the `holdReadLockCollection` replication API for the getting-in-sync procedure of shards was called during server shutdown. In this case that method could ask the transaction manager for a specific transaction, but wasn't returning one due to the server shutdown. * Agency cache clears change history. This keeps the change history, introduced in v3.7.4, from growing in size too much. * Bug-fix: Allow to unlink a view created on a SmartGraphEdge collection. * If a collection (or database) is dropped during the instantiation of an AQL query, the setup code now aborts with an ERROR_QUERY_COLLECTION_LOCK_FAILED and earlier. Before the setup code could abort with TRI_ERROR_INTERNAL in the same case. * Bug-fix: Creating an additional index on the edge collection of a disjoint SmartGraph could falsely result in an error: `Could not find all smart collections ...` This is now ruled out and indexes can be created as expected. * Fixed issue #12248: Web UI - Added missing HTML escaping in the setup script section of a foxx app. * Add parameter so `db.collection.truncate({compact: false})` will stop compaction from happening. Compaction may have performance impacts even if the truncate was invoked on nearly empty collections. * Instead of failing to connect to INADDR_ANY refuse it as a parameter, with a descriptive error message for novice users (issue #12871). * Fixed collection count which could be off after a server crash. v3.7.4 (2020-10-16) ------------------- * Data definition reconciliation in cluster has been modified extensively to greatly accelerate the creation of 1000s of databases through following means: - AgencyCache offers change sets API based on Raft index. - ClusterInfo caches are only updated using change sets. - Maintenance uses local as well as agency change sets to limit the scope of every runtime to these change sets. v3.7.3 (2020-10-14) ------------------- * Added the following metrics for synchronous replication in the cluster: - `arangodb_refused_followers_count`: Number of times a shard leader received a refusal answer from a follower during synchronous replication. - `arangodb_sync_wrong_checksum`: Number of times a mismatching shard checksum was detected when syncing shards. In case this happens, a resync will be triggered for the shard. * Fixed handling of failedLeaderJob. In case of a plan modification, that removes a server from the plan, e.g. reduce replication factor. Directly followed by a failure of the current shard leader, would reinsert the just removed server in the plan, which is undesired, we first need to have a full "desync" cycle on this server to be reusable in the plan again. * Make sure the optimizer doesn't pick another index than the TTL index itself while fulfilling the expiry of TTL. * Added optional verbose logging for agency write operations. This logging is configurable by using the new log topic "agencystore". The following log levels can be used for for the "agencystore" log topic to log writes to the agency: - DEBUG: will log all writes on the leader - TRACE: will log all writes on both leaders and followers The default log level for the "agencystore" log topic is WARN, meaning no agency writes will be logged. Turning on this logging can be used for auditing and debugging, but it is not recommended in the general case, as it can lead to large amounts of data being logged, which can have a performance impact and will lead to higher disk space usage. * Print image base address and CPU context (if available) in crash handler messages. * Added configuration option `--query.tracking-slow-queries` to decide whether slow queries are tracked extra. * Added configuration option `--query.tracking-with-querystring` to decide whether the query string is shown in the slow query log and the list of currently running queries. The option is true by default. When turned off, querystrings in the slow query log and the list of currently running queries are just shown as "". * Added configuration option `--query.tracking-with-datasources` to toggle whether the names of data sources used by queries are shown in the slow query log and the list of currently running queries. The option is false by default. When turned on, the names of data sources used by the query will be shown in the slow query log and the list of currently running queries. * Fixed handling of failoverCandidates. Sometimes, a server can still be a failoverCandidate even though it has been taken out of the Plan. With this fix, such a server is quickly taken out of failoverCandidates and it can never be re-added to the Plan before this has happened. * Fix #12693: SORT inside a subquery could sometimes swallow part of its input when it crossed boundaries of internal row batches. * Fixed issue BTS-212: Web UI doesn't let to make partial view update and partial view update should be audited (also reported as ES-700). Fixed link definition comparison logic: equality wasn`t properly detected and led to link recreation. * Added configuration option `--rocksdb.sync-delay-threshold`. This option can be used to track if any RocksDB WAL sync operation is delayed by more than the configured value (in milliseconds). The intention is to get aware of severely delayed WAL sync operations. * Add database, shard name and error information to several shard-related log messages. * Display shard names of a collection in the web interface when in the details view of the collection. * Added HTTP requests metrics for tracking the number of superuser and normal user requests separately: - `arangodb_http_request_statistics_superuser_requests`: Total number of HTTP requests executed by superuser/JWT - `arangodb_http_request_statistics_user_requests`: Total number of HTTP requests executed by clients * Fixed a bug in handling of followers which refuse to replicate operations. In the case that the follower has simply been dropped in the meantime, we now avoid an error reported by the shard leader. * Fix a performance regression when a LIMIT is combined with a COLLECT WITH COUNT INTO. Reported in ES-692. * Fix REST handler GET /_admin/status when called with URL parameter value `overview=true`. For generating the `hash` attribute in the response, the current Plan was retrieved and analyzed. Due to a change in the internal Plan format the REST handler code failed to pick up the number of servers, which resulted in the REST handler returning HTTP 500 in cluster mode. * Use rclone built from v1.51.0 source with go1.15.2 instead of prebuilt v1.51.0 release. * Fixed a bug in AQL COLLECT with OPTIONS { "hash" } that led to a quadratic runtime in the number of output rows. * Added startup option `--database.old-system-collections` to toggle automatic creation of system collections `_modules` and `_fishbowl`, along with their internal usage. These collections are useful only in very few cases, so it is normally not worth to create them in all databases. The `_modules` collection is only used to register custom JavaScript modules, for which there exists no API, and `_fishbowl` is used to store the temporary list of Foxx apps retrieved from the GitHub Foxx store. If the option value is `false` (which is the default from v3.8 onwards, but for v3.7 the default value is `true` for downwards-compatibility), the two collections will not be created for any new database. The `_fishbowl` collection will still be created dynamically when needed. If the option value is `true` (the default value in v3.7), the collections will be created regularly as before. The default value for the option is going to change to `false` in v3.8, meaning the collections will not be created anymore there by default. Any functionality related to the `_modules` system collection is deprecated and will be removed in ArangoDB v3.9. Two side effects of turning this option off are: * there will no be iteration over all databases at server startup just to check the contents of all `_modules` collections. * less collections/shards will be around for deployments that create a large number of databases. Already existing `_modules` and `_fishbowl` system collections will not be modified by this PR, even though they will likely be empty and unused. * Don't iterate over all databases at server startup in order to initialize the routing information. This is not necessary, as the routing information is global and not tied to a specific database. * Fixed a possible crash during instantiation of an AQL graph traversal. Reported in #12597. * Added safeguards against using V8 internally in environments that have JavaScript turned off via the `--javascript.enabled false` option. * Make scheduler properly count down the number of working threads in case an exception happens in a worker thread. * Turn off upgrade checks in arangod in alpha/beta/preview Enterprise builds, too. Previously it was already turned off in arangod for Enterprise builds already, but only for stable releases and not preview releases. * Fixed and extended LDAP log messages. * Added LDAP_OFF if referrals and restart are false. * If LDAP search fails, also retry (update to given number of retries). * Fixed infinite reload of the login window after logout of an LDAP user. * Make the reboot tracker catch failed coordinators, too. Previously the reboot tracker was invoked only when a DB server failed or was restarted, and when a coordinator was restarted. Now it will also act if a coordinator just fails (without restart). * Added scheduler thread creation/destruction metrics: - `arangodb_scheduler_threads_started`: Number of scheduler threads started - `arangodb_scheduler_threads_stopped`: Number of scheduler threads stopped * Added startup option `--query.max-runtime` to limit the maximum runtime of all AQL queries to a specified threshold value (in seconds). By default, the threshold is 0, meaning that the runtime of AQL queries is not limited. Setting it to any positive value will restrict the runtime of all AQL queries unless it is overwritten in the per-query "maxRuntime" query option. Please note that setting this option will affect *all* queries in all databases, and also queries issues for administration and database-internal purposes. If a query exceeds the configured runtime, it will be killed on the next occasion when the query checks its own status. Killing is best effort, so it is not guaranteed that a query will no longer than exactly the configured amount of time. * Ensure that the argument to an AQL OPTIONS clause is always an object which does not contain any dynamic (run-time) values. Previously, this was only enforced for traversal options and options for data-modification queries. This change extends the check to all occurrences of OPTIONS. * Added `details` option to figures command of a collection: `collection.figures(details)` Setting `details` to `true` will return extended storage engine-specific details to the figures. The details are intended for debugging ArangoDB itself and their format is subject to change. There is not much use in using the details from a client application. By default, `details` is set to `false`, so no details are returned and the behavior is identical to previous versions of ArangoDB. * Implement RebootTracker usage for AQL queries in case of coordinator restarts or failures. This will clean up the rest of an AQL query on dbservers more quickly and in particular release locks faster. * Serialize maintenance actions for each shard. This addresses lost document problems found in chaos testing. * Enforce a maximum result register usage limit in AQL queries. In an AQL query, every user-defined or internal (unnamed) variable will need a register to store results in. AQL queries that use more result registers than allowed (currently 1000) will now abort deterministically during the planning stage with error 32 (`resource limit exceeded`) and the error message "too many registers (1000) needed for AQL query". Before this fix, an AQL query that used more than 1000 result registers crashed the server when assertions were turned on, and the behavior was undefined when assertions were turned off. * Fixed some cases where subqueries in PRUNE did not result in a parse error, but either in an incomprehensible error (in 3.7), or undefined behavior during execution (pre 3.7). * Fixed an issue with audit logging misreporting some document requests as internal instead of logging the proper request information. * Add attributes `database` and `user` when tracking current and slow AQL queries. `database` contains the name of the database the query is/was running in, `user` contains the name of the user that started the query. These attributes will be returned in addition when calling the APIs for current and slow query inspection: * GET `/_api/query/current` and `require("arangodb/aql/queries").current()` * GET `/_api/query/slow` and `require("arangodb/aql/queries").slow()` The "slow query" log message has also been augmented to contain the database name and the user name. The `user` attribute is now also displayed in the web interface in the "Running queries" and "Slow queries" views. * Added metrics for V8 contexts usage: * `arangodb_v8_context_alive`: number of V8 contexts currently alive. * `arangodb_v8_context_busy`: number of V8 contexts currently busy. * `arangodb_v8_context_dirty`: number of V8 contexts currently dirty. * `arangodb_v8_context_free`: number of V8 contexts currently free. * `arangodb_v8_context_max`: maximum number of concurrent V8 contexts. * `arangodb_v8_context_min`: minimum number of concurrent V8 contexts. * Updated arangosync to 0.7.11. * Make followers in active failover run a compaction after they process a truncate operation and the truncate removed more than 4k documents. This can help to reclaim disk space on the follower earlier than without running the truncate. * The REST API PUT `/_api/collection//truncate` will now also run a compaction if the truncation affected more than 4k documents. This may add extra latency to the truncate operation, but can help to reclaim disk space earlier. * Added REST API PUT `/_admin/compact` for compacting the entire database data. This endpoint can be used to reclaim disk space after substantial data deletions have taken place. The command is also exposed via the JavaScript API as `db._compact();`. This command can cause a full rewrite of all data in all databases, which may take very long for large databases. It should thus only be used with care and only when additional I/O load can be tolerated for a prolonged time. This command requires superuser access and is only available for the RocksDB storage engine. * Don't allow creation of smart satellite graphs or collections (i.e. using `"isSmart":true` together with `"replicationFactor":"satellite"` when creating graphs or collections. This combination of parameters makes no sense, so that the server will now respond with "bad parameter" and an HTTP status code of HTTP 400 ("Bad request"). * Add exit code for ICU database loading startup errors. * Fixed issue #12507: SegFault when using an AQL for loop through edges. * Make the `IS_IPV4` AQL function behave identical on macOS as on other platforms. It previously allowed leading zeros in octets on macOS, whereas on other platforms they were disallowed. Now this is disallowed on macOS as well. * Added new metric "arangodb_aql_slow_query" for slow AQL queries, so this can be monitored more easily. * Added new metric "arangodb_scheduler_queue_length" for the scheduler's internal queue length. * Added new metric "arangodb_scheduler_queue_full_failures" for tracking cases of a full scheduler queue and dropping requests. * Added new metrics for the number of V8 contexts dynamically created and destroyed ("arangodb_v8_context_created" and "arangodb_v8_context_destroyed") and for the number of times a V8 context was entered and left ("arangodb_v8_context_entered" and "arangodb_v8_context_exited"). There is also a new metric for tracking the cases when a V8 context cannot be successfully acquired and an operation is not performed ("arangodb_v8_context_enter_failures"). * Added extra info to "queue full" and "giving up waiting for unused v8 context" log messages. * Request to the `/_admin/statistics` API now processed via the CLIENT_FAST lane. Previously they were handled in the CLIENT_SLOW lane, meaning that monitoring requests using that API didn't get through when the queue was rather full. * Introduce an internal high-water mark for the maximum row number that was written to in an AqlItemBlock. Using this number several operations on the whole block, such as cleaning up or copying can be made more efficient when run on only partially filled blocks. * Fixed issue BTS-169: cost estimation for LIMIT nodes showed wrong number of estimated items. v3.7.2.2 (2020-10-07) --------------------- * Fixed issue ES-664: SEARCH vs FILTER lookup performance. Consolidation functionality for ArangoSearch view links was able to hit non- mergable enormous amount of segments due to improper scheduling logic. * Fix for issue BTS-183: added pending operations purging before ArangoSearch index truncation. * Fixed: More cases in AQL can now react to a query being killed, so reaction time to query abortion is now shortened. This was a regression in comparison to 3.6 series. * Fixed internal issue #741: STARTS_WITH fails to accept 'array' as variable. * Fixed internal issue #738: PHRASE doesn't accept a reference to an array of arguments. * Fixed internal issue #747: fixed possible dangling open files in ArangoSearch index after remove operations. v3.7.2.1 (2020-09-02) --------------------- * Add option `--rocksdb.max-write-buffer-size-to-maintain` with default of 0. This configures how much memory RocksDB is allowed to use for immutable flushed memtables/write-buffers. The default of 0 will usually be good for all purposes and restores the 3.6 memory usage for write-buffers. v3.7.2 (2020-08-21) ------------------- * Fixed internal issue #744: LIMIT with only offset and constrained heap optimization will use estimation value for ArangoSearch views. * Make UPSERT statement with collection bind parameter behave identical to its non-bind parameter counterpart. For example, the query FOR d IN ["key_1", "key_2", "key_3", "key_1"] UPSERT d INSERT d UPDATE d IN @@collection would fail with "unique constraint violation" when used with a collection bind parameter, but the equivalent query FOR d IN ["key_1", "key_2", "key_3", "key_1"] UPSERT d INSERT d UPDATE d IN collectionName with a hard-coded collection name would succeed. This is now fixed so both queries have the same behavior (no failure) in single server. * Updated arangosync to 0.7.10. * Fixed internal issue #742: Add tick storage in index meta payload for ArangoSearch view links after collection truncate operation. * Fixed issue #12304: insert in transaction causing com.arangodb.ArangoDBException: Response: 500, Error: 4 - Builder value not yet sealed. This happened when too deeply-nested documents (more than 63 levels of nesting) were inserted. While indefinite nesting is still not supported, the error message has been corrected from the internal HTTP 500 error "Builder value not yet sealed" to the correct HTTP 400 "Bad parameter". * Fixed: During a move-shard job which moves the leader there is a situation in which the old owner of a shard can reclaim ownership (after having resigned already), with a small race where it allows to write documents only locally, but then continue the move-shard to a server without those documents. An additional bug in the MoveShard Supervision job would then leave the shard in a bad configuration with a resigned leader permanently in charge. * Fixed a problem with potentially lost updates because a failover could happen at a wrong time or a restarted leader could come back at an unlucky time. * Fixed BTS-167: Lingering Queries - Canceled from the UI. This fixes queries not vanishing from the list of running queries in the web UI in case the query was canceled using the "Cancel" button in web UI's query editor. * Fixed bad behavior in agency supervision in some corner cases involving already resigned leaders in Current. * Fixed issue ES-664: the optimizer rule `inline-subqueries` must not pull out subqueries that contain a COLLECT statement if the subquery is itself called from within a loop. Otherwise the COLLECT will be applied to the values in the outer FOR loop, which can produce a different result. * Fixed that dropping a vanished follower works again. An exception response to the replication request is now handled properly. * Make AQL user-defined functions (UDFs) work in a cluster in case the UDF runs an AQL query inside its own function code (BTS-159). * Fixed hotbackup S3 credentials validation and error reporting for upload and download. * Fixed a blockage on hotbackup when writes are happening concurrently, since followers could no longer replicate leader transactions. * Fix: The 'sorted' COLLECT variant would return undefined instead of null when grouping by a null value. * Fix: writeConcern is now honored correctly (ES-655). * Fixed internal issue #739: ArangoSearch filter volatility takes into account calculation nodes dependencies. * Fixed OASIS-278 issue: Added proper sort/calc nodes cleanup for late materialzation after OneShard optimization. * Slightly improve the performance of cluster DDL maintenance operations. * Added AQL functions `IS_IPV4`, `IPV4_TO_NUMBER`, `IPV4_FROM_NUMBER` for IPv4 address checks and conversions. * Added AQL function `PRODUCT` to calculate the product of an array. * Improve performance of internal cluster Plan and Current reload operations. * Fixed issue #12349: arangosh compact Arangoerror 404. * Improve performance of many non-subquery AQL queries, by optimizing away some storage overhead for subquery context data. * Always fetch data for /_api/cluster/agency-dump from leader of the agency. Add option "redirectToLeader=true" to internal /_api/agency/state API. * Fixed regression with view use-after-create in cluster (BTS-137). * Slightly improved the performance of some k-shortest-path queries. v3.7.1 (2020-08-07) ------------------- * Fixed issue #12297: ArangoDB 3.6.5 Swagger Error? This issue caused the Swagger UI for displaying the APIs of user-defined Foxx services to remain invisible in the web UI, because of a JavaScript exception. This PR fixes the JavaScript exception, so the services API is displayed again properly. * Fixed issue #12248: Web UI on 3.6.5: 404 error on adding new index. This issue caused "404: not found" errors when creating indexes via the web UI. The indexes were created successfully despite the error message popping up. This fix removes the misleading unconditional error message. v3.7.1-rc.1 (2020-07-24) ------------------------ * Added startup option `--rocksdb.encryption-key-rotation` to activate/deactivate the encryption key rotation REST API. The API is disabled by default. * Add internal caching for LogicalCollection objects inside ClusterInfo::loadPlan. This allows avoiding the recreation of LogicalCollection objects that did not change from one loadPlan run to the next. It reduces CPU usage considerably on both Coordinators and DB-servers. * Fixed reading analyzers revisions for freshly updated cluster. * Fixed undefined behavior in AQL COLLECT with multiple group variables (issue #12267). If you are grouping on "large" values that occur multiple times in different groups, and two different groups with the same large value are written to different batches in the output then the memory could be invalid. e.g. the following query is affected: ``` FOR d IN documents FOR batchSizeFiller IN 1..1001 COLLECT largeVal = d.largeValue, t = batchSizeFiller RETURN 1 ``` * Revive faster out-of-range comparator for secondary index scans that do a full collection index scan for index types "hash", "skiplist", "persistent". * Conflict error codes (1200) will now use the proper error message instead of a generic and misleading "precondition failed". * Improve performance of agency cache by not copying the hierarchical Node tree result for serialization, but serializing it directly. * Turn off maintenance threads on Coordinators, as they are not needed there. * Fixed internal issue #733: Primary sort compression in ArangoSearch views now used properly. * Change HTTP response code for error 1450 ("too many shards") from HTTP 500 to HTTP 400, as this is clearly a client error. * Fix spurious lock timeout errors when restoring collections. * Make sure cluster statistics in web UI work in case a coordinator is down. * Updated arangosync to 0.7.8. * Fixed a race between a new request and the keepAlive timeout. * Fixed hotbackup upload and download with encryption at rest key indirection. * Fixed crash in cleanup of parallel traversal queries. * Added cluster metrics `arangodb_load_plan_accum_runtime_msec` and `arangodb_load_current_accum_runtime_msec` to track the total time spent in `loadPlan()` and `loadCurrent()` operations. * Fixed wrong reporting of failures in all maintenance failure counter metrics (`arangodb_maintenance_action_failure_counter`). Previously, each successful maintenance operation was reported as a failure, so the failure counters actually were counters for the number of successful maintenance actions. * Adjusted the scale of the `arangodb_maintenance_action_queue_time_msec` to cover a more useful range. * The filter executor will now overfetch data again if followed by a limit, same as in 3.6 series. The following queries are effected: ``` something FILTER a == b LIMIT 10 ``` something will now be asked for a full batch, instead of only 10 documents. * In rare cases SmartBFS could use a wrong index for looking up edges. This is fixed now. * The internally used JS-based ClusterComm send request function can now again use JSON, and does not require VelocyPack anymore. This fixes an issue with Foxx-App management (install, updated, remove) got delayed in a sharded environment, all servers do get all apps eventually, now the fast-path will work again. * Fixed a rare race in Agents, if the leader is rebooted quickly there is a chance that it is still assumed to be the leader, but delivers a state shortly in the past. * Keep the list of last-acknowledged entires in Agency more consistent. During leadership take-over it was possible to get into a situation that the new leader does not successfully report the agency config, which was eventually fixed by the Agent itself. Now this situation is impossible. * Fixed a race in the ConnectionPool which could lease out a connection that got its idle timeout after the lease was completed. This could lead to sporadic network failures in TLS and to inefficiencies with TCP. * Fixed restoring a SmartGraph into a database that already contains that same graph. The use case is restoring a SmartGraph from backup, apply some modifications, which are undesired, and then resetting it to the restored state, without dropping the database. One way to achieve this is to use arangorestore with the `overwrite` option on the same dataset, effectively resetting the SmartGraph to the original state. Without this fix, the workaround for is to either drop the graph (or the database) before the restore call, yielding an identical result. * Fixed potential garbled output in syslog log output for the program name. * Fixed infinite recursion when printing error messages about invalid logger configuration on startup. * Fixed sporadic use-after-free ASan issue in logger shutdown. * Added missing state rollback for failed attempt-based write locking of spin locks. * Disable internal network protocol switch for cluster-internal communication, and hard-code the internal communication protocol to HTTP. * Added vertex collection validation in case of a SmartGraph edge definition update. * Fix cluster-internal request forwarding for VST requests that do not have any Content-Type header set. Such requests could have been caused by the Java driver (ES-635). * Fixed issue OASIS-252: Hotbackup agency locks without clientId. * Fixed internal-issue #726: added restore handling for custom analyzers. * Fixed a potential agency crash if trace logging is on. * Network layer now reports connection setup issues in more cases this replaces some INTERNAL_ERROR reports by more precise errors, those are only reached during failover scenarios. * Fixed a document parsing bug in the Web UI. This issue occurred in the document list view in case a document had an attribute called `length`. The result was an incorrect representation of the document preview. * Re-enable access to GET `/_admin/cluster/numberOfServers` for all users by default. Requests to PUT `/_admin/cluster/numberOfServers` require admin user privileges. This restores the pre-3.7 behavior. In contrast to pre-3.7 behavior, the `--server.harden` option now can be used to restrict access to GET `/_admin/cluster/numberOfServers` to admin users, too. This can be used to lock down this API for non-admin users entirely. * Fixed crash in HTTP2 implementation. * Improve readability of running and slow queries in web UI by properly left- aligning the query strings. * ClusterInfo will wait for syncer threads to shutdown. * Correct some log entries. * Allow changing collection properties for smart edge collections as well. Previously, collection property changes for smart edge collections were not propagated. * Fixed BTS-110: Fulltext index with minLength <= 0 not allowed. * Web UI: Removed unnecessary menubar entry in case of database node inspection. * Adjust arangodump integration test to desired behavior, and make sure arangodump behaves as specified when invoking it with non-existing collections. * Disallow using V8 dependent functions in SEARCH statement. * Remove superfluous `%>` output in the UI modal dialog in case the JSON editor was embedded. * The Web UI now displays an error message inside the node information view in case the user has no access to retrieve the neccessary information. * The `_from` and `_to` attributes of an edge document can now be edited from within the UI. * Fixed a misleading error message in AQL. * Fixed issue ES-609: "Transaction already in use" error when running transaction. Added option `--transaction.streaming-lock-timeout` to control the timeout in seconds in case of parallel access to a streaming transaction. * Fixed internal issue BTS-107, offset over the main query passed through a subquery which has modification access to shards could yield incorrect results, if shards are large enough and skipping was large enough, both to overflow the batch-size in AQL for each shard individually. The following query would be affected: FOR x IN 1..100 LET sub = ( FOR y IN Collection UPDATE y WITH {updated: true} IN Collection RETURN new ) LIMIT 5, 10 RETURN {x,sub} If a shard in Collection, has enough entries to fill a batch, the second shard could run into the issue that it actually does not skip the first 5 main queries, but reports their results in addition. This has the negative side effect that merging the subqueries back together was off. * Fix undistribute-remove-after-enum-coll which would allow calculations to be pushed to a DBServer which are not allowed to run there. * Removed the edge id's hover styling in case of embedded document editor in the Web UI as this functionality is disabled. This was misleading because the elements are not clickable. * Internal issue BTS-71: Fixed error handling regarding communication with the agency. This could in a specific case cause collection creation in a cluster report success when it failed. * Internal issue BTS-71: In a cluster, for collections in creation, suspend supervision jobs concerning replication factor until creation is completed. Previously, it could cause collection creation to fail (e.g. when a server failed during creation), even when it didn't have to. * Internal issue BTS-71: Added a precondition to prevent creating a collection with an invalid `distributeShardsLike` property. * In case a document got requested via the UI of a collection which does not exist, the UI now properly displays an error view instead of having a bad display state. * Returned `AQL_WARNING()` to emit warnings from UDFs. * Fixed internal issue #725: Added analyzers revision for _system database in queries. * The Web UI is not disabling the query import button after file upload takes place. * In case of a graph deletion failure, the Web UI displays now the correct error message. * Privatized load plan / current in cluster info and cleanup following agency cache implementation. * Allow removal of existing schemas by saving a schema of either `null` or `{}` (empty object). Using an empty string as schema will produce an error in the web interface and will not remove the schema. The change also adjusts the behavior for the SCHEMA_VALIDATE AQL function in case the first parameter was no document/object. In this case, the function will now return nulland register a warning in the query, so the user can handle it. * The Web UI is now reporting errors properly in case of editing ArangoSearch Views with invalid properties. * Fixed bug in IResearchViewExecutor that lead to only up to 1000 rows being produced. * Fixed an error scenario where a call could miss count skip. It was triggered in the case of Gather in Cluster, if we skipped over a full shard, and the shard did actually skip, but there are more documents to skip on another shard. * Fixed that the hotbackup agency lock is released under all circumstances using scope guards. This addresses a rare case in which the lock was left behind. * Allow restoring collections from v3.3.0 with their all-numeric collection GUID values, by creating a new, unambiguous collection GUID for them. v3.3.0 had a bug because it created all-numeric GUID values, which can be confused with numeric collection ids in lookups. v3.3.1 already changed the GUID routine to produce something non-numeric already, but collections created with v3.3.0 can still have an ambiguous GUID. This fix adjusts the restore routine to drop such GUID values, so it only changes something if v3.3.0 collections are dumped, dropped on the server and then restored with the flawed GUIDs. * Fixed hotbackup agency lock cleanup procedure. * The Web UI's replication view is now checking the replication state automatically without the need of a manual reload. * Changing the current users profile icon in the Web UI now renders the new icon and data directly without the need of a full UI browser reload. * Fixed cluster behavior with HotBackup and non-existing backups on DB-Servers. * Fixed that, when performing a graph AQL query while a (graceful) failover for the leader of the system collections is in progress, ArangoDB would report a "Graph not found error". The real error, namely that an agency transaction failed, was swallowed in the graph lookup code due to a wrong error code being used from Fuerte. We now generate a more appropriate 503 - Service Unavailable error. * Fixed bad behavior that led to unnecessary additional revision tree rebuilding on server restart. * Only advance shard version after follower is reported in sync in agency. * Disabled new, potentially unsafe revision-based storage format. * Allow for faster cluster shutdown. This should reduce the number of shutdown hangers in case the agents are stopped already and then coordinators or DB-Servers are shut down. * Fixed issue ES-598. Web UI now shows correct permissions in case wildcard database level and wildcard collection level permissions are both set. * Fixed non-deterministic test failure in Pregel WCC test. * Add a recovery test to check that there are no warnings at server start after a graceful shutdown. * Fixed unintentional connection re-use for cluster-internal communications. * Fixed problem with newer replication protocol and ArangoSearch which could lead to server crashes during normal operation. * Allow AQL queries on DB-Servers again. This is not an official supported feature, but is sometimes used for debugging. Previous changes made it impossible to run a query on a local shard. * Fix restoring old arangodumps from ArangoDB 3.3 and before, which had index information stored in slightly different places in the dump. * Fix internal test helper function `removeCost` to really remove costs. * Fix invalid calls to `AgencyCommResult::slice()` method, which must only be made in case of an agency result was retrieved successfully. In case the call to the agency was not successful, `slice()` must not be called on it. This change makes sure this invariant holds true. * Fix potential AQL query shutdown hanger. * Modified the exception handling in the RestHandler. Asynchronous communication could lead to less detailed failure information. * Use smarter default value preset in web UI for replication factor in case there are constraints established for the replication factor by the startup options `--cluster.min-replication-factor` and `--cluster.max-replication-factor`. * Added permissions check before trying to read data from `_analyzers` collection. If these permissions are not there, no load is performed (user can not use analyzers from database anyway). * Make the reboot tracker catch a failed server and permanently removed servers. This allows other servers in the cluster to move on more quickly, when a server fails and does not immediately come back. * Added WCC pregel algorithm for weakly connected components. * Display "geoJson" value for geo indexes in index overview in web UI. * Allow specifying arbitrary vertex attributes when creating a vertex via the graph viewer in the web UI. This is necessary for collections that have a schema and that require new vertices to have certain attributes/values. * Pass through AQL query errors to UI graph viewer without turning them into "Internal Server Error"s. * Augment nodes view in web UI with node id and full datetime of startup. * Fixed "Create Debug Package" in web UI in cluster mode. * Pick up selected "write concern" value entered in web UI when creating a collection in the cluster. Previously, the "write concern" value was not picked up correctly and collections were created with the default write concern. * Fixed the creation of edges in the web UI, which did not work due to a JavaScript error. * Fixed velocypack validator for proper check of keys. * Handle missing "shardingStrategy" in Plan agency entries for collections. "shardingStrategy" entries can be missing for collections created with ArangoDB clusters in version 3.3 or earlier. A missing "shardingStrategy" entry can cause issues for certain queries that need to check the shard distribution on the DB-Servers. This change transparently handles missing "shardingStrategy" entries. v3.7.1-beta.1 (2020-06-07) -------------------------- * Fixed issue #8941: `frontendConfig.basePath` is duplicated for users API call * Taken collection dropping from fast track in maintenance. This avoids blocking fast track maintenance threads when a shard cannot immediately be dropped because of some pending lock. * Updated ArangoDB Starter to 0.14.15. * Fixed a spuriously occurring shutdown deadlock between AgencyCallbackRegistry and AgencyCache. * Fixed `"Plan" is not an object in agency` error messages when restarting DB-Servers that contained ArangoSearch Views with links. * Fixed misordering of skipped number of rows report. Only triggered if you do a modification on a subquery nesting level 2 or more, e.g.: ``` LET s1 = ( LET s2 = ( INSERT {} INTO collection ) COLLECT WITH COUNT INTO c RETURN c ) ``` Here the c would be off, or a count on main query would be off. * Fixed crash in execution of non-spliced subqueries if remainder of subquery is skipped. * Added missing mutex to ConnectionPool::cancelConnections(). * Foxx API now respects "idiomatic" flag being explicitly set to false. * Made modification subqueries non-passthrough. The passthrough logic only works if exactly as many output rows are produced as input rows are injected. If the subquery with modification is skipped however this API is violated, we perform the subquery with the modification, but then discard the input if skipped). * Optimized Web UI styling for ArangoSearch Views when in read-only mode. * Fixed a bug in the internal `recalculateCount()` method for adjusting collection counts. The bug could have led to count adjustments being applied with the wrong sign. * Removed external file loading in a used 3rd party library (Web UI). * More substantial metrics in agency. * arangodump and arangorestore will now fail when using the `--collection` option and none of the specified collections actually exist in the database (on dump) or in the dump to restore (on restore). In case some of the specified collections exist, arangodump/restore will issue warnings about the invalid collections, but will continue to work for the valid collections. * Improved network send request for more robustness. * Added multiple RocksDB configuration options to arangod: * `--rocksdb.cache-index-and-filter-blocks` to make the RocksDB block cache quota also include RocksDB memtable sizes. * `--rocksdb.cache-index-and-filter-blocks-with-high-priority` to use cache index and filter blocks with high priority making index and filter blocks be less likely to be evicted than data blocks. * `--rocksdb.pin-l0-filter-and-index-blocks-in-cache` make filter and index blocks be pinned and only evicted from cache when the table reader is freed. * `--rocksdb.pin-top-level-index-and-filter` make the top-level index of partitioned filter and index blocks pinned and only be evicted from cache when the table reader is freed. * Don't move potentially expensive AQL function calls into loops in the `remove-unnecessary-calculations-rule`. For example, in the query LET x = NOOPT(1..100) LET values = SORTED(x) FOR j IN 1..100 FILTER j IN values RETURN j there is only one use of the `values` variable. So the optimizer can remove that variable and replace the filter condition with `FILTER j IN SORTED(x)`. However, that would move the potentially expensive function call into the inner loop, which could be a pessimization. Now the optimizer will not move the calculation of values into the loop when it merges calculations in the `remove-unnecessary-calculations` optimizer rule. * Fixed modification executors inside of a subquery, where the subquery decided to fetch all rows from upstream first and the amount of rows is higher then the batch size. * Fixed reporting of skipped number of documents if we have a LIMIT x, 0 right after the modification. * Added exceptions catching in agency callbacks. * Fixed issue #11104 case with complex condition on second branch of OR operator. * Fixed bad behavior when dropping followers. A follower should be dropped immediately when it is officially FAILED, not only after a longish timeout. * Fixed a bug in CollectionNameResolver which could lead to an extended busy spin on a core when a collection was dropped, but documents of it still remained in the WAL. * Fixed return value of fuerte::H1Connection in case of timeout. * Usability improvements for arangodump and arangorestore: - when a dump is restricted to one or multiple collections using the `--collection` option of arangodump, warnings are issued for all specified collections that are not present in the database. Previously there were no warnings, so missing a collection by misspelling its name could easily go unnoticed. - when a restore is restricted to one or multiple collections using the `--collection` option of arangorestore, warnings are issued for all specified collections that are not present in the dump. Previously there were no warnings, so missing a collection by misspelling its name could easily go unnoticed. - when a dump was taken using the `--overwrite` option, there was no check that validated whether the encryption mode used in the existing dump directory was the same as the requested encryption mode. This could have led to dump directories with both encrypted and unencrypted files. This was only the case when using `--overwrite true`, which is not the default. - when a restore was performed using the `--encryption.keyfile` option, there was no check whether the to-be-restored files were actually encrypted. Now this check is enforced and arangorestore will bail out with an error if the requested encryption mode for restore is not the same as for the stored dump files. * Fixed traversal issue: If you have a traversal with different minDepth and maxDepth values and filter on path elements that are larger then minDepth, in a way that a shorter path would match the condition, the shorter paths would in some cases not be returned, even if they are valid. e.g. FOR v, e, p IN 1..3 OUTBOUND @start Graph "myGraph" FILTER p.vertices[2].label != "foo" RETURN p In the above query, a path of length 1 would be valid. There p.vertices[2] does not exist => Evaluated to `null`. `null`.label is again evaluated to `null` => `null != "foo"` is true, so the path is valid. * Fixed traversal issue: If you have a filter on the path that is based on a variable value, which could not be deduced as constant during runtime, in a sharded GeneralGraph the filter was not applied correctly. SmartGraphs and SingleServer traversals are not effected by this issue. Also OneShard traversals are not effected. * Added `securityScheme`, `securityScope` and `security` methods to Foxx endpoints to allow defining Swagger security schemes. * Foxx routes now always have a Swagger `operationId`. If the route is unnamed, a distinct operationId will be generated based on the HTTP method and URL. * Fixed, if you have a collection access within a Subquery, where the main query is fully skipped and the "splice-subqueries" rule is active. The information of the skip was not transported correctly. This could cause incorrect counting reports. If splice-subqueries are disabled, or the main-query is only partly skipped, everything worked as expected. * Expanded -std=c++17 flag to all compilers. * Fixed issue ES-600: log out of LDAP automatically. Option `--ldap.allow-offline` was added with `false` value by default to let server use cached offline credentials in case of LDAP server communication was lost and ArangoDB can't prolong cache online. * Include all requests to the web interface at endpoint `/_admin/aardvark` in a node's requests statistics. Previously, requests going to `/_admin/aardvark` were excluded from the request statistics if and only if the requested database was the `_system` database. v3.7.1-alpha.2 (2020-05-27) --------------------------- * Fixed an issue with truncate of a collection after a dbserver was restarted very quickly. This could block arangosync from making progress because the _jobs collection could no longer be truncated. * Web UI is now using pre-compiled templates. * Fixed issue #11652: Error: AQL: missing variable #2 (1) for node #6 (FilterNode) while planning registers. * Fixed internal issue #650: Added analyzer cache update before ArangoSearch view creation/update. * Update OpenLDAP to 2.4.50. * Fixed issue ES-554: failed to build filter while querying ArangoSearch view. This change provides global fix for ArangoSearch analyzers distribution and management within a cluster. * Fixed issue ES-545: failing modification operations led to random crashes with "Index out of bound" error message. * Fix edge cases in RocksDB primary index range lookups for operators >= and <. * Fixed issue #11525: Address security vulnerability by updating Swagger-UI dependency (upgraded Swagger UI to 3.25.1). * Updated arangosync to 0.7.5. v3.7.1-alpha.1 (2020-05-15) --------------------------- * Fixed ability to edit graph edge in Graph Viewer of web UI. * Fixed issue #10371: For k-shortest-paths queries on certain graphs a condition to abort search was too eager and hence missed the shortest path between two vertices and returned a longer one. * Added feature: Disjoint SmartGraphs SmartGraphs have been extended to a new subtype, called `Disjoint SmartGraphs`. A Disjoint SmartGraph prohibits edges between different SmartGraph components. In case the graph schema can be represented without the need of connected SmartGraph components, a Disjoint SmartGraph should be used as this knowledge can be used by the internal optimizer to gain even better query traversal execution times. * Fixed issue ES-605: arangodump broken when using encryption. * Fixed a lockup in dropCollection due to a mutex being held for too long. * Add an optimizer rule that enables execution of certain subqueries on a DB Server. For this optimization to work, the subquery must contain exactly one DISTRIBUTE/GATHER pair and only access at most one collection. This proves particularly useful for traversals, shortest path, and k-shortest paths queries on disjoint smart graphs where the entire traversal is executed on the DB Server without involvement of a coordinator. * ClusterInfo does its own updating of plan / current caches. * Properly forward requests based on the transaction header. * Fix a bug in Agency poll, which would sometimes return a wrong firstIndex. * Made several improvements to AQL's internal register planning. This will improve query performance in several cases; mainly of large queries with many variables, and of spliced subqueries. This also fixes the github issues #10853 and #11358 which reported performance degradation caused by subquery splicing. * Fixed issue #11590: Querying for document by _key returning only a single seemingly random property on entity ("old", in this case). This fixes single-key document lookups in the cluster for simple by-key AQL queries, such as `FOR doc IN collection FILTER doc._key == @key RETURN doc` in case the document has either an "old" or a "new" attribute. * Restored behavior of Foxx API documentation being expanded to show all routes rather than collapsing all sections by default. * Add optimization for subqueries for which only the number of results matters. The optimization will be triggered for read-only subqueries that use a full collection scan or an index scan, without any additional filtering (early pruning or document post-filtering) and without LIMIT. It will help in the following situation: FOR doc IN ... LET count = COUNT( FOR sub IN subCollection FILTER sub._from == doc._id RETURN sub ) ... The restrictions are that the subquery result must only be used with the COUNT/LENGTH function and not for anything else. The subquery itself must be read-only (no data-modification subquery), not use nested FOR loops nor LIMIT, nor a FILTER condition or calculation that requires accessing the document data. Accessing index data is supported for filtering, but not for further calculations. If the optimization is triggered, it will show up in the query execution plan under the name `optimize-count`. * Integrated libiresearch log topic properly into ArangoDB logging system. * Fix a bad deadlock because transaction cleanup was pushed with too low priority. * Allow specifying graph names as unquoted string in an AQL graph traversal query, e.g. `FOR ... IN ... GRAPH abc`. Previously, the graph name had to be a bind parameter or a string enclosed in quotes. * loadPlan and loadCurrent have been fixed to not miss out on increments. * ClusterInfo and all data definition changes use the local agency cache. * Coordinators and DB servers cache the agency's read key value store. * Agency offers the new poll API to subscribe to the Raft log stream. * Added option `--rocksdb.edge-cache` to toggle in-memory caching for edges. The option is turned on by default. This normally helps with performance in read-only and read-mostly workloads. * Fixed a bug in Maintenance which could prevent collection creation from working (made CreateCollection action idempotent). * Fix potential undefined behavior in some operations issued to the REST handler at `/_api/collection` in cluster mode. * `--cluster.agency-prefix` marked as obsolete. Did never work and is not supported. * Removed old AgencyComm. * Improve continuation behavior of AQL queries. We post the continuation handler on lane CLUSTER_AQL instead of CLIENT_AQL. This leads to MEDIUM prio instead of LOW. * Fixed a sleeping barber in fuerte. Added TLA+ model to prove that there is not another one hiding somewhere. * Fix spurious bugs in `resilience_move` tests due to replication context of to-be-dropped collections lingering around until timeout. * Fixed issue 11243: `db._isSystem()` returns true for a non-system DB. * Fix REST API `/_admin/cluster/removeServer`. This also fixes the removal of servers via the web interface. * Updated OpenSSL to 1.1.1g. * Lower the default values for `--rocksdb.block-cache-size` and `--rocksdb.total-write-buffer-size` for systems with less than 1 GB of RAM. For these systems the default value were presumably too high and needed manual adjustment. * Allow to override the detected number of available CPU cores via an environment variable ARANGODB_OVERRIDE_DETECTED_NUMBER_OF_CORES. * Fix a bug in the optimizer which handled combinations of SmartJoins with query components that had to run on a coordinator wrong. * Remove the scatter-arangosearch-view-in-cluster optimizer rule; it was integrated into scatter-in-cluster. * Issue #11447: expose extra RocksDB options `--rocksdb.target-file-size-base` and `--rocksdb.target-file-size-multiplier` to make them configurable. * Allow STARTS_WITH function to process several prefixes with min match count parameter. * Added collection property changes into auditing. * Fix creation of example graphs when `--cluster.min-replication-factor` is set and no replication factor was specified for a graph. In this case, the new graph will use the configured minimum replication factor. * Show some index details in index overview of the web interface, e.g. the `minLength` attribute for fulltext indexes or the `expireAfter` attribute for TTL indexes. * Added startup option `--server.validate-utf8-strings`. This option controls whether strings in incoming JSON or VPack requests are validated for UTF-8 conformity. If set to `true` (which is the default value), all strings are validated, and requests containing invalid UTF-8 data will be rejected. Setting the option to `false` will turn off the UTF-8 string validation for incoming requests. This mode should only be used for deployments that are known to already contain invalid UTF-8 data and to keep them operational until the wrong string encoding is fixed in the data. * Improve continuation behavior of AQL queries. We post the continuation handler on lane CLUSTER_AQL instead of CLIENT_AQL. This leads to MEDIUM prio instead of LOW. * When relaying requests to other coordinators in a load-balanced setup, don't forward the "http/1.1" HTTP header from the remote response. Forwarding that header in the same way as any other header is not required, as each response will store and set its status separately anyway. Adding it to the headers again will only cause confusion and some client applications to choke. * Fixed issue #11104: `ANALYZER` function doesn't correctly process `!= OR` queries. * Obsoleted startup option `--database.maximal-journal-size`. This option was useful for the MMFiles storage engine only, but did not have an effect with the RocksDB engine. Using this startup option is not an error, but has no effect anymore. * Added `JACCARD` AQL function. * storedValues property is removed from ArangoSearch link properties output. * Added primarySortCompression property to ArangoSearch views. * Added compression property to ArangoSearch view storedValues. * Removed deprecated MMFiles storage engine and also the `arango-dfdb`(datafile debugger) executable that could be used to validate MMFiles datafiles. This change also obsoletes all MMFiles-specific startup options in the `--wal.*` section. Using these startup options is not an error, but has no effect anymore. * Fixed a bug in the agency supervision, which ignored the `failoverCandidates` field. * Added `INTERLEAVE` AQL function. * Upgraded bundled RocksDB library to version 6.8.0. v3.7.0 (2020-04-11) ------------------- * Updated OpenSSL to 1.1.1f. * Fixed a bug which occurred if a DB-Server was shut down exactly when it was supposed to resign from its leadership for a shard. * Fix a bug in the agency supervision, which could declare an already FAILED dbserver temporarily as GOOD again after an agency leader change. * Added overwrite mode "ignore" for document inserts. This mode allows ignoring primary key conflicts on insert when the target document already exists. The following overwrite modes now exist: - "ignore": if a document with the specified `_key` value already exists, nothing will be done, and no write operation will be carried out. The insert operation will return success in this case. This mode does not support returning the old or new document versions using the `returnOld` and `returnNew` attributes. - "replace": if a document with the specified `_key` value already exists, it will be overwritten with the specified document value. This mode will also be used when no overwrite mode is specified but the `overwrite` flag is set to `true`. - "update": if a document with the specified `_key` value already exists, it will be patched (partially updated) with the specified document value. The overwrite mode "ignore" can also be used from AQL INSERT operations by specifying it in the INSERT's `OPTIONS`, e.g. INSERT { _key: ..., .... } INTO collection OPTIONS { overwriteMode: "ignore" } Again, when the overwrite mode "ignore" is used from AQL, it does not support returning the old or new document versions. Using "RETURN OLD" in an INSERT operation that uses the "ignore" overwrite mode will trigger a parse error, as there will be no old version returned, and "RETURN NEW" will only return the document in case it was inserted. In case the document already existed, "RETURN NEW" will return "null". The main use case of inserting documents with overwrite mode "ignore" is to make sure that certain documents exist in the cheapest possible way. In case the target document already exists, the "ignore" mode is most efficient, as it will not retrieve the existing document from storage and not write any updates to it. v3.7.0-preview.1 (2020-03-27) ----------------------------- * Added AQL function `IN_RANGE`. * Added startup option `--ssl.prefer-http1-in-alpn` to optionally let the server prefer HTTP/1.1 over HTTP/2 in ALPN protocol negotiations. * Compilation issues with wrong cv-qualifiers and unnecessary temporary copying. v3.7.0-alpha.2 (2020-03-20) --------------------------- * Add DTRACE points to track a request through the infrastructure. * Fixed issue #11275: indexes backward compatibility broken in 3.5+. * Updated snowball dependency to the latest version. More stemmers are available. Built-in analyzer list is unchanged. * Reactive REST API endpoint at `/_admin/auth/reload`, as it is called by DC2DC. * Fix an endless loop in FollowerInfo::persistInAgency which could trigger a hanger if a collection was dropped at the wrong time. * Updated LZ4 dependency to version 1.9.2. * Fix cluster representation of the collection figures for RocksDB. * Changed behavior for creating new collections in OneShard databases (i.e. databases with "sharding" attribute set to "single"): Previously it was allowed to override "distributeShardsLike" and "numberOfShards" for each new collection. The default values were "_graphs" and 1 and were not modified if the user did not alter them, but it was still possible to alter them. This is now (silenly) ignored. Any attempt to set any value for "distributeShardsLike" or "numberOfShards" for new collections in a OneShard database will silently be ignored. The collection will automatically be sharded like the sharding prototype and will have a single shard. The behavior has been adjusted to match the behavior when creating collections in a cluster started with `--cluster.force-one-shard true` option. Here any user-supplied values for "distributeShardsLike" or "numberOfShards" were always ignored. Now the behavior is identical for OneShard databases and databases in cluster started with `--cluster.force-one-shard true`. The web interface now also hides the "Distribute shards like" settings in this case, and makes the "Number of shards" input box read-only. * Fix premature access to temporary path before a user-specified path was read from the config options. * Rebuild UI and update swagger. * Added ability to store values in ArangoSearch views. * Added LIKE operator/function support to SEARCH clause. * Added NGRAM_SIMILARITY and NGRAM_POSITIONAL_SIMILARITY functions for calculating Ngram similarity. * Added ngram fuzzy search. Supported by NGRAM_MATCH filter function for SEARCH clause. * Allow to override the detected total amount of memory via an environment variable ARANGODB_OVERRIDE_DETECTED_TOTAL_MEMORY. * `splice-subqueries` optimization is not limited by any type of operation within the subquery any more. It can now be applied on every subquery and will be by default. However they may be a performance impact on some queries where splice-subqueries are not as performant as non-spliced subqueries. This is due to internal memory management right now and will be addressed in future versions. Spliced subqueries can be less performant if the query around the subquery is complex and requires lots of variables, or variables with large content, but the subquery itself does not require a lot of variables and produces many intermediate results s.t. good batching within the query does not pay off against memory overhead. * Supervision to clean up zombie servers after 24h, if no responsibility for shards. * Fix SORT RAND() LIMIT 1 optimization for RocksDB when only a projection of the attributes was used. When a projection was used and that projection was covered by an index (e.g. `_key` via the primary index), then the access pattern was transformed from random order collection seek to an index access, which always resulted in the same index entry to be returned and not a random one. * Mark server startup options `--foxx.*`, `--frontend.*` and `--javascript.*` as single server and Coordinator only for documentation (`--dump-options`). * Supervision hot backup and supervision maintenance modes ttl fix. * Fix a bug that leads to graph traversals yielding empty output when none of the output variables (vertex, edge, path) are used. This is relevant when a query is only interested in a COUNT of the outputs, for example. * MoveShard to check, if target is in sync follower before promotion to leader. * Agency ttl bug fix. * Added the SNI feature for TLS. This means that one can configure multiple server keys and certificate chains and the system dynamically uses the right one depending on the value of the TLS servername extension. This allows to use different TLS setups for the same server which is reachable behind different DNS names, for example (Enterprise Edition only). * Do not create a reboot tracker for empty serverId ubin sync repl. * Fix the usage of the AQL functions `CALL` and `APPLY` for calling user-defined AQL functions when invoking an AQL query from the arangosh or a client application. Previously, invoking an AQL query and using the `CALL` or `APPLY` AQL functions to call user-defined AQL function caused undefined behavior. * Improved graph traversal performance via some internal code refactoring: - Traversal cursors are reused instead of recreated from scratch, if possible. This can save lots of calls to the memory management subsystem. - Unnecessary checks have been removed from the cursors, by ensuring some invariants. - Each vertex lookup needs to perform slightly less work. The traversal speedups observed by these changes alone were around 8 to 10% for single-server traversals and traversals in OneShard setups. Cluster traversals will also benefit from these changes, but to a lesser extent. This is because the network roundtrips have a higher share of the total query execution times there. * Traversal performance can also be improved by not fetching the visited vertices from the storage engine in case the traversal query does not refer to them. For example, in the query FOR v, e, p IN 1..3 OUTBOUND 'collection/startVertex' edges RETURN e the vertex variable (`v`) is never accessed, making it unnecessary to fetch the vertices from storage. If this optimization is applied, the traversal node will be marked with `/* vertex optimized away */` in the query's execution plan output. * The existing optimizer rule "move-calculations-down" is now able to also move unrelated subqueries beyond SORT and LIMIT instructions, which can help avoid the execution of subqueries for which the results are later discarded. For example, in the query FOR doc IN collection1 LET sub1 = FIRST(FOR sub IN coll2 FILTER sub.ref == doc._key RETURN sub) LET sub2 = FIRST(FOR sub IN coll3 FILTER sub.ref == doc._key RETURN sub) SORT sub1 LIMIT 10 RETURN { doc, sub1, sub2 } the execution of the `sub2` subquery can be delayed to after the SORT and LIMIT, turning it into FOR doc IN collection1 LET sub1 = FIRST(FOR sub IN coll2 FILTER sub.ref == doc._key RETURN sub) SORT sub1 LIMIT 10 LET sub2 = FIRST(FOR sub IN coll3 FILTER sub.ref == doc._key RETURN sub) RETURN { doc, sub1, sub2 } * Added JSON-Schema (draft-4) document validation. The validation can be specified by providing the new `schema` collection property when creating a new collection or when updating the properties of an existing collection. db.mycollection.properties({ schema: { rule : { a : { type : "array", items : { type : "number", maximum : 6 }}}, message : "Json-Schema validation failed" } }); * Fix supervision mode detection when unlocking agency in hot backup. * Foxx Response#throw will no longer generate incomplete error objects when passed invalid arguments. * Foxx will now always generate a numeric HTTP status code even if the response status code has been set to a non-numeric value. * Make arangobench return a proper error message when its initial attempt to create the test collection fails. * Added traversal options `vertexCollections` and `edgeCollections` to restrict traversal to certain vertex or edge collections. The use case for `vertexCollections` is to not follow any edges that will point to other than the specified vertex collections, e.g. FOR v, e, p IN 1..3 OUTBOUND 'products/123' components OPTIONS { vertexCollections: [ "bolts", "screws" ] } The traversal's start vertex is always considered valid, regardless of whether it is present in the `vertexCollections` option. The use case for `edgeCollections` is to not take into consideration any edges from edge collections other than the specified ones, e.g. FOR v, e, p IN 1..3 OUTBOUND 'products/123' GRAPH 'components' OPTIONS { edgeCollections: [ "productsToBolts", "productsToScrews" ] } This is mostly useful in the context of named graphs, when the named graph contains many edge collections. Not restricting the edge collections for the traversal will make the traversal search for edges in all edge collections of the graph, which can be expensive. In case it is known that only certain edges from the named graph are needed, the `edgeCollections` option can be a handy performance optimization. * Make arangobench return a proper error message when its initial attempt to create the test collection fails. * In some cases with a COLLECT LIMIT situation on a small limit the collect does more calls to upstream than without a limit to provide the same result. We improved this situation and made sure that LIMIT does not cause the COLLECT to fetch too few input rows. There is a chance that queries with a very small amount of data might suffer from this modification, most queries will benefit however. * Use OpenSSL's EVP interface for SHA256 instead of the deprecated low-level message digest APIs. * Prevent startup error "specify either '--server.jwt-secret-keyfile' or '--server.jwt-secret-folder' but not both." from occurring when specifying the JWT secret keyfile option. * Honor `cacheEnabled` attribute when updating the properties of a collection in case of the cluster. Previously, this attribute was ignored in the cluster, but it shouldn't have been. * Fixed issue #11137: http-proxy can't pass delete method req.body to foxx. * Disable "collect-in-cluster" AQL optimizer rule in case a LIMIT node is between the COLLECT and the source data. In this case it is not safe to apply the distributed collect, as it may alter the results. v3.7.0-alpha.1 (2020-02-19) --------------------------- * Rebuild UI. * Fix arangorestore: If a smartGraphAttribute value was changed after document creation, the _key of the document becomes inconsistent with the smartGraphAttribute entry. However, the sharding of documents stays intact. When restoring data from a dump that contains a document with inconsistent _key and smartGraphAttribute this would cause an error and the document would not be inserted. This is fixed by ignoring this inconsistency in the case of restore. * Updated rclone to 1.51.0. * Fixed a memory leak in ModificationExecutors. * Updated arangosync to 0.7.3. * Updated ArangoDB Starter to 0.14.14. * Fixed handling of `geometry` attributes in query editor of the web UI. Previously, document attributes named `geometry` were treated in a special way, assuming they were JSON objects. In case a document contained a `geometry` attribute that had a non-object value (e.g. `null`) the web UI threw a JavaScript exception and would not display AQL query results properly. * Disable "collect-in-cluster" AQL optimizer rule in case a LIMIT node is between the COLLECT and the source data. In this case it is not safe to apply the distributed collect, as it may alter the results. * Fixed internal issue #4932: COLLECT WITH COUNT together with FILTER yields zero. This bugfix fixes an issue when skipping over documents in an index scan using a covering index and also at the same time using an early-pruning filter. In this case wrong document data may have been injected into the filter condition for filtering, with the filter wrongfully deciding which documents to filter out. * Added crash handler for Linux builds that taps into the following signals: - SIGSEGV (segmentation violation) - SIGBUS (bus error) - SIGILL (illegal instruction) - SIGFPE (floating point exception) In case the arangod process catches one these signals, the crash handler tries to log a message and a backtrace into the installation's logfile before terminating the process. * Removed dependency on libcurl library. * Remove storage engine selection from package installers. The default storage engine for new installations is now RocksDB. The MMFiles storage engine cannot be selected for new installations. * Remove misleading error messages about upgrade commands to carry out in case of a database upgrade. These commands are highly platform-dependent and also depend on whether ArangoDB is started manually, via the ArangoDB starter or as a service. In order to not confuse end users, remove the potentially misleading instructions. * Clear backups from DB servers and agency, when plan unchanged not met and not allowing for inconsistency. * V8 version upgrade to 7.9.317; ICU version upgrade to 64.2. - JSON parsing is roughly 60% faster than in V8 7.1; you should prefer JSON.parse(string) over deeply nested javascript variable declarations. - several new javascript language features * Clean out server job checks preconditions plan version unchanged for start. * Cluster collection creation preconditioned on involved db servers not in process of being cleaned and Fixed. * Fixed a bug, where a single host agency logged too early. This issue caused spurious "Could not integrate into cluster" error messages when starting coordinators. * Fixed issue #11051: AQL query wrongfully returning no results on 3.6.1. This change prevents the "late-document-materialization" optimizer rule from kicking in in cases when an index scan also uses early pruning of index entries. In this case, the late document materialization will lead to the filter condition used in early pruning not being applied correctly, potentially producing wrong results. * Fixed internal issue #596: Added ability to disable DNF conversion in SEARCH condition for ArangoSearch views. As this conversion might result in overcomplicated disjunction and heavily slowdown execution. * Implement a new API to reload TLS server key and certificates as well as client certificate CA. This allows to rotate TLS certificates without a restart. One can also query the currently loaded state of the TLS data to decide if a reload has actually happened. The new certificates will only be used for new TLS-based connections. * Add acquisition of system report to arangod instances. * JS `query` helper function can now be used with options. * Fix a bug in collection creation with `distributeShardsLike`. * Added load-balancing support for listing currently running and slow AQL queries, killing running AQL queries and clearing the list of slow AQL queries. This change also will also modify the values returned in the "id" attribute of AQL query objects. While the values of the "id" attribute remain strings with numeric content, the ids will now start at arbitrary offsets after server start and are supposed to have much higher numeric values than in previous ArangoDB versions. In previous ArangoDB versions, query ids always started at value 1 after a server start/restart and were increased in increments of 1. This change may lead to query ids being greater than what a 4 byte integer can hold, which may affect client applications that treat the ids as numeric values and do not have proper support for integer numbers requiring more than 4 byte storage. * remove unused REST API /_admin/aql/reload, and make the JavaScript bindings for `internal.reloadAqlFunctions()` do nothing. The reason for this is that AQL functions are automatically reloaded when functions are changed, and there is no need to call these APIs manually. * Fixed issue #10949: k shortest paths behavior wrong with zero weights. * Added `REPLACE_NTH` AQL function to replace a member inside an array. * Added JWT secret rotation (Enterprise Edition only). * The `--dump-options` command for arangod now also provides the "components" attribute that shows for which components (agent, coordinator, db server, single server) an option is relevant. Additionally, each option provides an "os" attribute that indicates on which operating systems the option is supported. This can be used when reading the options descriptions programmatically, e.g. for auto-generating the documentation. * Add update-insert operation similar to existing replace-insert functionality of insert. Like for the existing variant the `overwrite` flag has to be set to true. Then the update version can be selected by setting the `overwriteMode` to `"update"`. v3.6.1 (2020-01-29) ------------------- * Updated ArangoDB Starter to 0.14.13. * Added HotBackup events into auditing. * Internal statistics API now uses integer timestamps instead of doubles. The old behavior sometimes leads to failed requests because of parse errors which occurred in the internally used JavaScript joi library. * Fixed issue #10896: Variables defined inside spliced subqueries would leak into following COLLECT ... INTO var operations. * Fixed COLLECT not invalidating variables for following COLLECT ... INTO var operations. * Removed an unnecessary and wrong request from within the Web UI to the `/_admin/cluster/health` API. This lead to unauthorized network calls. * Allowed PHRASE function to process empty arrays without generating error. * Add acquisition of system report to arangod instances. * React dev mode now supports hot reload combined with proxy for development. * Fix issue #10897, using COLLECT in a subquery could lead to unexpected and confusing error messages * Fix potential nullptr dereference in view optimizer rule, when there were was a LIMIT outside of a FOR loop. * Added missing "global" parameter in Swagger REST API documentation for some replication endpoints. * Fixed an edge case in VelocyPack when the padding of a 1-byte offsetSize array is removed but the first few entries of the array contain a Slice of type None. * Fix Foxxmaster failover retry loop to not spin forever and give up after a while. * Fix "random" Valgrind "invalid free/delete" errors caused by usage of `alignas(64)` for shared_ptrs in the SupervisedScheduler. * MoveShard to check, if target is in sync follower before promotion to leader. * Fixed issue #10867: arangod based binaries lack product version info and icon on Windows * Fixed internal traversal edge collection cache being filled up correctly. Edges are able to point to other edges, but those we're not applied to the cache. * Fixed issue #10852: Nested spliced subqueries could return wrong results in some cases when some of the concerned subqueries did not return any values. * Fix string comparison bug that lead to traversal queries accepting prefixes of "edges" and "vertices" to be used as object accessors for the path object. * Fix bug affecting spliced subqueries when memory blocks are reused. * Now clearing an internal map inside the traverser engine correctly. * Add ability to choose logging to file in Windows installer of ArangoDB server (enabled by default). ArangoDB-logs folder with arangod.log should be stored in %PROGRAMDATA% and %LOCALAPPDATA% for all users and single user installation respectively. * Fix installation of arangoinspect and libraries in Windows client installer. * Disable cluster AQL parallelization for queries that contain traversal, shortest path or k_shortest_path nodes. This avoids potential undefined behavior in case a parallel GatherNode is used in graph queries. * Fixed internal issue #656: While executing large amount of concurrent insert/removes iresearch seems to leak open file handles. This results in error 32 in cleanup (observed only on Windows as Linux doesn`t have file sharing locks). * Updated arangosync to 0.7.2. * Fix Windows client package JS installation paths. * Added option that makes ArangoDB write logfiles in the Windows installer. v3.6.0 (2020-01-08) ------------------- * Do not create a reboot tracker for empty serverId ubin sync repl. * Update swagger. v3.6.0-rc.2 (2019-12-23) ------------------------ * Fixed issue #10725: Wrong document count shown inside an empty collection in the web UI. * Fixed bug that made AQL queries eligible for parallelization even in case they couldn't be parallelized, leading to undefined behavior due to the thread races. * Fixed bug in `--query.parallelize-gather-writes-behavior` which led to the option not working correctly. * Agency relational operators TTL fix. * Improve latency in scheduler. * Fixed internal issue #4748: Editing a single edgeDefinition using the graph API failed if it was not shared between all available graphs. * Fixed undefined behavior on node delete. * Fixed agency invalid operation. * Added google tests for Node. * Bugfix: An AQL ternary expression with the condition being true at query compile time would not execute its false branch. * When starting the RocksDB engine, first create small-sized WAL files. This is a precaution so that when repeatedly trying to start an arangod instance, an instant instance startup failure will not lead to the disk filling up so quickly with WAL file data. The WAL file size is increased later on in the startup sequence, so everything should be fine if the startup works. This fixes a problem with the disk filling up quickly when the arangodb starter tries to start an instance 100 times in a row but instantaneously gets a failure back from it. * Fixed a permissions bug for /_admin/cluster/rebalanceShards. v3.6.0-rc.1 (2019-12-10) ------------------------ * Renamed document / index / vertex "handle" to "identifier" / "id" for consistency in documentation and error messages. * Fixed a bug in smart graph bfs traversals that might violate path uniqueness requirements in rare cases. * Fixed permissions for dump/restore. * Rename `minReplicationFactor` into `writeConcern` to make it consistent with `--cluster.write-concern` and avoid confusion with `--cluster.min-replication-factor` * ArangoDB web UI switched build system from grunt to webpack. It supports React now, but the previous framework is still in use. * Enable the `parallelize-gather` AQL optimizer rule for certain write queries. The optimization is turned on by default and can be disabled by setting the startup option `--query.parallelize-gather-writes` to `false`. * Shard synchronization readlock aware of rebootId. * Bugfix: In an AQL cluster query, when gathering unsorted data in combination with a LIMIT with non-zero offset, if this offset exactly matches the number of documents in the first shards consumed, the rest of the documents was not returned. * REMOTE and GATHER no longer make subqueries unsuitable for the `splice-subqueries` optimization. * New internal counter and histogram support. * Add a Prometheus endpoint for metrics, expose new metrics, old statistics and RocksDB metrics. * Fixed known issue #509: ArangoSearch index consolidation does not work during creation of a link on existing collection which may lead to massive file descriptors consumption. * Added support of array comparison operators to ArangoSearch. * Added support of arrays to TOKENS function. * Added support of arrays to PHRASE function. * Added a new optimization called `late-document-materialization`, `late-document-materialization` for indexes and views correspondingly. This optimization reduces amount of documents to read from storage engine to the limit explicitly stated in LIMIT node. * Added support of "Edge N-grams" to `text` analyzer. * Added support of UTF8 to `ngram` analyzer. * Added ability to mark beginning/end of the sequence to `ngram` analyzer. * Added a new subquery optimization called `splice-subqueries`. This optimization splices the execution of subqueries inside the execution of their surrounding query. For subqueries with a fairly small number of returned documents this saves a significant amount of overhead initializing cursors. This optimization is performed by the optimizer rule `splice-subqueries`, and currently only works on suitable subqueries; A subquery becomes unsuitable if it contains a LIMIT, REMOTE, GATHER or a COLLECT node where the operation is WITH COUNT INTO. A subquery also becomes unsuitable if it is contained in an (sub)query which contains unsuitable operations after the subquery. The implementation of this feature required reworking the dataflow query execution. * Added startup option `--query.optimizer-rules` to selectively enable or disable optimizer rules by default. The option can be specified multiple times, and takes the same input as the query option of the same name. For example, to turn off the rule _use-indexes-for-sort_, use --query.optimizer-rules "-use-indexes-for-sort" The purpose of this option is to be able to enable potential future experimental optimizer rules, which may be shipped in a disabled-by-default state. * Allow the AQL query optimizer to remove the DistributeNodes for several data-modification queries. So far, only REMOVE queries benefitted. Now the optimization can also be applied for REPLACE and UPDATE queries in case the query does not use LIMIT and there is no further cluster-internal communication after the REMOVE, REPLACE or UPDATE node. * Include ArangoSearch data in HotBackups. * Allow to restore 3.5 HotBackups in 3.6. * Fixed an issue where removeServer left behind current coordinators * Allow usage of AQL function `RANDOM_TOKEN` with an argument value of `0`. This now produces an empty string, whereas in older versions this threw an invalid value exception. * Add startup option `--rocksdb.exclusive-writes` to avoid write-write conflicts. This options allows for an easier transition from MMFiles to the RocksDB storage engine, but comes with a big performance penalty as all collections will be locked exclusively for writes. * Added new maxRuntime option for queries. If a query does not finish execution within the given time (in seconds) it will be killed. * Added limit for AQL range materialization to prevent out-of-memory errors. When materializing ranges created by either the AQL `RANGE` function or by using the built-in `..` operator (e.g. `1 .. 1000000`), a check is now performed if the range is too big to be materialized. The threshold value is set to 10 million members. Ranges with at most that many members can be materialized, ranges with more members will fail to materialize and abort the query with the exception `number out of range` (error code 1504). It is still possible to create ranges with more than 10 million members as long as they are not materialized. For example, the following is still valid: FOR i IN 1 .. 1000000000 INSERT {_key: CONCAT('test', i)} INTO collection * No longer put system services into `_apps` on single server. On cluster, this has never worked. This was unnecessary. * Added AQL optimizer rule "move-filters-into-enumerate", to allow for early pruning of non-matching documents while full-scanning or index-scanning documents. This optimization can help to avoid a lot of temporary document copies. * Added "SmartJoins for Views" to the ArangoDB Enterprise Edition that allows running cluster joins between two certain sharded collections or views with performance close to that of a local join operation. * Allow collection names to be at most 256 characters long, instead of 64 characters in previous versions. * Upgraded bundled Boost library version to 1.71. * Use `-std=c++17` for ArangoDB compilation. * Made the mechanism in the Web UI of replacing and upgrading a foxx app more clear. * Show shards of all collections (including system collections) in the web UI's shard distribution view. This is necessary to access the prototype collections of a collection sharded via `distributeShardsLike` in case the prototype is a system collection, and the prototype should be moved to another server. * Rclone URL normalization. * Disallow using `_id` or `_rev` as shard keys in clustered collections. Using these attributes for sharding was not supported before, but didn't trigger any errors. Instead, collections were created and silently using `_key` as the shard key, without making the caller aware of that an unsupported shard key was used. * Use execvp instead of execv in HotBackup restore. * Re-enabled the AQL sort-limit optimization rule in conjunction with fullCount in the cluster. It now also may speed up fullCount with sorted indexes and a limit. * Make the scheduler enforce the configured queue lengths. The values of the options `--server.scheduler-queue-size`, `--server.prio1-size` and `--server.maximal-queue-size` will now be honored and not exceeded. The default queue sizes in the scheduler for requests buffering have also been changed as follows: request type before now ----------------------------------- high priority 128 4096 medium priority 1048576 4096 low priority 4096 4096 The queue sizes can still be adjusted at server start using the above- mentioned startup options. * Add replicationFactor, minReplicationFactor and sharding strategy to database creation dialog in web UI. Preselect database default values for collection creation in web UI. * Add new JavaScript function `db._properties()` that provides information about the current database's properties. * Add new options `sharding` and `replicationFactor` for database creation methods. The specified values will provide the defaults for all collections created in a database. Valid values for `sharding` are `""`, "flexible", "single". The first 2 values are treated equally. Values for `replicationFactor` are natural numbers or the string `satellite`. * Add new server option `--cluster.default-replication-factor` that allows to set the default replication factor for non-system collections (default: 1). * Enabled IPO with cmake as an option, default is on for release builds without google tests. * Bugfix: The AQL sort-limit optimization was applied in some cases it shouldn't, resulting in undefined behavior. * Remove operations for documents in the cluster will now use an optimization, if all sharding keys are specified. Should the sharding keys not match the values in the actual document, a not found error will be returned. * Retry hot backup list in cluster for 2 minutes before reporting error. * Allowing inconsistent rather than forcing hot backups. * Geo functions will now have better error reporting on invalid input. * Upgraded bundled jemalloc library to version 5.2.1. * Added TransactionStatistics to ServerStatistics (transactions started / aborted / committed and number of intermediate commits). * Added AQL function DATE_ROUND to bin a date/time into a set of equal-distance buckets. * Enforced the valid date range for working with date/time in AQL. The valid date ranges for any AQL date/time function are: - for string date/time values: `"0000-01-01T00:00:00.000Z"` (including) up to `"9999-12-31T23:59:59.999Z"` (including) - for numeric date/time values: -62167219200000 (including) up to 253402300799999 (including). These values are the numeric equivalents of `"0000-01-01T00:00:00.000Z"` and `"9999-12-31T23:59:59.999Z"`. Any date/time values outside the given range that are passed into an AQL date function will make the function return `null` and trigger a warning in the query, which can optionally be escalated to an error and stop the query. Any date/time operations that produce date/time outside the valid ranges stated above will make the function return `null` and trigger a warning too. An example for this is. DATE_SUBTRACT("2018-08-22T10:49:00+02:00", 100000, "years") * Fixed bug in MoveShard::abort which causes a duplicate entry in the follower list. (Internal Bug #4378) * Updated TOKENS function to deal with primitive types and arrays. v3.5.3 (2019-11-28) ------------------- * Fixed GET _api/gharial to also include the name property in every returned graph. This is a consistency fix within the API as all other APIs include the name. As a work around the returned _key can be used, which is identical to the name. * The _users collection is now properly restored when using arangorestore. * Allow the optimizer to use indexes when a collection attribute is compared to an expansion followed by an attribute name, e.g. `doc.value IN something[*].name`. * Updated arangosync to 0.7.0. * Fixed issue #10470: The WebUI now shows potential errors and details which occured using _api/import (e.g. unique constraint violated). * Fixed issue #10440: Incorrect sorting with sort criteria partially covered by index. * Make the timeouts for replication requests (for active failover and master-slave replication configurable via startup options: --replication.connect-timeout --replication.request-timeout * Fixed internal issue #4647: dead Coordinators are not removed for agency. * Fixed UPSERT matching. Empty objects in the `UPSERT { ... }` expression will not be omitted anymore: db._collection("col").insert({ "find" : "me" }); db._query(` UPSERT { "find" : "me", "foo" : {} } UPDATE { "foo" : "not gonna happen" } INSERT { "find" : "me", "foo" : {} } INTO col `) This will now correctly insert a document instead of updating the existing, that only partially matches the upsert-expression. * Fixed undefined behaviour with creation of ArangoSearch links with custom analyzers in cluster environment. * Fixed internal issue #651: analyzer duplication in _analyzers collection. * Fixed internal issue #4597: rebalanceShards API cannot work on any database other than the _system database. * Stop putting system services in _apps on single server, this has never worked on cluster and was not needed. * Fixed issue #10371: K_SHORTEST_PATHS LIMIT 1 can not return the shortest path. Now the shortest path is returned as the first one in such queries. * Improve killability of some types of cluster AQL queries. Previously, several cluster queries, especially those containing a `DistributeNode` in their execution plans, did not respond to a kill instruction. This change also introduces a new query status "killed", which may now be returned by the REST APIs at `/_api/query/current` and `/_api/query/slow` in the `state` attribute of each query. * Improve shutdown of some cluster AQL queries on the coordinator in case the query has multiple coordinator snippets (true for queries involving more than one collection) and the database server(s) cannot be reached on query shutdown. In this case the proper shutdown of the coordinator parts of the query previously was deferred until the coordinator snippets were removed by the automatic garbage collection. Now, the cleanup of the coordinator snippets will happen much more quickly, which reduces the chances of the queries blocking resources. * Fixed ArangoSearch index removes being discarded on commiting consolidation results with pending removes after some segments under consolidation were already committed. * Assertion fail when no timestamp in agency's persistence. * Fixed internal issue #647: custom analyzer provokes errors on Active Failover deployment. * Upgraded bundled version of libcurl to 7.66.0. * When starting a coordinator, wait up to 15 seconds for it to appear in the agency under key `Supervision/Health` before reporting as "ready". This is necessary because if the coordinator reports ready beforehand and is used to create databases etc., the supervision may remove all of the jobs started by non-ready coordinators, considering them to be from a failed coordinator. To avoid huge startup delays, the startup will proceed after waiting futilely for 15 seconds and log a message. * Fixed issue #10270: Query: Expecting type Array or Object (while executing). * Fix a problem with AQL constrained sort in the cluster, which might abort queries. The AQL sort-limit optimization rule may now also speed up fullCount with sorted indexes and a limit in the cluster. * Prevent spurious log message "Scheduler queue is filled more than 50% in last x s" from occurring when this is not the case. Due to a data race, the message could previously also occur if the queue was empty. * The General Graph document API is now consistent with the document API in its error messages. When attempting to create / modify edges pointing to non existing vertex collections HTTP 400 is returned instead of 404. * Disallow the usage of subqueries inside AQL traversal PRUNE conditions. Using subqueries inside PRUNE conditions causes undefined behavior, so such queries will now be aborted early on with a parse error instead of running into undefined behavior. * Fixed available flag for hotbackup. * Fixed list with id for partially available hotbackups. * Fixed agency TTL bug happening under certain rare conditions. * Improved performance of some agency helper functions. * Fixed search not working in document view while in code mode. * Fixed issue #10090: fix repeatable seek to the same document in SEARCH operations for ArangoSearch views. * Fixed issue #10193: Arangoexport does not handle line feeds when exporting as csv. * Removed debug log messages "found comm task ..." that could be logged on server shutdown. * Fixed issue #10183: Arangoimport imports on _system when you try to create a new database. This bugfix fixes the output of arangoimport, which could display a wrong target database for the import if the option `--create-database` was used. * Fixed issue #10158: Invalid Query Crashes ArangoDB. This fixes traversal queries that are run on a static empty start vertex string. v.3.5.2 (2019-11-06) -------------------- * Fixed ArangoSearch upgrade procedure from previous minor version and patches. * Separately account for superuser and user request traffic. v3.5.1 (2019-10-07) ------------------- * Properly report parse errors for extraneous unterminated string literals at the end of AQL query strings. For example, in the query `RETURN 1 "abc`, the `RETURN 1` part was parsed fully, and the `"abc` part at the end was parsed until the EOF and then forgotten. But as the fully parsed tokens `RETURN 1` already form a proper query, the unterminated string literal at the end was not reported as a parse error. This is now fixed for unterminated string literals in double and single quotes as well as unterminated multi-line comments at the end of the query string. * Fix config directory handling, so we don't trap into UNC path lookups on Windows. * Ignore symlinks when copying JavaScript files at startup via the option `--javascript.copy-installation`. This potentially fixes the following error message at startup: Error copying JS installation files to '...': failed to open source file ...: No such file or directory * Added startup option `--cluster.max-number-of-shards` for restricting the maximum number of shards when creating new collections. The default value for this setting is `1000`, which is also the previously hard-coded built-in limit. A value of `0` for this option means "unrestricted". When the setting is adjusted, it will not affect already existing collections, but only collections that are created or restored afterwards. * Added startup options for managing the replication factor for newly created collections: - `--cluster.min-replication-factor`: this settings controls the minimum replication factor value that is permitted when creating new collections. No collections can be created which have a replication factor value below this setting's value. The default value is 1. - `--cluster.max-replication-factor`: this settings controls the maximum replication factor value that is permitted when creating new collections. No collections can be created which have a replication factor value above this setting's value. The default value is 10. - `--cluster.default-replication-factor`: this settings controls the default replication factor value that is used when creating new collections and no value of replication factor has been specified. If no value is set for this option, the value of the option `--cluster.min-replication-factor` will be used. * Fixed unintended multiple unlock commands from coordinator to transaction locked db servers. * DB server locking / unlocking for hot backup revisited and enhanced. * Rely on reboot ids for declaring end of cluster hot restore on coordinators. * Obtain new unique IDs via a background thread. * Fixed issue #10078: FULLTEXT with sort on same field not working. * Fixed issue #10062: AQL: could not extract custom attribute. * Fix compilation issue with clang 10. * Fixed error message for error 1928 ("not in orphan") to "collection is not in list of orphan collections". * Fix strange shutdown hanger which came from the fact that currently libgcc/libmusl wrongly detect multi-threadedness in statically linked executables. * Fixed a shutdown bug coming from a read/write lock race. * Fixed a bug in the edge cache's internal memory accounting, which led to the edge cache underreporting its current memory usage. * Fixed "ArangoDB is not running in cluster mode" errors in active failover setups. This affected at least /_admin/cluster/health. * Made the mechanism in the Web UI of replacing and upgrading a foxx app more clear. * Fixed the AQL sort-limit optimization which was applied in some cases it should not, resulting in undefined behaviour. * Add --server.statistics-history flag to allow disabling of only the historical statistics. Also added rocksdbengine.write.amplification.x100 statistics for measurement of compaction option impact. Enabled non-historical statistics for agents. * Fixed AQL constrained-heap sort in conjunction with fullCount. * Added support for AQL expressions such as `a NOT LIKE b`, `a NOT =~ b` and `a NOT !~ b`. Previous versions of ArangoDB did not support these expressions, and using them in an AQL query resulted in a parse error. * Disallow creation of TTL indexes on sub-attributes. Creation of such indexes was not caught before, but the resulting indexes were defunct. From now on the creation of TTL indexes on sub- attributes is disallowed. * Added HotBackup feature. * Improved database creation within the cluster. In the case of coordinator outages during the creation of the database there was a chance that not all relevant system collections had been created. Although this database was accessible now some features did not work as expected (e.g. creation of graphs). We modified creation of a new database as an all or nothing operation and only allow access to the database after all system collections are properly perpared to avoid the above inconsistencies. Also creation of databases are now secured against coordinator outages, they will either be fully created or not visible and eventually dropped. This does not require any change on the client code. * Added UI support to create documents in a collection using smartGraphAttribute and/or smartJoinAttribute. * Add count of objects to latency reporting in arangoimport. * Harden database creation against spurious "duplicate name" errors that were caused by other parallel operations lazily creating required system collections in the same database. * Fixed internal issue #633: made ArangoSearch functions BOOST, ANALYZER, MIN_MATCH callable with constant arguments. This will allow running queries where all arguments for these functions are deterministic and do not depend on loop variables. * Automatically turn values for deprecated startup option parameters `--log.use-microtime` and `--log.use-local-time` into equivalent option values of the new, preferred option `--log.time-format`. * Drop collection action to timeout more quickly to stay on fast lane. * Make arangorestore restore data into the `_users` collection last. This is to ensure that arangorestore does not overwrite the credentials of its invoker while the restore is running, but only at the very end of the process. This change also makes arangorestore restore the `_system` database last if it is started with the `--all-databases` option. * Fixed the removal (including a collection drop) of an orphanCollection from a graph definition when using the arango shell. The boolean flag whether to drop the collection or not was not transferred properly. * Check for duplicate server endpoints registered in the agency in sub-keys of `/Current/ServersRegistered`. Duplicate endpoints will be registered if more than one arangod instance is started with the same value for startup option `--cluster.my-address`. This can happen unintentionally due to typos in the configuration, copy&paste remainders etc. In case a duplicate endpoint is detected on startup, a warning will be written to the log, indicating which other server has already "acquired" the same endpoint. * Make graph operations in general-graph transaction-aware. * Fixed adding an orphan collections as the first collection in a SmartGraph. * Fixed non-deterministic occurrences of "document not found" errors in sharded collections with custom shard keys (i.e. non-`_key`) and multi-document lookups. * Fixed issue #9862: ServerException: RestHandler/RestCursorHandler.cpp:279 This fixes an issue with the RocksDB primary index IN iterator not resetting its internal iterator after being rearmed with new lookup values (which only happens if the IN iterator is called from an inner FOR loop). * Geo functions will now have better error reporting on invalid input. * The graph viewer of the web interface now tries to find a vertex document of all available vertex collections before it aborts. * Fixed issue #9795: fix AQL `NOT IN` clause in SEARCH operations for ArangoSearch views. * Make minimum timeout for synchronous replication configurable via parameter (--cluster.synchronous-replication-timeout-minimum) and increase default value to prevent dropping followers unnecessarily. * Added support for TLS 1.3 for the arangod server and the client tools. The default TLS protocol for the arangod server is still TLS 1.2 however, in order to keep compatibility with previous versions of ArangoDB. The arangod server and any of the client tools can be started with option `--ssl.protocol 6` to make use of TLS 1.3. To configure the TLS version for arangod instances started by the ArangoDB starter, one can use the `--all.ssl.protocol=VALUE` startup option for the ArangoDB starter, where VALUE is one of the following: - 4 = TLSv1 - 5 = TLSv1.2 - 6 = TLSv1.3 * Fixed parsing of "NOT IN" in AQL, which previously didn't correctly parse "NOT IN_RANGE(...)" because it checked if the "NOT" token was followed by whitespace and then the two letters "IN". * Changed log level for message "keep alive timeout - closing stream!" from INFO to DEBUG. * Don't create temporary directories named "arangosh_XXXXXX" anymore. * Add TransactionStatistics to ServerStatistics (transactions started / aborted / committed and number of intermediate commits). * Upgraded version of bundled curl library to 7.65.3. * Don't retry persisting follower information for collections/shards already dropped. The previous implementation retried (unsuccessfully in this case) for up to 2 hours, occupying one scheduler thread. * Fixed internal issue #4407: remove storage engine warning. * Agents to remove callback entries when responded to with code 404. * Fixed internal issue #622: Analyzer cache is now invalidated for dropped database. * Show query string length and cacheability information in query explain output. * The AQL functions `FULLTEXT`, `NEAR`, `WITHIN` and `WITHIN_RECTANGLE` are now marked as cacheable, so they can be used in conjunction with the AQL query results cache on a single server. * Fixed issue #9612: fix ArangoSearch views getting out of sync with collection. * Fix an issue with potential spurious wakeups in the internal scheduler code. * Changes the _idle_ timeout of stream transactions to 10 seconds and the total per DB server size of stream transaction data to 128 MB. The idle timer is restarted after every operation in a stream transaction, so it is not the total timeout for the transaction. These limits were documented in the manual for stream transactions since 3.5.0, but are enforced only as of 3.5.1. Enforcing the limits is useful to free up resources from abandoned transactions. * Consistently honor the return value of all attempts to queue tasks in the internal scheduler. Previously some call sites did not check the return value of internal queueing operations, and if the scheduler queue was full, operations that were thought to be requeued were silently dropped. Now, there will be reactions on such failures. Requeuing an important task with a time offset (Scheduler::queueDelay) is now also retried on failure (queue full) up to at most five minutes. If after five minutes such a task still cannot be queued, a fatal error will be logged and the server process will be aborted. * Made index selection much more deterministic in case there are multiple competing indexes. * Fixed issue #9654: honor value of `--rocksdb.max-write-buffer-number` if it is set to at least 9 (which is the recommended value). Ignore it if it is set to a lower value than 9, and warn the end user about it. Previous versions of ArangoDB always silently ignored the value of this setting and effectively hard-coded the value to 9. * Fixed internal issue #4378: fix bug in MoveShard::abort which causes a duplicate entry in the follower list. * Fixed cut'n'pasting code from the documentation into arangosh. * Fixed issue #9652: fix ArangoSearch wrong name collision and raising "Name collision detected" error during creation of a custom analyzer with stopwords. * Fixed an agency bug found in Windows tests. * Added initial support for wgs84 reference ellipsoid in GEO_DISTANCE through third optional parameter to AQL function. * Added support for area calculations with GEO_AREA AQL function. * Correct RocksDB statistics to report sums from column families instead of single value from default column family. * Fixed agency nodes to not create bogus keys on delete / observe / unobserve. * Fixed issue #9660: fix multiple plans processing during optimization of AQL query which uses ArangoSearch scorers. * Fixed issue #9679: improved error message for FULLTEXT function invocation failures. * Fixed error message "Invalid argument: assume_tracked is set but it is not tracked yet" when trying to write to the same keys in concurrent RocksDB transactions. This error will now be reported as a "Lock timeout" error, with error code 1200. * Added resign leadership job to supervision. * Hide MMFiles-specific information in web UI (in detail view of a collection) when the storage engine is not MMFiles or when the information is not available. * Keep followers in sync if the old leader resigned and stopped writes. * Abort a FailedLeader job when its _to_ server fails. * Decreased unnecessary wait times for agency callbacks in case they were called earlier than expected by main thread. * Make arangosh not close the connection after the user aborts an operation. This restores the same behavior as in previous versions of ArangoDB, which also left the connection open. * Refactor maintenance to use a TakeoverShardLeadership job. This fixes a bug that a shard follower could have set the wrong leader for the shard locally. v3.5.0 (2019-08-20) ------------------- * Fix web UI link to ArangoSearch views documentation. * Rebuild swagger for web UI and documentation. v3.5.0-rc.7 (2019-08-01) ------------------------ * Upgraded arangodb starter version to 0.14.12. v3.5.0-rc.6 (2019-07-29) ------------------------ * Fixed issue #9459: Optimization rule remove-collect-variables does not KEEP all necessary data. * Added gzip and encryption options to arangoimport and arangoexport. * Added missing REST API route GET /_api/transaction for retrieving the list of currently ongoing transactions. * Fixed issue #9558: RTRIM not working as expected. * Added startup error for bad temporary directory setting. If the temporary directory (--temp.path) setting is identical to the database directory (`--database.directory`) this can eventually lead to data loss, as temporary files may be created inside the temporary directory, causing overwrites of existing database files/directories with the same names. Additionally the temporary directory may be cleaned at some point, and this would lead to an unintended cleanup of the database files/directories as well. Now, if the database directory and temporary directory are set to the same path, there will be a startup warning about potential data loss (though in ArangoDB 3.4 allowing to continue the startup - in 3.5 and higher we will abort the startup). * Make TTL indexes behave like other indexes on creation. If a TTL index is already present on a collection, the previous behavior was to make subsequent calls to `ensureIndex` fail unconditionally with the error "there can only be one ttl index per collection". Now we are comparing the attributes of the to-be-created index with the attributes of the existing TTL index and make it only fail when the attributes differ. If the attributes are identical, the `ensureIndex` call succeeds and returns the existing index. v3.5.0-rc.5 (2019-07-22) ------------------------ * MinReplicationFactor: Collections can now be created with a minimal replication factor (minReplicationFactor), which defaults to 1. If minReplicationFactor > 1 a collection will go into "read-only" mode as soon as it has less than minReplicationFactor many in-sync followers. With this mechanism users can avoid to have collections diverge too much in case of failure scenarios. minReplicationFactor can have the values: 1 <= minReplicationFactor <= replicationFactor. Having minReplicationFactor == 1 ArangoDB behaves the same way as in any previous version. * Fixed a query abort error with SmartJoins if both collections were restricted to a single shard using the "restrict-to-single-shard" optimizer rule. * Fixed a performance regression of COLLECT WITH COUNT INTO. * Fixed some races in cluster collection creation, which allowed collections with the same name to be created in parallel under some rare conditions. * arangoimport would not stop, much less report, communications errors. Add CSV reporting of line numbers that are impacted during such errors * Prevent rare cases of duplicate DDL actions being executed by Maintenance. * Coordinator code was reporting rocksdb error codes, but not the associated detail message. Corrected. * The keep alive timeout specified via --http.keep-alive-timeout is now being honored * Replication requests on Document API are now on higher priority then client-triggered requests. This should help to keep in-sync replication up and running even if the server is overloaded. * Bugfix: Save distinct WAL ticks for multiple replication clients from the same server. Also, when a follower is added for synchronous replication, the WAL tick held by the client is freed immediately, rather than waiting for a timeout. The corresponding APIs get a new parameter `syncerId`, which, if given, supersedes `serverId`. This affects the routes /_api/wal/tail, /_api/replication/batch, /_api/replication/logger-follow and the internal route /_api/replication/addFollower. The new field `syncerId` is also added to the response of /_api/replication/logger-state. * Disallow indexing the `_id` attribute even as a sub-attribute. Previous versions of ArangoDB did not allow creating indexes on the `_id` attribute, but didn't check if an index was created on the `_id` attribute of a sub-attribute, e.g. `referredTo._id` or `data[*]._id`. Such indexes could be created with previous versions of ArangoDB, but they were non-functional. From now on, such indexes cannot be created anymore, and any attempts to create them will fail. * Added option `--replication.max-parallel-tailing-invocations` to limit the maximum number of concurrent WAL tailing invocations. The option can be used to limit the usage of the WAL tailing APIs in order to control server load. * Fixed agency bug with election lock step. * Fixed some error reporting and logging in Maintenance. * Fixed an error condition in which an ex-leader for a short still believed to be the leader and wrongly reported to Current. v3.5.0-rc.4 (2019-06-15) ------------------------ * Speed up collection creation process in cluster, if not all agency callbacks are delivered successfully. * Increased performance of document inserts, by reducing the number of checks in unique / primary indexes. * Fixed a callback function in the web UI where the variable `this` was out of scope. * Fixed editing a user within the web UI if the user added a gravatar profile picture. * Allow pregel to select the shard key via `shardKeyAttribute` in pregel start parameters. * Added --server.jwt-secret-keyfile to arangosh. * Fixed internal issue #4040: gharial api is now checking existence of `_from` and `_to` vertices during edge replacements and edge updates. * Fixed `Buffer.alloc` method. * `Buffer` is now iterable and accepts `ArrayBuffer` values as input. * Fix timeout-response in case of simultaneous index create/drop in cluster. * Enabled dropping an index by its name. * Fixed lookup of index from collection by fully qualified name, e.g. `db.testCollection.index('testCollection/primary')`. * Fixed agency bug with TTL object writes discovered in 3.4.6. v3.5.0-rc.3 (2019-05-31) ------------------------ * Fixed issue #9106: Sparse Skiplist Index on multiple fields not used for FILTER + SORT query. Allow AQL query optimizer to use sparse indexes in more cases, specifically when indexes could not be used for filtering and there finally was an `EnumerateCollectionNode` in the query execution plan followed by a `SortNode`. In this case, sparse indexes were not considered for enumeration in sorted order, because it was unclear to the optimizer if the result set would contain null values or not. * Upgraded RocksDB to version 6.2. * Updated ArangoDB Starter to 0.14.4. * The system collection '_jobs' will from now on be created with non-unique, non-sparse indexes. * Bugfix for smart graph traversals with uniqueVertices: path, which could sometimes lead to erroneous traversal results. * Pregel algorithms can be run with the option "useMemoryMaps: true" to be able to run algorithms on data that is bigger than the available RAM. * Fixed a race in TTL thread deactivation/shutdown. * Fixed internal issue #3919: The web UI is now using precompiled ejs templates. * Fixed agency issue in abort of cleanOutServer job. v3.5.0-rc.2 (2019-05-23) ------------------------ * Fixed "collection not found" exception during setup of 3-way SmartJoin queries in the cluster. * Fixed an edge case of handling `null` values in the AQL function `MIN` for input sequences that started with a `null` value. In this case, `null` was always returned as the minimum value even though other non-null values may have followed, and `MIN` was supposed to return `null` only if there are no input values or all input values are `null`. * Fixed a crash when posting an async request to the server using the "x-arango-async" request header and the server's request queue was full. * Added error code 1240 "incomplete read" for RocksDB-based reads which cannot retrieve documents due to the RocksDB block cache being size-restricted (with size limit enforced) and uncompressed data blocks not fitting into the block cache. The error can only occur for collection or index scans with the RocksDB storage engine when the RocksDB block cache is used and set to a very small size, plus its maximum size is enforced by setting the `--rocksdb.enforce-block-cache-size-limit` option to `true`. Previously these incomplete reads could have been ignored silently, making collection or index scans return less documents than there were actually present. * Fixed internal issue #3918: added optional second parameter "withId" to AQL function PREGEL_RESULT. This parameter defaults to `false`. When set to `true` the results of the Pregel computation run will also contain the `_id` attribute for each vertex and not just `_key`. This allows distinguishing vertices from different vertex collections. * Made Swagger UI work with HTTPS-enabled ArangoDBs too. * Improved scheduler performance for single-connection cases. * Internally switched unit tests framework from catch to gtest. * Disabled selection of index types "hash" and "skiplist" in the web interface when using the RocksDB engine. The index types "hash", "skiplist" and "persistent" are just aliases of each other with the RocksDB engine, so there is no need to offer all of them. After initially only offering "hash" indexes, we decided to only offer indexes of type "persistent", as it is technically the most appropriate description. * Fixed client id lookup table in state. v3.5.0-rc.1 (2019-05-14) ------------------------ * Removed bug during start up with a single agent, that leads to dbserver crash. * Fixed issue #7011: description when replacing a foxx application was misleading. * Fixed issue #8841: Graph Viewer dropped ability to edit an edge after rerendering. * Upgraded arangodb starter version to 0.14.3. * ArangoQueryStreamCursor.prototype.id needs to be a string, v8 32 bit integers can't hold the full data. * Upgraded Swagger UI to 3.22.1. * Added --compress-output flag to arangodump. Activates gzip compression for collection data. Metadata files, such as .structure.json and .view.json, do not get compressed. No option is needed for arangorestore to restore .data.json.gz files. * Added options to make server more secure: - `--server.harden`: denies access to certain REST APIs that return server internals - `--foxx.api`: set to false disables Foxx management API - `--foxx.store`: set to false disables Foxx UI - `--javascript.allow-port-testing`: enables internal.testPort() - `--javascript.allow-external-process-control`: enables external process control - `--javascript.harden`: disables getPid() and logLevel() - `--javascript.startup-options-whitelist`: control startup options visible in JavaScript - `--javascript.environment-variables-whitelist`: control environment variables visible in JavaScript - `--javascript.endpoints-whitelist`: control accessible endpoints in JavaScript - `--javascript.files-whitelist`: control file access in JavaScript Note: There is a [detailed description of all options](https://www.arangodb.com/docs/devel/security-security-options.html). * Prevented arangod from making a call to www.arangodb.com at startup. This call was done to check for available updates, but it could have contributed to small startup delays in case outgoing connections were blocked. * Removed support for undocumented HTTP header `x-arango-v8-context`, which allowed controlling in which particular V8 context number a JavaScript-based action was executed. This header was only used internally for testing. * `db._query` now handles additional arguments correctly when passing an AQL query object instead of a query string and separate bindVars. * Added req.auth property to Foxx. * Added collection.documentId method to derive document id from key. * Indexes created with the 'inBackground', will not hold an exclusive collection lock for the entire index creation period (rocksdb only). * Fixed internal issue #536: ArangoSearch may crash server during term lookup. * Fixed internal issue #2946: Create graph autocomplete was not working under certain circumstances. * Added `filter` option to Foxx HTTP API for running tests. * Added function `db..getResponsibleShard()` to find out which is the responsible shard for a given document. Note that this function is only available in a cluster coordinator. * Updated bundled version of jemalloc memory allocator to 5.2.0. * Don't create per-database system collection `_frontend` automatically. This collection is only needed by the web UI, and it can be created lazily when needed. * Added logging option `--log.time-format` to configure the time format used in log output. The possible values for this option are: - timestamp: unix timestamps, in seconds - timestamp-millis: unix timestamps, in seconds, with millisecond precision - timestamp-micros: unix timestamps, in seconds, with microsecond precision - uptime: seconds since server start - uptime-millis: seconds since server start, with millisecond precision - uptime-micros: seconds since server start, with microsecond precision - utc-datestring: UTC-based date and time in format YYYY-MM-DDTHH:MM:SSZ - utc-datestring-millis: UTC-based date and time in format YYYY-MM-DDTHH:MM:SS.FFFZ - local-datestring: local date and time in format YYYY-MM-DDTHH:MM:SS This change deprecates the existing options `--log.use-microtime` and `--log.use-localtime`, because the functionality provided by these options is covered by `--log.time-format` too. * Added "SmartJoins" to the ArangoDB Enterprise Edition that allows running cluster joins between two certain sharded collections with performance close to that of a local join operation. * Fixed internal issue #3815: fixed the removal of connected edges when removing a vertex graph node in a smart graph environment. * Show startup warning in case kernel setting `vm.overcommit_memory` is set to a value of 2 and the jemalloc memory allocator is in use. This combination does not play well together. * Added AQL function DECODE_REV for decomposing `_rev` values into their individual parts. * Added AQL functions CRC32 and FNV64 for hashing data. * Renamed attribute key `openssl-version` in server/client tool version details output to `openssl-version-compile-time`. This change affects the output produced when starting one of the ArangoDB executables with the `--version` command. It also changes the attribute name in the detailed response of the `/_api/version` REST API. * Fixed the sorting of the databases in the database selection dropdown in the web ui. The sort order differed based on whether authentication was enabled or disabled. (Internal issue #2276) * Improved the shards view in the web ui if there is only one shard to display. (Internal issue #3546) * Restricted the allowed query names for user defined custom queries within the web ui. (Internal issue #3789) * Upgraded bundled RocksDB version to 6.0. * Added "--log.ids" option to arangod. The default value for this option is `true`. Setting the option to `false` will prevent embedding unique log ids into all log messages generated by ArangoDB C++ code. The unique ids allow for easy access to the location in the source code from which a message originates. This should help customers to configure custom monitoring/alerting based on specific log id occurrences and will also be helpful for support and development in identifying problems. * Fixed wrong equals behavior on arrays with ArangoSearch. (Internal issue #8294) * Fixed ArangoSearch range query sometimes not working correctly with numeric values. (Internal issue #528) * Changed type of the startup option `--rocksdb.recycle-log-file-num` from numeric to boolean, as this is also the type the options has in the RocksDB library. * Renamed hidden startup option `--rocksdb.delayed_write_rate` to the more consistent `--rocksdb.delayed-write-rate`. When the old option name is used, the arangod startup will be aborted with a descriptive error message. * If not explicitly configured, make agency nodes start removing their unused WAL files a few seconds after the completed server startup already. This is because on agency nodes, unused WAL files do not need to be retained for potential replication clients to read them. * Added option `--all-databases` to arangodump and arangorestore. When set to true, this makes arangodump dump all available databases the current user has access to. The option `--all-databases` cannot be used in combination with the option `--server.database`. When `--all-databases` is used, arangodump will create a subdirectory with the data of each dumped database. Databases will be dumped one after the other. However, inside each database, the collections of the database can be dumped in parallel using multiple threads. For arangorestore, this makes it restore all databases from inside the subdirectories of the specified dump directory. Using the option for arangorestore only makes sense for dumps created with arangodump and the `--all-databases` option. As for arangodump, arangorestore cannot be invoked with the options `--all-databases` and `--server.database` at the same time. Additionally, the option `--force-same-database` cannot be used together with `--all-databases`. If the to-be-restored databases do not exist on the target server, then restoring data into them will fail unless the option `--create-database` is also specified. Please note that in this case a database user must be used that has access to the `_system` database, in order to create the databases on restore. * Added index hints feature to AQL. * Added "name" property for indices. If a name is not specified on index creation, one will be auto-generated. * Under normal circumstances there should be no need to connect to a database server in a cluster with one of the client tools, and it is likely that any user operations carried out there with one of the client tools may cause trouble. The client tools arangosh, arangodump and arangorestore will now emit a warning when connecting with them to a database server node in a cluster. * Fixed compaction behavior of followers. * Added "random" masking to mask any data type, added wildcard masking. * Added option `--console.history` to arangosh for controlling whether the command-line history should be loaded from and persisted in a file. The default value for this option is `true`. Setting it to `false` will make arangosh not load any command-line history from the history file, and not store the current session's history when the shell is exited. The command-line history will then only be available in the current shell session. * Display the server role when connecting arangosh against a server (e.g. SINGLE, COORDINATOR). * Fixed overflow in Windows NowNanos in RocksDB. * Allowed MoveShard from leader to a follower, thus swapping the two. * Supervision fix: Satellite collections, various fixes. * Added coordinator route for agency dump. * Supervision fix: abort MoveShard job does not leave a lock behind. * Supervision fix: abort MoveShard (leader) job moves forwards when point of no return has been reached. * Supervision fix: abort CleanOutServer job does not leave server in ToBeCleanedServers. * Supervision fix: move shard with data stopped to early due to wrong usage of compare function. * Supervision fix: AddFollower only counts good followers, fixing a situation after a FailedLeader job could not find a new working follower. * Supervision fix: FailedLeader now also considers temporarily BAD servers as replacement followers and does not block servers which currently receive a new shard. * Supervision fix: Servers in ToBeCleanedServers are no longer considered as replacement servers. * Maintenance fix: added precondition of unchanged Plan in phase2. * Added "PRUNE " to AQL Traversals. This allows to early abort searching of unnecessary branches within a traversal. PRUNE is only allowed in the Traversal statement and only between the graph definition and the options of the traversal. e.g.: `FOR v, e, p IN 1..3 OUTBOUND @source GRAPH "myGraph" PRUNE v.value == "bar" OPTIONS {} /* These options remain optional */ RETURN v` for more details refer to the documentation chapter. * Fixed a display issue when editing a graph within the web UI. * Fixed some escaping issues within the web UI. * Follow up to fix JWT authentication in arangosh (Internal issue #7530): also fixed reconnect. * Now also syncing _jobs and _queues collections in active failover mode. * Upgraded lodash to 4.17.11 because of CVE-2018-16487. * `--query.registry-ttl` is now honored in single-server mode, and cursor TTLs are now honored on DBServers in cluster mode. * Added "TTL" index type, for optional auto-expiration of documents. * Disabled selection of index types "persistent" and "skiplist" in the web interface when using the RocksDB engine. The index types "hash", "skiplist" and "persistent" are just aliases of each other with the RocksDB engine, so there is no need to offer all of them. * Fixed JS AQL query objects with empty query strings not being recognized as AQL queries. * Updated JavaScript dependencies, including semver major updates to joi, mocha and eslint. The eslint config modules were incompatible with the existing coding style, so the old rules were inlined and the config dependencies removed. Note that joi removed Joi.date().format() in v10.0.0. You can bundle your own version of joi if you need to rely on version-dependent features. - accepts: 1.3.4 -> 1.3.5 - ansi_up: 2.0.2 -> 4.0.3 - content-disposition: 0.5.2 -> 0.5.3 - dedent: 0.6.0 -> 0.7.0 - error-stack-parser: 1.3.6 -> 2.0.2 - eslint: 2.13.1 -> 5.16.0 - eslint-config-semistandard: 6.0.2 -> removed - eslint-config-standard: 5.3.1 -> removed - eslint-plugin-promise: 1.3.2 -> removed - eslint-plugin-standard: 1.3.2 -> removed - highlight.js: 9.12.0 -> 9.15.6 - http-errors: 1.6.2 -> 1.7.2 - iconv-lite: 0.4.19 -> 0.4.24 - joi: 9.2.0 -> 14.3.1 - joi-to-json-schema: 2.3.0 -> 4.0.1 - js-yaml: 3.10.0 -> 3.13.1 - marked: 0.3.9 -> 0.6.2 - mime-types: 2.1.12 -> 2.1.22 - mocha: 2.5.3 -> 6.1.3 - qs: 6.5.1 -> 6.7.0 - semver: 5.4.1 -> 6.0.0 - statuses: 1.4.0 -> 1.5.0 - timezone: 1.0.13 -> 1.0.22 - type-is: 1.6.15 -> 1.6.16 - underscore: 1.8.3 -> 1.9.1 * Updated V8 to 7.1.302.28. New V8 behavior introduced herein: - ES2016 changed the default timezone of date strings to be conditional on whether a time part is included. The semantics were a compromise approach based on web compatibility feedback from V8, but until now, we have been shipping ES5.1 default timezone semantics. This patch implements the new semantics, following ChakraCore and SpiderMonkey (though JSC implements V8's previous semantics). * Fixed JS AQL query objects with empty query strings not being recognized as AQL queries. * Report run-time openssl version (for dynamically linked executables). * Added greeting warning about maintainer mode. * Improved insertion time into non-unique secondary indexes with the RocksDB engine. * Fixed possible segfault when using COLLECT with a LIMIT and an offset. * Fixed COLLECT forgetting top-level variables after 1000 rows. * Added sort-limit optimization in AQL; Improved memory usage and execution time for some queries. * Upgraded to OpenSSL 1.1.0j. * Added configurable masking of dumped data via `arangodump` tool to obfuscate exported sensible data. * Fixed arangoimp script for MacOSX CLI Bundle. * Added "peakMemoryUsage" in query results figures, showing the peak memory usage of the executed query. In a cluster, the value the peak memory usage of all shards, but it is not summed up across shards. * Fixed an issue where a crashed coordinator can lead to some Foxx queue jobs erroneously either left hanging or being restarted. * Fixed bind values of `null` are not replaced by empty string anymore, when toggling between json and table view in the web-ui. (Internal issue #7900) * Fixed regression on ISO8601 string compatibility in AQL. (Internal issue #2786) millisecond parts of AQL date values were limited to up to 3 digits. Now the length of the millisecond part is unrestricted, but the millisecond precision is still limited to up to 3 digits. * The RocksDB primary index can now be used by the optimizer to optimize queries that use `_key` or `_id` for sorting or for range queries. * The web UI will now by default show the documents of a collection lexicographically sorted when sorting documents by their `_key` values. Previous versions of ArangoDB tried to interpret `_key` values as numeric values if possible and sorted by these. That previous sort strategy never used an index and could have caused unnecessary overhead. The new version will now use an index for sorting for the RocksDB engine, but may change the order in which documents are shown in the web UI (e.g. now a `_key` value of "10" will be shown before a `_key` value of "9"). * Fixed known issue #445: ArangoSearch ignores `_id` attribute even if `includeAllFields` is set to `true`. * Upgraded bundled boost library to version 1.69.0. * Upgraded bundled curl library to version 7.63. * Used base64url to encode and decode JWT parts. * Added --server.jwt-secret-keyfile option. * Sped up data-modification operations in exclusive transactions in the RocksDB storage engine. * An AQL query that uses the edge index only and returns the opposite side of the edge can now be executed in a more optimized way, e.g. FOR edge IN edgeCollection FILTER edge._from == "v/1" RETURN edge._to is fully covered by RocksDB edge index. For MMFiles this rule does not apply. * Reverted accidental change to error handling in geo index. In previous versions, if non-valid geo coordinates were contained in the indexed field of a document, the document was simply ignored an not indexed. In 3.4.0, this was accidentally changed to generate an error, which caused the upgrade procedure to break in some cases. * Fixed TypeError being thrown instead of validation errors when Foxx manifest validation fails. * Improved confirmation dialog when clicking the Truncate button in the Web UI. (Internal issue #2786) * Made `--help-all` now also show all hidden program options. Previously hidden program options were only returned when invoking arangod or a client tool with the cryptic `--help-.` option. Now `--help-all` simply returns them as well. The program options JSON description returned by `--dump-options` was also improved as follows: - the new boolean attribute "dynamic" indicates whether the option has a dynamic default value, i.e. a value that depends on the target host capabilities or configuration - the new boolean attribute "requiresValue" indicates whether a boolean option requires a value of "true" or "false" when specified. If "requiresValue" is false, then the option can be specified without a boolean value following it, and the option will still be set to true, e.g. `--server.authentication` is identical to `--server.authentication true`. - the new "category" attribute will contain a value of "command" for command-like options, such as `--version`, `--dump-options`, `--dump-dependencies` etc., and "option" for all others. * Fixed issue #7586: a running query within the user interface was not shown if the active view was `Running Queries` or `Slow Query History`. * Fixed issue #7743: Query processing discrepancy between Rocks and MMFiles databases. This change enforces the invalidation of variables in AQL queries after usage of a COLLECT statement as documented. The documentation for variable invalidation claims that The COLLECT statement will eliminate all local variables in the current scope. After COLLECT only the variables introduced by COLLECT itself are available. However, the described behavior was not enforced when a COLLECT was preceded by a FOR loop that was itself preceded by a COLLECT. In the following query the final RETURN statement accesses variable `key1` though the variable should have been invalidated by the COLLECT directly before it: `FOR x1 IN 1..2 COLLECT key1 = x1 FOR x2 IN 1..2 COLLECT key2 = x2 RETURN [key2, key1]` In previous releases, this query was parsed ok, but the contents of variable `key1` in the final RETURN statement were undefined. This change is about making queries as the above fail with a parse error, as an unknown variable `key1` is accessed here, avoiding the undefined behavior. This is also in line with what the documentation states about variable invalidation. * Fixed issue #7763: Collect after update does not execute updates. * Fixed issue #7749: AQL query result changed for COLLECT used on empty data/array. * Fixed issue #7757: Using multiple filters on nested objects produces wrong results. * Fixed a rare thread local dead lock situation in replication: If a follower tries to get in sync in the last steps it requires a lock on the leader. If the follower cancels the lock before the leader has succeeded with locking we can end up with one thread being deadlocked. * Allowed usage of floating point values in AQL without leading zeros, e.g. `.1234`. Previous versions of ArangoDB required a leading zero in front of the decimal separator, i.e `0.1234`. * Foxx `req.makeAbsolute` now will return meaningful values when ArangoDB is using a unix socket endpoint. URLs generated when using a unix socket follow the format http://unix:: used by other JS tooling. * Updated joi library (Web UI), improved foxx mount path validation. * Do not create `_routing` collection for new installations/new databases, as it is not needed anymore. Redirects to the web interface's login screen, which were previously handled by entries in the `_routing` collection are now handled from the responsible REST action handler directly. Existing `_routing` collections will not be touched as they may contain other entries as well, and will continue to work. * Do not create `_modules` collection for new databases/installations. `_modules` is only needed for custom modules, and in case a custom module is defined via `defineModule`, the _modules collection will be created lazily automatically. Existing modules in existing `_modules` collections will remain functional even after this change * Disabled in-memory cache for edge and traversal data on agency nodes, as it is not needed there. * Removed bundled Valgrind headers, removed JavaScript variable `valgrind` from the `internal` module. * Upgraded JEMalloc version to 5.1.0. * Use `-std=c++14` for ArangoDB compilation. * In case of resigned leader, set `isReady=false` in clusterInventory. * Abort RemoveFollower job if not enough in-sync followers or leader failure. * Fixed shrinkCluster for satelliteCollections. * Fixed crash in agency supervision when leadership is lost. * Leader's timestamps in append entries protocol. * Sped up agency supervision in case of many jobs. * Fixed log spam in agency supervision when leader resigned. * Made AddFollower less aggressive. * Several agency performance improvements, mostly avoiding copying. * Priority queue for maintenance jobs. * Do not wait for replication after each job execution in Supervision. * Added support for configuring custom Analyzers via JavaScript and REST. v3.4.7 (2019-07-02) ------------------- * updated arangosync to 0.6.4 * bug-fix for a very rare race condition on cluster collection creation process. It can only occur in the following situation: 1) DBServer successfully creates all shards for collections, and reports it back 2) DBServer dies 3) Agency recognized death and reorganizes shards 4) Coordinator receives (1) but not (3) and decided everything was good 5) => Coordinator reverts (3) only on this collection 6) Coordinator receives (3) now The bugfix disallows (5) if (3) is run in Agency. * fixed internal issue #4040: gharial api is now checking existence of `_from` and `_to` vertices during edge replacements and edge updates * fix timeout-response in case of simultaneous index create/drop in cluster * Fixed editing a user within the web UI if the user added a gravatar profile picture * fixed internal issue #4172: on agent servers the RocksDB WAL files in the archive directory were retained due to an error. Now they are removed, eventually * fix error reporting in arangorestore. previously, when the server returned an HTTP 404 error, arangorestore unconditionally translated this into a "collection not found" error, which may have masked an actually different server-side error. Instead, make arangorestore return the server's actual error message as is. * allow pregel to select the shard key via `shardKeyAttribute` in pregel start parameters * Speed up collection creation process in cluster, if not all agency callbacks are delivered successfully. * Fixed parsing of ArangoDB config files with inlined comments. Previous versions didn't handle line comments properly if they were appended to an otherwise valid option value. For example, the comment in the line max-total-wal-size = 1024000 # 1M was not ignored and made part of the value. In the end, the value was interpreted as if max-total-wal-size = 10240001000000 was specified. This version fixes the handling of comments in the config files so that they behave as intended. * Fixed AQL query tracing for traversals, this lead to wrong "calls" count in query profiles * Pregel algorithms can be run with the option "useMemoryMaps: true" to be able to run algorithms on data that is bigger than the available RAM. * Bugfix for smart graph traversals with uniqueVertices: path, which could sometimes lead to erroneous traversal results * The system-collection '_jobs' will from now on use non-unique, non-sparse indexes. * fixed internal issue #3919: The web UI is now using precompiled ejs templates. * properly create VERSION files for any newly created databases again, as it has been the case in 3.3 and before. If no VERSION files are created, this may cause problems on upgrade to 3.5. The 3.5 branch was also modified to handle this problem gracefully. * fixed "collection not found" exception during setup of 3-way SmartJoin queries in the cluster * fixed an edge case of handling `null` values in the AQL function `MIN` for input sequences that started with a `null` value. In this case, `null` was always returned as the minimum value even though other non-null values may have followed, and `MIN` was supposed to return `null` only if there are no input values or all input values are `null`. * fix client id lookup table in agency * Fix agency's handling of object assignments with TTL * reduce resident memory usage when creating non-unique indexes with the RocksDB storage engine * Fix agency election bug. v3.4.6.1 (2019-06-05) --------------------- * upgraded ArangoDB Starter version to 0.14.5 * Speed up collection creation process in cluster, if not all agency callbacks are delivered successfully. * Fix agency's handling of object assignments with TTL v3.4.6 (2019-05-21) ------------------- * updated ArangoDB Starter to 0.14.4 * fixed internal issue #3912: improved the performance of graph creation with multiple relations. They do now create multiple collections within a single round-trip in the agency. * fixed a crash when posting an async request to the server using the "x-arango-async" request header and the server's request queue was full * added error code 1240 "incomplete read" for RocksDB-based reads which cannot retrieve documents due to the RocksDB block cache being size-restricted (with size limit enforced) and uncompressed data blocks not fitting into the block cache The error can only occur for collection or index scans with the RocksDB storage engine when the RocksDB block cache is used and set to a very small size, plus its maximum size is enforced by setting the `--rocksdb.enforce-block-cache-size-limit` option to `true`. Previously these incomplete reads could have been ignored silently, making collection or index scans return less documents than there were actually present. * fixed internal issue #3918: added optional second parameter "withId" to AQL function PREGEL_RESULT this parameter defaults to `false`. When set to `true` the results of the Pregel computation run will also contain the `_id` attribute for each vertex and not just `_key`. This allows distinguishing vertices from different vertex collections. * fixed issue #7011: description when replacing a foxx application was misleading * fixed issue #8841: Graph Viewer dropped ability to edit an edge after rerendering. * upgraded arangodb starter version to 0.14.3 * fix removal of sort order for sorted results that were passed into a RETURN DISTINCT in an AQL query running on a collection with more than a single shard. In this case, the results from the individual shards were sorted, but the overall result was not sorted according to the sort criterion. * removed bug during start up with a single agent, that leads to dbserver crash. * fix the creation of debug packages (in the web interface) for queries that involve smart graphs and/or multiple edge collections from a traversal * add --compress-output flag to arangodump. Activates gzip compression for collection data. Metadata files, such as .structure.json and .view.json, do not get compressed. No option is needed for arangorestore to restore .data.json.gz files. * fixed internal issue #3893: correct wrong option name `--database.ignore-logfile-errors` in error message and correct it to `--database.ignore-datafile-errors` * fixed internal issue #2946: create graph autocomplete was not working under certain circumstances. * fixed a memory leak in PRUNE operation for traversals. The leak occurred only when PRUNE was actively used in queries. * fixed a crash (SIGSEGV) when opening a RocksDB database directory that contained an empty (0 bytes filesize) journal file and encryption was in use. * improve test performance by doing the 0.1 delay between phase 1 and 2 only if phase 1 is slow * port back timestamp replication in agency from devel * better logging when scheduler queue is half-full and full * fixed internal issue #536: ArangoSearch query may crash during internal lookup in some cases due to invalid index structure for exact input data v3.4.5 (2019-03-27) ------------------- * fixed a shutdown issue when the server was shut down while there were active Pregel jobs executing * fixed internal issue #3815: fixed the removal of connected edges when removing a vertex graph node in a smart graph environment. * added AQL functions CRC32 and FNV64 for hashing data * internal issue #2276: fixed the sorting of the databases in the database selection dropdown in the web ui. The sort order differed based on whether authentication was enabled or disabled. * fixed internal issue #3789: restricted the allowed query names for user- defined custom queries within the web ui. * fixed internal issue #3546: improved the shards view in the web ui if there is only one shard to display. * fixed a display issues when editing a graph within the web ui * fixed internal-issue #3787: Let user know about conflicting attribute in AQL queries if information is available. * fixed issue #8294: wrong equals behavior on arrays with ArangoSearch * fixed internal issue #528: ArangoSearch range query sometimes doesn't work correctly with numeric values * fixed internal issue #3757: when restarting a follower in active failover mode, try an incremental sync instead of a full resync. Fixed also the case where a double resync was made * don't check for the presence of ArangoDB upgrades available when firing up an arangosh enterprise edition build * added startup option `--rocksdb.allow-fallocate` When set to true, allows RocksDB to use the fallocate call. If false, fallocate calls are bypassed and no preallocation is done. Preallocation is turned on by default, but can be turned off for operating system versions that are known to have issues with it. This option only has an effect on operating systems that support fallocate. * added startup option `--rocksdb.limit-open-files-at-startup` If set to true, this will limit the amount of .sst files RocksDB will inspect at startup, which can reduce the number of IO operations performed at start. * don't run compact() on a collection after a truncate() was done in the same transaction running compact() in the same transaction will only increase the data size on disk due to RocksDB not being able to remove any documents physically due to the snapshot that is taken at transaction start. This change also exposes db..compact() in the arangosh, in order to manually run a compaction on the data range of a collection should it be needed for maintenance. * don't attempt to remove non-existing WAL files, because such attempt will trigger unnecessary error log messages in the RocksDB library * updated arangosync to 0.6.3 * added --log.file-mode to specify a file mode of newly created log files * added --log.file-group to specify the group of newly created log files * fixed some escaping issues within the web ui. * fixed issue #8359: How to update a document with unique index (constraints)? * when restarting a follower in active failover mode, try an incremental sync instead of a full resync * add "PRUNE " to AQL Traversals (internal issue #3068). This allows to early abort searching of unnecessary branches within a traversal. PRUNE is only allowed in the Traversal statement and only between the graphdefinition and the options of the traversal. e.g.: FOR v, e, p IN 1..3 OUTBOUND @source GRAPH "myGraph" PRUNE v.value == "bar" OPTIONS {} /* These options remain optional */ RETURN v for more details refer to the documentation chapter. * added option `--console.history` to arangosh for controlling whether the command-line history should be loaded from and persisted in a file. The default value for this option is `true`. Setting it to `false` will make arangosh not load any command-line history from the history file, and not store the current session's history when the shell is exited. The command-line history will then only be available in the current shell session. * display the server role when connecting arangosh against a server (e.g. SINGLE, COORDINATOR) * added replication applier state figures `totalDocuments` and `totalRemovals` to access the number of document insert/replace operations and the number of document removals operations separately. Also added figures `totalApplyTime` and `totalFetchTime` for determining the total time the replication spent for applying changes or fetching new data from the master. Also added are the figures `averageApplyTime` and `averageFetchTime`, which show the average time spent for applying a batch or for fetching data from the master, resp. * fixed race condition in which the value of the informational replication applier figure `ticksBehind` could underflow and thus show a very huge number of ticks. * always clear all ongoing replication transactions on the slave if the slave discovers the data it has asked for is not present anymore on the master and the `requireFromPresent` value for the applier is set to `false`. In this case aborting the ongoing transactions on the slave is necessary because they may have held exclusive locks on collections, which may otherwise not be released. * added option `--rocksdb.wal-archive-size-limit` for controlling the maximum total size (in bytes) of archived WAL files. The default is 0 (meaning: unlimited). When setting the value to a size bigger than 0, the RocksDB storage engine will force a removal of archived WAL files if the total size of the archive exceeds the configured size. The option can be used to get rid of archived WAL files in a disk size-constrained environment. Note that archived WAL files are normally deleted automatically after a short while when there is no follower attached that may read from the archive. However, in case when there are followers attached that may read from the archive, WAL files normally remain in the archive until their contents have been streamed to the followers. In case there are slow followers that cannot catch up this will cause a growth of the WAL files archive over time. The option `--rocksdb.wal-archive-size-limit` can now be used to force a deletion of WAL files from the archive even if there are followers attached that may want to read the archive. In case the option is set and a leader deletes files from the archive that followers want to read, this will abort the replication on the followers. Followers can however restart the replication doing a resync. * agents need to be able to overwrite a compacted state with same _key * in case of resigned leader, set isReady=false in clusterInventory * abort RemoveFollower job if not enough in-sync followers or leader failure * fix shrinkCluster for satelliteCollections * fix crash in agency supervision when leadership is lost * speed up supervision in agency for large numbers of jobs * fix log spamming after leader resignation in agency * make AddFollower less aggressive * fix cases, where invalid json could be generated in agents' store dumps * coordinator route for full agency dumps contains compactions and time stamps * lots of agency performance improvements, mostly avoiding copying * priority queue for maintenance jobs * do not wait for replication after each job execution in Supervision * fix a blockage in MoveShard if a failover happens during the operation * check health of servers in Current before scheduling removeFollower jobs * wait for statistics collections to be created before running resilience tests * fix ttl values in agency when key overwritten with no ttl v3.4.4 (2019-03-12) ------------------- * added missing test for success in failed leader: this could lead to a crash * follow up to fix JWT authentication in arangosh (#7530): also fix reconnect * now also syncing _jobs and _queues collections in active failover mode * fixed overflow in Windows NowNanos in RocksDB * fixed issue #8165: AQL optimizer does not pick up multiple geo index * when creating a new database with an initial user, set the database permission for this user as specified in the documentation * Supervision fix: abort MoveShard job does not leave a lock behind, * Supervision fix: abort MoveShard (leader) job moves forwards when point of no return has been reached, * Supervision fix: abort CleanOutServer job does not leave server in ToBeCleanedServers, * Supervision fix: move shard with data stopped to early due to wrong usage of compare function * Supervision fix: AddFollower only counts good followers, fixing a situation after a FailedLeader job could not find a new working follower * Supervision fix: FailedLeader now also considers temporarily BAD servers as replacement followers and does not block servers which currently receive a new shard * Supervision fix: Servers in ToBeCleanedServers are no longer considered as replacement servers * Maintenance fix: added precondition of unchanged Plan in phase2 * Allow MoveShard from leader to a follower, thus swapping the two * Supervision fix: SatelliteCollections, various fixes * Add coordinator route for agency dump * speed up replication of transactions containing updates of existing documents. The replication protocol does not provide any information on whether a document was inserted on the master or updated/replaced. Therefore the slave will always try an insert first, and move to a replace if the insert fails with "unique constraint violation". This case is however very costly in a bigger transaction, as the rollback of the insert will force the underlying RocksDB write batch to be entirely rewritten. To circumvent rewriting entire write batches, we now do a quick check if the target document already exists, and then branch to either insert or replace internally. v3.4.3 (2019-02-19) ------------------- * fixed JS AQL query objects with empty query strings not being recognized as AQL queries * fixed issue #8137: NULL input field generates U_ILLEGAL_ARGUMENT_ERROR * fixed issue #8108: AQL variable - not working query since upgrade to 3.4 release * fixed possible segfault when using COLLECT with a LIMIT and an offset * fixed COLLECT forgetting top-level variables after 1000 rows * fix undefined behavior when calling user-defined AQL functions from an AQL query via a streaming cursor * fix broken validation of tick range in arangodump * updated bundled curl library to version 7.63.0 * added "peakMemoryUsage" in query results figures, showing the peak memory usage of the executed query. In a cluster, the value contains the peak memory usage across all shards, but it is not summed up across shards. * data masking: better documentation, fixed default phone number, changed default range to -100 and 100 for integer masking function * fix supervision's failed server handling to transactionally create all failed leader/followers along v3.4.2.1 (2019-02-01) --------------------- * upgrade to new velocypack version v3.4.2 (2019-01-24) ------------------- * added configurable masking of dumped data via `arangodump` tool to obfuscate exported sensible data * upgraded to OpenSSL 1.1.0j * fixed an issue with AQL query IN index lookup conditions being converted into empty arrays when they were shared between multiple nodes of a lookup condition that used an IN array lookup in an OR that was multiplied due to DNF transformations This issue affected queries such as the following FILTER (... && ...) || doc.indexAttribute IN non-empty-array * upgraded arangodb starter version to 0.14.0 * upgraded arangosync version to 0.6.2 * fixed an issue where a crashed coordinator can lead to some Foxx queue jobs erroneously either left hanging or being restarted * fix issue #7903: Regression on ISO8601 string compatibility in AQL millisecond parts of AQL date values were limited to up to 3 digits. Now the length of the millisecond part is unrestricted, but the millisecond precision is still limited to up to 3 digits. * fix issue #7900: Bind values of `null` are not replaced by empty string anymore, when toggling between json and table view in the web-ui. * Use base64url to encode and decode JWT parts. * added AQL function `CHECK_DOCUMENT` for document validity checks * when detecting parse errors in the JSON input sent to the restore API, now abort with a proper error containing the problem description instead of aborting but hiding there was a problem. * do not respond with an internal error in case of JSON parse errors detected in incoming HTTP requests * added arangorestore option `--cleanup-duplicate-attributes` to clean up input documents with redundant attribute names Importing such documents without the option set will make arangorestore fail with an error, and setting the option will make the restore process clean up the input by using just the first specified value for each redundant attribute. * the arangorestore options `--default-number-of-shards` and `--default-replication-factor` are now deprecated in favor of the much more powerful options `--number-of-shards` and `--replication-factor` The new options `--number-of-shards` and `--replication-factor` allow specifying default values for the number of shards and the replication factor, resp. for all restored collections. If specified, these default values will be used regardless of whether the number of shards or the replication factor values are already present in the metadata of the dumped collections. It is also possible to override the values on a per-collection level by specifying the options multiple times, e.g. --number-of-shards 2 --number-of-shards mycollection=3 --number-of-shards test=4 The above will create all collections with 2 shards, except the collection "mycollection" (3 shards) and "test" (4 shards). By omitting the default value, it is also possible to use the number of shards/replication factor values from the dump for all collections but the explicitly specified ones, e.g. --number-of-shards mycollection=3 --number-of-shards test=4 This will use the number of shards as specified in the dump, except for the collections "mycollection" and "test". The `--replication-factor` option works similarly. * validate uniqueness of attribute names in AQL in cases in which it was not done before. When constructing AQL objects via object literals, there was no validation about object attribute names being unique. For example, it was possible to create objects with duplicate attribute names as follows: INSERT { a: 1, a: 2 } INTO collection This resulted in a document having two "a" attributes, which is obviously undesired. Now, when an attribute value is used multiple times, only the first assigned value will be used for that attribute in AQL. It is not possible to specify the same attribute multiple times and overwrite the attribute's value with by that. That means in the above example, the value of "a" will be 1, and not 2. This changes the behavior for overriding attribute values in AQL compared to previous versions of ArangoDB, as previous versions in some cases allowed duplicate attribute names in objects/documents (which is undesired) and in other cases used the _last_ value assigned to an attribute instead of the _first_ value. In order to explicitly override a value in an existing object, use the AQL MERGE function. To avoid all these issues, users are encouraged to use unambiguous attribute names in objects/documents in AQL. Outside of AQL, specifying the same attribute multiple times may even result in a parse error, e.g. when sending such data to ArangoDB's HTTP REST API. * fixed issue #7834: AQL Query crashes instance * Added --server.jwt-secret-keyfile option. * Improve single threaded performance by scheduler optimization. * Releveling logging in maintenance v3.4.1 (2018-12-19) ------------------- * fixed issue #7757: Using multiple filters on nested objects produces wrong results * fixed issue #7763: Collect after update does not execute updates * fixed issue #7586: a running query within the user interface was not shown if the active view was `Running Queries` or `Slow Query History`. * fixed issue #7749: AQL Query result changed for COLLECT used on empty data/array * fixed a rare thread local dead lock situation in replication: If a follower tries to get in sync in the last steps it requires a lock on the leader. If the follower cancels the lock before the leader has succeeded with locking we can end up with one thread being deadlocked. * fix thread shutdown in _WIN32 builds Previous versions used a wrong comparison logic to determine the current thread id when shutting down a thread, leading to threads hanging in their destructors on thread shutdown * reverted accidental change to error handling in geo index In previous versions, if non-valid geo coordinates were contained in the indexed field of a document, the document was simply ignored an not indexed. In 3.4.0, this was accidentally changed to generate an error, which caused the upgrade procedure to break in some cases. * fixed TypeError being thrown instead of validation errors when Foxx manifest validation fails * make AQL REMOVE operations use less memory with the RocksDB storage engine the previous implementation of batch removals read everything to remove into memory first before carrying out the first remove operation. The new version will only read in about 1000 documents each time and then remove these. Queries such as FOR doc IN collection FILTER ... REMOVE doc IN collection will benefit from this change in terms of memory usage. * make `--help-all` now also show all hidden program options Previously hidden program options were only returned when invoking arangod or a client tool with the cryptic `--help-.` option. Now `--help-all` simply returns them as well. The program options JSON description returned by `--dump-options` was also improved as follows: - the new boolean attribute "dynamic" indicates whether the option has a dynamic default value, i.e. a value that depends on the target host capabilities or configuration - the new boolean attribute "requiresValue" indicates whether a boolean option requires a value of "true" or "false" when specified. If "requiresValue" is false, then the option can be specified without a boolean value following it, and the option will still be set to true, e.g. `--server.authentication` is identical to `--server.authentication true`. - the new "category" attribute will contain a value of "command" for command-like options, such as `--version`, `--dump-options`, `--dump-dependencies` etc., and "option" for all others. * Fixed a bug in synchronous replication initialization for, where a shard's db server is rebooted during that period v3.4.0 (2018-12-06) ------------------- * Add license key checking to enterprise version in Docker containers. v3.4.0-rc.5 (2018-11-29) ------------------------ * Persist and check default language (locale) selection. Previously we would not check if the language (`--default-language`) had changed when the server was restarted. This could cause issues with indexes over text fields, as it will resulted in undefined behavior within RocksDB (potentially missing entries, corruption, etc.). Now if the language is changed, ArangoDB will print out an error message on startup and abort. * fixed issue #7522: FILTER logic totally broke for my query in 3.4-rc4 * export version and storage engine in `_admin/cluster/health` for Coordinators and DBServers. * restrict the total amount of data to build up in all in-memory RocksDB write buffers by default to a certain fraction of the available physical RAM. This helps restricting memory usage for the arangod process, but may have an effect on the RocksDB storage engine's write performance. In ArangoDB 3.3 the governing configuration option `--rocksdb.total-write-buffer-size` had a default value of `0`, which meant that the memory usage was not limited. ArangoDB 3.4 now changes the default value to about 50% of available physical RAM, and 512MiB for setups with less than 4GiB of RAM. * lower default value for `--cache.size` startup option from about 30% of physical RAM to about 25% percent of physical RAM. * fix internal issue #2786: improved confirmation dialog when clicking the truncate button in the web UI * Updated joi library (web UI), improved Foxx mount path validation * disable startup warning for Linux kernel variable `vm.overcommit_memory` settings values of 0 or 1. Effectively `overcommit_memory` settings value of 0 or 1 fix two memory-allocation related issues with the default memory allocator used in ArangoDB release builds on 64bit Linux. The issues will remain when running with an `overcommit_memory` settings value of 2, so this is now discouraged. Setting `overcommit_memory` to 0 or 1 (0 is the Linux kernel's default) fixes issues with increasing numbers of memory mappings for the arangod process (which may lead to an out-of-memory situation if the kernel's maximum number of mappings threshold is hit) and an increasing amount of memory that the kernel counts as "committed". With an `overcommit_memory` setting of 0 or 1, an arangod process may either be killed by the kernel's OOM killer or will die with a segfault when accessing memory it has allocated before but the kernel could not provide later on. This is still more acceptable than the kernel not providing any more memory to the process when there is still physical memory left, which may have occurred with an `overcommit_memory` setting of 2 after the arangod process had done lots of allocations. In summary, the recommendation for the `overcommit_memory` setting is now to set it to 0 or 1 (0 is kernel default) and not use 2. * fixed Foxx complaining about valid `$schema` value in manifest.json * fix for supervision, which started failing servers using old transient store * fixed a bug where indexes are used in the cluster while still being built on the db servers * fix move leader shard: wait until all but the old leader are in-sync. This fixes some unstable tests. * cluster health features more elaborate agent records * agency's supervision edited for advertised endpoints v3.4.0-rc.4 (2018-11-04) ------------------------ * fixed Foxx queues not retrying jobs with infinite `maxFailures` * increase AQL query string parsing performance for queries with many (100K+) string values contained in the query string * increase timeouts for inter-node communication in the cluster * fixed undefined behavior in `/_api/import` when importing a single document went wrong * replication bugfixes * stop printing `connection class corrupted` in arangosh when just starting the arangosh without a connection to a server and running code such as `require("internal")`, the shell always printed "connection class corrupted", which was somewhat misleading. * add separate option `--query.slow-streaming-threshold` for tracking slow streaming queries with a different timeout value * increase maximum number of collections/shards in an AQL query from 256 to 2048 * don't rely on `_modules` collection being present and usable for arangod startup * force connection timeout to be 7 seconds to allow libcurl time to retry lost DNS queries. * fixes a routing issue within the web ui after the use of views * fixes some graph data parsing issues in the ui, e.g. cleaning up duplicate edges inside the graph viewer. * in a cluster environment, the arangod process now exits if wrong credentials are used during the startup process. * added option `--rocksdb.total-write-buffer-size` to limit total memory usage across all RocksDB in-memory write buffers * suppress warnings from statistics background threads such as `WARNING caught exception during statistics processing: Expecting Object` during version upgrade v3.4.0-rc.3 (2018-10-23) ------------------------ * fixed handling of broken Foxx services Installation now also fails when the service encounters an error when executed. Upgrading or replacing with a broken service will still result in the broken services being installed. * restored error pages for broken Foxx services Services that could not be executed will now show an error page (with helpful information if development mode is enabled) instead of a generic 404 response. Requests to the service that do not prefer HTML (i.e. not a browser window) will receive a JSON formatted 503 error response instead. * added support for `force` flag when upgrading Foxx services Using the `force` flag when upgrading or replacing a service falls back to installing the service if it does not already exist. * The order of JSON object attribute keys in JSON return values will now be "random" in more cases. In JSON, there is no defined order for object attribute keys anyway, so ArangoDB is taking the freedom to return the attribute keys in a non-deterministic, seemingly unordered way. * Fixed an AQL bug where the `optimize-traversals` rule was falsely applied to extensions with inline expressions and thereby ignoring them * fix side-effects of sorting larger arrays (>= 16 members) of constant literal values in AQL, when the array was used not only for IN-value filtering but also later in the query. The array values were sorted so the IN-value lookup could use a binary search instead of a linear search, but this did not take into account that the array could have been used elsewhere in the query, e.g. as a return value. The fix will create a copy of the array and sort the copy, leaving the original array untouched. * disallow empty LDAP password * fixes validation of allowed or not allowed foxx service mount paths within the Web UI * The single database or single coordinator statistics in a cluster environment within the Web UI sometimes got called way too often. This caused artifacts in the graphs, which is now fixed. * An aardvark statistics route could not collect and sum up the statistics of all coordinators if one of them was ahead and had more results than the others * Web UI now checks if server statistics are enabled before it sends its first request to the statistics API * fix internal issue #486: immediate deletion (right after creation) of a view with a link to one collection and indexed data reports failure but removes the link * fix internal issue #480: link to a collection is not added to a view if it was already added to other view * fix internal issues #407, #445: limit ArangoSearch memory consumption so that it won't cause OOM while indexing large collections * upgraded arangodb starter version to 0.13.5 * removed undocumented `db..toArray()` function from ArangoShell * prevent creation of collections and views with the same in cluster setups * fixed issue #6770: document update: ignoreRevs parameter ignored * added AQL query optimizer rules `simplify-conditions` and `fuse-filters` * improve inter-server communication performance: - move all response processing off Communicator's socket management thread - create multiple Communicator objects with ClusterComm, route via round robin - adjust Scheduler threads to always be active, and have designated priorities. * fix internal issue #2770: the Query Profiling modal dialog in the Web UI was slightly malformed. * fix internal issue #2035: the Web UI now updates its indices view to check whether new indices exist or not. * fix internal issue #6808: newly created databases within the Web UI did not appear when used Internet Explorer 11 as a browser. * fix internal issue #2957: the Web UI was not able to display more than 1000 documents, even when it was set to a higher amount. * fix internal issue #2688: the Web UI's graph viewer created malformed node labels if a node was expanded multiple times. * fix internal issue #2785: web ui's sort dialog sometimes got rendered, even if it should not. * fix internal issue #2764: the waitForSync property of a satellite collection could not be changed via the Web UI * dynamically manage libcurl's number of open connections to increase performance by reducing the number of socket close and then reopen cycles * recover short server id from agency after a restart of a cluster node this fixes problems with short server ids being set to 0 after a node restart, which then prevented cursor result load-forwarding between multiple coordinators to work properly this should fix arangojs#573 * increased default timeouts in replication this decreases the chances of followers not getting in sync with leaders because of replication operations timing out * include forward-ported diagnostic options for debugging LDAP connections * fixed internal issue #3065: fix variable replacements by the AQL query optimizer in arangosearch view search conditions The consequence of the missing replacements was that some queries using view search conditions could have failed with error messages such as "missing variable #3 (a) for node #7 (EnumerateViewNode) while planning registers" * fixed internal issue #1983: the Web UI was showing a deletion confirmation multiple times. * Restricted usage of views in AQL, they will throw an error now (e.g. "FOR v, e, p IN 1 OUTBOUND @start edgeCollection, view") instead of failing the server. * Allow VIEWs within the AQL "WITH" statement in cluster environment. This will now prepare the query for all collections linked within a view. (e.g. "WITH view FOR v, e, p IN OUTBOUND 'collectionInView/123' edgeCollection" will now be executed properly and not fail with unregistered collection any more) * Properly check permissions for all collections linked to a view when instantiating an AQL query in cluster environment * support installation of ArangoDB on Windows into directories with multibyte character filenames on Windows platforms that used a non-UTF8-codepage This was supported on other platforms before, but never worked for ArangoDB's Windows version * display shard synchronization progress for collections outside of the `_system` database * change memory protection settings for memory given back to by the bundled JEMalloc memory allocator. This avoids splitting of existing memory mappings due to changes of the protection settings * added missing implementation for `DeleteRangeCF` in RocksDB WAL tailing handler * fixed agents busy looping gossip * handle missing `_frontend` collections gracefully the `_frontend` system collection is not required for normal ArangoDB operations, so if it is missing for whatever reason, ensure that normal operations can go on. v3.4.0-rc.2 (2018-09-30) ------------------------ * upgraded arangosync version to 0.6.0 * upgraded arangodb starter version to 0.13.3 * fixed issue #6611: Properly display JSON properties of user defined foxx services configuration within the web UI * improved shards display in web UI: included arrows to better visualize that collection name sections can be expanded and collapsed * added nesting support for `aql` template strings * added support for `undefined` and AQL literals to `aql.literal` * added `aql.join` function * fixed issue #6583: Agency node segfaults if sent an authenticated HTTP request is sent to its port * fixed issue #6601: Context cancelled (never ending query) * added more AQL query results cache inspection and control functionality * fixed undefined behavior in AQL query result cache * the query editor within the web UI is now catching HTTP 501 responses properly * added AQL VERSION function to return the server version as a string * added startup parameter `--cluster.advertised-endpoints` * AQL query optimizer now makes better choices regarding indexes to use in a query when there are multiple competing indexes and some of them are prefixes of others In this case, the optimizer could have preferred indexes that covered less attributes, but it should rather pick the indexes that covered more attributes. For example, if there was an index on ["a"] and another index on ["a", "b"], then previously the optimizer may have picked the index on just ["a"] instead the index on ["a", "b"] for queries that used all index attributes but did range queries on them (e.g. `FILTER doc.a == @val1 && doc.b >= @val2`). * Added compression for the AQL intermediate results transfer in the cluster, leading to less data being transferred between coordinator and database servers in many cases * forward-ported a bugfix from RocksDB (https://github.com/facebook/rocksdb/pull/4386) that fixes range deletions (used internally in ArangoDB when dropping or truncating collections) The non-working range deletes could have triggered errors such as `deletion check in index drop failed - not all documents in the index have been deleted.` when dropping or truncating collections * improve error messages in Windows installer * allow retrying installation in Windows installer in case an existing database is still running and needs to be manually shut down before continuing with the installation * fix database backup functionality in Windows installer * fixed memory leak in `/_api/batch` REST handler * `db._profileQuery()` now also tracks operations triggered when using `LIMIT` clauses in a query * added proper error messages when using views as an argument to AQL functions (doing so triggered an `internal error` before) * fixed return value encoding for collection ids ("cid" attribute") in REST API `/_api/replication/logger-follow` * fixed dumping and restoring of views with arangodump and arangorestore * fix replication from 3.3 to 3.4 * fixed some TLS errors that occurred when combining HTTPS/TLS transport with the VelocyStream protocol (VST) That combination could have led to spurious errors such as "TLS padding error" or "Tag mismatch" and connections being closed * make synchronous replication detect more error cases when followers cannot apply the changes from the leader * fixed issue #6379: RocksDB arangorestore time degeneration on dead documents * fixed issue #6495: Document not found when removing records * fixed undefined behavior in cluster plan-loading procedure that may have unintentionally modified a shared structure * reduce overhead of function initialization in AQL COLLECT aggregate functions, for functions COUNT/LENGTH, SUM and AVG this optimization will only be noticable when the COLLECT produces many groups and the "hash" COLLECT variant is used * fixed potential out-of-bounds access in admin log REST handler `/_admin/log`, which could have led to the server returning an HTTP 500 error * catch more exceptions in replication and handle them appropriately * agency endpoint updates now go through RAFT * fixed a cleanup issue in Current when a follower was removed from Plan * catch exceptions in MaintenanceWorker thread * fixed a bug in cleanOutServer which could lead to a cleaned out server still being a follower for some shard v3.4.0-rc.1 (2018-09-06) ------------------------ * Release Candidate for 3.4.0, please check the `ReleaseNotes/KnownIssues34.md` file for a list of known issues. * upgraded bundled RocksDB version to 5.16.0 * upgraded bundled Snappy compression library to 1.1.7 * fixed issue #5941: if using breadth first search in traversals uniqueness checks on path (vertices and edges) have not been applied. In SmartGraphs the checks have been executed properly. * added more detailed progress output to arangorestore, showing the percentage of how much data is restored for bigger collections plus a set of overview statistics after each processed collection * added option `--rocksdb.use-file-logging` to enable writing of RocksDB's own informational LOG files into RocksDB's database directory. This option is turned off by default, but can be enabled for debugging RocksDB internals and performance. * improved error messages when managing Foxx services Install/replace/upgrade will now provide additional information when an error is encountered during setup. Errors encountered during a `require` call will also include information about the underlying cause in the error message. * fixed some Foxx script names being displayed incorrectly in web UI and Foxx CLI * major revision of the maintenance feature * added `uuidv4` and `genRandomBytes` methods to crypto module * added `hexSlice` methods `hexWrite` to JS Buffer type * added `Buffer.from`, `Buffer.of`, `Buffer.alloc` and `Buffer.allocUnsafe` for improved compatibility with Node.js * Foxx HTTP API errors now log stacktraces * fixed issue #5831: custom queries in the ui could not be loaded if the user only has read access to the _system database. * fixed issue #6128: ArangoDb Cluster: Task moved from DBS to Coordinator * fixed some web ui action events related to Running Queries view and Slow Queries History view * fixed internal issue #2566: corrected web UI alignment of the nodes table * fixed issue #5736: Foxx HTTP API responds with 500 error when request body is too short * fixed issue #6106: Arithmetic operator type casting documentation incorrect * The arangosh now supports the velocystream transport protocol via the schemas "vst+tcp://", "vst+ssl://", "vst+unix://" schemes. * The server will no longer lowercase the input in --server.endpoint. This means Unix domain socket paths will now be treated as specified, previously they were lowercased * fixed logging of requests. A wrong log level was used * fixed issue #5943: misplaced database ui icon and wrong cursor type were used * fixed issue #5354: updated the web UI JSON editor, improved usability * fixed issue #5648: fixed error message when saving unsupported document types * fixed internal issue #2812: Cluster fails to create many indexes in parallel * Added C++ implementation, load balancer support, and user restriction to Pregel API. If an execution is accessed on a different coordinator than where it was created, the request(s) will be forwarded to the correct coordinator. If an execution is accessed by a different user than the one who created it, the request will be denied. * the AQL editor in the web UI now supports detailed AQL query profiling * fixed issue #5884: Subquery nodes are no longer created on DBServers * intermediate commits in the RocksDB engine are now only enabled in standalone AQL queries (not within a JS transaction), standalone truncate as well as for the "import" API * the AQL editor in the web UI now supports GeoJSON types and is able to render them. * fixed issue #5035: fixed a vulnerability issue within the web ui's index view * PR #5552: add "--latency true" option to arangoimport. Lists microsecond latency * added `"pbkdf2"` method to `@arangodb/foxx/auth` module * the `@arangodb/foxx/auth` module now uses a different method to generate salts, so salts are no longer guaranteed to be alphanumeric * fixed internal issue #2567: the Web UI was showing the possibility to move a shard from a follower to the current leader * Renamed RocksDB engine-specific statistics figure `rocksdb.block-cache-used` to `rocksdb.block-cache-usage` in output of `db._engineStats()` The new figure name is in line with the statistics that the RocksDB library provides in its new versions. * Added RocksDB engine-specific statistics figures `rocksdb.block-cache-capacity`, `rocksdb.block-cache-pinned-usage` as well as level-specific figures `rocksdb.num-files-at-level` and `rocksdb.compression-ratio-at-level` in output of `db._engineStats()` * Added RocksDB-engine configuration option `--rocksdb.block-align-data-blocks` If set to true, data blocks are aligned on lesser of page size and block size, which may waste some memory but may reduce the number of cross-page I/Os operations. * Usage RocksDB format version 3 for new block-based tables * Bugfix: The AQL syntax variants `UPDATE/REPLACE k WITH d` now correctly take _rev from k instead of d (when ignoreRevs is false) and ignore d._rev. * Added C++ implementation, load balancer support, and user restriction to tasks API If a task is accessed on a different coordinator than where it was created, the request(s) will be forwarded to the correct coordinator. If a task is accessed by a different user than the one who created it, the request will be denied. * Added load balancer support and user-restriction to async jobs API. If an async job is accessed on a different coordinator than where it was created, the request(s) will be forwarded to the correct coordinator. If a job is accessed by a different user than the one who created it, the request will be denied. * switch default storage engine from MMFiles to RocksDB In ArangoDB 3.4, the default storage engine for new installations is the RocksDB engine. This differs to previous versions (3.2 and 3.3), in which the default storage engine was the MMFiles engine. The MMFiles engine can still be explicitly selected as the storage engine for all new installations. It's only that the "auto" setting for selecting the storage engine will now use the RocksDB engine instead of MMFiles engine. In the following scenarios, the effectively selected storage engine for new installations will be RocksDB: * `--server.storage-engine rocksdb` * `--server.storage-engine auto` * `--server.storage-engine` option not specified The MMFiles storage engine will be selected for new installations only when explicitly selected: * `--server.storage-engine mmfiles` On upgrade, any existing ArangoDB installation will keep its previously selected storage engine. The change of the default storage engine is thus only relevant for new ArangoDB installations and/or existing cluster setups for which new server nodes get added later. All server nodes in a cluster setup should use the same storage engine to work reliably. Using different storage engines in a cluster is unsupported. * added collection.indexes() as an alias for collection.getIndexes() * disable V8 engine and JavaScript APIs for agency nodes * renamed MMFiles engine compactor thread from "Compactor" to "MMFilesCompactor". This change will be visible only on systems which allow assigning names to threads. * added configuration option `--rocksdb.sync-interval` This option specifies interval (in milliseconds) that ArangoDB will use to automatically synchronize data in RocksDB's write-ahead log (WAL) files to disk. Automatic syncs will only be performed for not-yet synchronized data, and only for operations that have been executed without the *waitForSync* attribute. Automatic synchronization is performed by a background thread. The default sync interval is 100 milliseconds. Note: this option is not supported on Windows platforms. Setting the sync interval to a value greater 0 will produce a startup warning. * added AQL functions `TO_BASE64`, `TO_HEX`, `ENCODE_URI_COMPONENT` and `SOUNDEX` * PR #5857: RocksDB engine would frequently request a new DelayToken. This caused excessive write delay on the next Put() call. Alternate approach taken. * changed the thread handling in the scheduler. `--server.maximal-threads` will be the maximum number of threads for the scheduler. * The option `--server.threads` is now obsolete. * use sparse indexes in more cases now, when it is clear that the index attribute value cannot be null * introduce SingleRemoteOperationNode via "optimize-cluster-single-document-operations" optimizer rule, which triggers single document operations directly from the coordinator instead of using a full-featured AQL setup. This saves cluster roundtrips. Queries directly referencing the document key benefit from this: UPDATE {_key: '1'} WITH {foo: 'bar'} IN collection RETURN OLD * Added load balancer support and user-restriction to cursor API. If a cursor is accessed on a different coordinator than where it was created, the requests will be forwarded to the correct coordinator. If a cursor is accessed by a different user than the one who created it, the request will be denied. * if authentication is turned on requests to databases by users with insufficient rights will be answered with the HTTP forbidden (401) response. * upgraded bundled RocksDB library version to 5.15 * added key generators `uuid` and `padded` The `uuid` key generator generates universally unique 128 bit keys, which are stored in hexadecimal human-readable format. The `padded` key generator generates keys of a fixed length (16 bytes) in ascending lexicographical sort order. * The REST API of `/_admin/status` added: "operationMode" filed with same meaning as the "mode" field and field "readOnly" that has the inverted meaning of the field "writeOpsEnabled". The old field names will be deprecated in upcoming versions. * added `COUNT_DISTINCT` AQL function * make AQL optimizer rule `collect-in-cluster` optimize aggregation functions `AVERAGE`, `VARIANCE`, `STDDEV`, `UNIQUE`, `SORTED_UNIQUE` and `COUNT_DISTINCT` in a cluster by pushing parts of the aggregation onto the DB servers and only doing the total aggregation on the coordinator * replace JavaScript functions FULLTEXT, NEAR, WITHIN and WITHIN_RECTANGLE with regular AQL subqueries via a new optimizer rule "replace-function-with-index". * the existing "fulltext-index-optimizer" optimizer rule has been removed because its duty is now handled by the "replace-function-with-index" rule. * added option "--latency true" option to arangoimport. Lists microsecond latency statistics on 10 second intervals. * fixed internal issue #2256: ui, document id not showing up when deleting a document * fixed internal issue #2163: wrong labels within foxx validation of service input parameters * fixed internal issue #2160: fixed misplaced tooltips in indices view * Added exclusive option for rocksdb collections. Modifying AQL queries can now set the exclusive option as well as it can be set on JavaScript transactions. * added optimizer rule "optimize-subqueries", which makes qualifying subqueries return less data The rule fires in the following situations: * in case only a few results are used from a non-modifying subquery, the rule will add a LIMIT statement into the subquery. For example LET docs = ( FOR doc IN collection FILTER ... RETURN doc ) RETURN docs[0] will be turned into LET docs = ( FOR doc IN collection FILTER ... LIMIT 1 RETURN doc ) RETURN docs[0] Another optimization performed by this rule is to modify the result value of subqueries in case only the number of results is checked later. For example RETURN LENGTH( FOR doc IN collection FILTER ... RETURN doc ) will be turned into RETURN LENGTH( FOR doc IN collection FILTER ... RETURN true ) This saves copying the document data from the subquery to the outer scope and may enable follow-up optimizations. * fixed Foxx queues bug when queues are created in a request handler with an ArangoDB authentication header * abort startup when using SSLv2 for a server endpoint, or when connecting with a client tool via an SSLv2 connection. SSLv2 has been disabled in the OpenSSL library by default in recent versions because of security vulnerabilities inherent in this protocol. As it is not safe at all to use this protocol, the support for it has also been stopped in ArangoDB. End users that use SSLv2 for connecting to ArangoDB should change the protocol from SSLv2 to TLSv12 if possible, by adjusting the value of the `--ssl.protocol` startup option. * added `overwrite` option to document insert operations to allow for easier syncing. This implements almost the much inquired UPSERT. In reality it is a REPSERT (replace/insert) because only replacement and not modification of documents is possible. The option does not work in cluster collections with custom sharding. * added startup option `--log.escape` This option toggles the escaping of log output. If set to `true` (which is the default value), then the logging will work as before, and the following characters in the log output are escaped: * the carriage return character (hex 0d) * the newline character (hex 0a) * the tabstop character (hex 09) * any other characters with an ordinal value less than hex 20 If the option is set to `false`, no characters are escaped. Characters with an ordinal value less than hex 20 will not be printed in this mode but will be replaced with a space character (hex 20). A side effect of turning off the escaping is that it will reduce the CPU overhead for the logging. However, this will only be noticable when logging is set to a very verbose level (e.g. debug or trace). * increased the default values for the startup options `--javascript.gc-interval` from every 1000 to every 2000 requests, and for `--javascript.gc-frequency` from 30 to 60 seconds This will make the V8 garbage collection run less often by default than in previous versions, reducing CPU load a bit and leaving more contexts available on average. * added `/_admin/repair/distributeShardsLike` that repairs collections with distributeShardsLike where the shards aren't actually distributed like in the prototype collection, as could happen due to internal issue #1770 * Fixed issue #4271: Change the behavior of the `fullCount` option for AQL query cursors so that it will only take into account `LIMIT` statements on the top level of the query. `LIMIT` statements in subqueries will not have any effect on the `fullCount` results any more. * We added a new geo-spatial index implementation. On the RocksDB storage engine all installations will need to be upgraded with `--database.auto-upgrade true`. New geo indexes will now only report with the type `geo` instead of `geo1` or `geo2`. The index types `geo1` and `geo2` are now deprecated. Additionally we removed the deprecated flags `constraint` and `ignoreNull` from geo index definitions, these fields were initially deprecated in ArangoDB 2.5 * Add revision id to RocksDB values in primary indexes to speed up replication (~10x). * PR #5238: Create a default pacing algorithm for arangoimport to avoid TimeoutErrors on VMs with limited disk throughput * Starting a cluster with coordinators and DB servers using different storage engines is unsupported. Doing it anyway will now produce a warning on startup * fixed issue #4919: C++ implementation of LIKE function now matches the old and correct behavior of the javascript implementation. * added `--json` option to arangovpack, allowing to treat its input as plain JSON data make arangovpack work without any configuration file * added experimental arangodb startup option `--javascript.enabled` to enable/disable the initialization of the V8 JavaScript engine. Only expected to work on single-servers and agency deployments * pull request #5201: eliminate race scenario where handlePlanChange could run infinite times after an execution exceeded 7.4 second time span * UI: fixed an unreasonable event bug within the modal view engine * pull request #5114: detect shutdown more quickly on heartbeat thread of coordinator and DB servers * fixed issue #3811: gharial api is now checking existence of `_from` and `_to` vertices during edge creation * There is a new method `_profileQuery` on the database object to execute a query and print an explain with annotated runtime information. * Query cursors can now be created with option `profile`, with a value of 0, 1 or 2. This will cause queries to include more statistics in their results and will allow tracing of queries. * fixed internal issue #2147: fixed database filter in UI * fixed internal issue #2149: number of documents in the UI is not adjusted after moving them * fixed internal issue #2150: UI - loading a saved query does not update the list of bind parameters * removed option `--cluster.my-local-info` in favor of persisted server UUIDs The option `--cluster.my-local-info` was deprecated since ArangoDB 3.3. * added new collection property `cacheEnabled` which enables in-memory caching for documents and primary index entries. Available only when using RocksDB * arangodump now supports `--threads` option to dump collections in parallel * arangorestore now supports `--threads` option to restore collections in parallel * Improvement: The AQL query planner in cluster is now a bit more clever and can prepare AQL queries with less network overhead. This should speed up simple queries in cluster mode, on complex queries it will most likely not show any performance effect. It will especially show effects on collections with a very high amount of Shards. * removed remainders of dysfunctional `/_admin/cluster-test` and `/_admin/clusterCheckPort` API endpoints and removed them from documentation * added new query option `stream` to enable streaming query execution via the `POST /_api/cursor` rest interface. * fixed issue #4698: databases within the UI are now displayed in a sorted order. * Behavior of permissions for databases and collections changed: The new fallback rule for databases for which an access level is not explicitly specified: Choose the higher access level of: * A wildcard database grant * A database grant on the `_system` database The new fallback rule for collections for which an access level is not explicitly specified: Choose the higher access level of: * Any wildcard access grant in the same database, or on "*/*" * The access level for the current database * The access level for the `_system` database * fixed issue #4583: add AQL ASSERT and AQL WARN * renamed startup option `--replication.automatic-failover` to `--replication.active-failover` using the old option name will still work in ArangoDB 3.4, but the old option will be removed afterwards * index selectivity estimates for RocksDB engine are now eventually consistent This change addresses a previous issue where some index updates could be "lost" from the view of the internal selectivity estimate, leading to inaccurate estimates. The issue is solved now, but there can be up to a second or so delay before updates are reflected in the estimates. * support `returnOld` and `returnNew` attributes for in the following HTTP REST APIs: * /_api/gharial//vertex/ * /_api/gharial//edge/ The exception from this is that the HTTP DELETE verb for these APIs does not support `returnOld` because that would make the existing API incompatible * fixed internal issue #478: remove unused and undocumented REST API endpoints _admin/statistics/short and _admin/statistics/long These APIs were available in ArangoDB's REST API, but have not been called by ArangoDB itself nor have they been part of the documented API. They have been superseded by other REST APIs and were partially dysfunctional. Therefore these two endpoints have been removed entirely. * fixed issue #1532: reload users on restore * fixed internal issue #1475: when restoring a cluster dump to a single server ignore indexes of type primary and edge since we mustn't create them here. * fixed internal issue #1439: improve performance of any-iterator for RocksDB * issue #1190: added option `--create-database` for arangoimport * UI: updated dygraph js library to version 2.1.0 * renamed arangoimp to arangoimport for consistency Release packages will still install arangoimp as a symlink so user scripts invoking arangoimp do not need to be changed * UI: Shard distribution view now has an accordion view instead of displaying all shards of all collections at once. * fixed issue #4393: broken handling of unix domain sockets in JS_Download * added AQL function `IS_KEY` this function checks if the value passed to it can be used as a document key, i.e. as the value of the `_key` attribute * added AQL functions `SORTED` and `SORTED_UNIQUE` `SORTED` will return a sorted version of the input array using AQL's internal comparison order `SORTED_UNIQUE` will do the same, but additionally removes duplicates. * added C++ implementation for AQL functions `DATE_NOW`, `DATE_ISO8601`, `DATE_TIMESTAMP`, `IS_DATESTRING`, `DATE_DAYOFWEEK`, `DATE_YEAR`, `DATE_MONTH`, `DATE_DAY`, `DATE_HOUR`, `DATE_MINUTE`, `DATE_SECOND`, `DATE_MILLISECOND`, `DATE_DAYOFYEAR`, `DATE_ISOWEEK`, `DATE_LEAPYEAR`, `DATE_QUARTER`, `DATE_DAYS_IN_MONTH`, `DATE_ADD`, `DATE_SUBTRACT`, `DATE_DIFF`, `DATE_COMPARE`, `TRANSLATE` and `SHA512` * fixed a bug where clusterinfo missed changes to plan after agency callback is registered for create collection * Foxx manifest.json files can now contain a $schema key with the value of "http://json.schemastore.org/foxx-manifest" to improve tooling support. * fixed agency restart from compaction without data * fixed agency's log compaction for internal issue #2249 * only load Plan and Current from agency when actually needed v3.3.21 (XXXX-XX-XX) -------------------- * fixed TypeError being thrown instead of validation errors when Foxx manifest validation fails * fixed issue #7586: a running query within the user interface was not shown if the active view was `Running Queries` or `Slow Query History`. * improve Windows installer error messages, fix Windows installer backup routine and exit code handling * make AQL REMOVE operations use less memory with the RocksDB storage engine the previous implementation of batch removals read everything to remove into memory first before carrying out the first remove operation. The new version will only read in about 1000 documents each time and then remove these. Queries such as FOR doc IN collection FILTER ... REMOVE doc IN collection will benefit from this change in terms of memory usage. v3.3.20 (2018-11-28) -------------------- * upgraded arangodb starter version to 0.13.9 * Added RocksDB option `--rocksdb.total-write-buffer-size` to limit total memory usage across all RocksDB in-memory write buffers The total amount of data to build up in all in-memory buffers (backed by log files). This option, together with the block cache size configuration option, can be used to limit memory usage. If set to 0, the memory usage is not limited. This is the default setting in 3.3. The default setting may be adjusted in future versions of ArangoDB. If set to a value greater than 0, this will cap the memory usage for write buffers, but may have an effect on write performance. * Added RocksDB configuration option `--rocksdb.enforce-block-cache-size-limit` Whether or not the maximum size of the RocksDB block cache is strictly enforced. This option can be set to limit the memory usage of the block cache to at most the specified size. If then inserting a data block into the cache would exceed the cache's capacity, the data block will not be inserted. If the flag is not set, a data block may still get inserted into the cache. It is evicted later, but the cache may temporarily grow beyond its capacity limit. * Export version and storage engine in cluster health * Potential fix for issue #7407: arangorestore very slow converting from mmfiles to rocksdb * Updated joi library (Web UI), improved foxx mount path validation * fix internal issue #2786: improved confirmation dialog when clicking the Truncate button in the Web UI * fix for supervision, which started failing servers using old transient store * fixed Foxx queues not retrying jobs with infinite `maxFailures` * Fixed a race condition in a coordinator, it could happen in rare cases and only with the maintainer mode enabled if the creation of a collection is in progress and at the same time a deletion is forced. * disable startup warning for Linux kernel variable `vm.overcommit_memory` settings values of 0 or 1. Effectively `overcommit_memory` settings value of 0 or 1 fix two memory-allocation related issues with the default memory allocator used in ArangoDB release builds on 64bit Linux. The issues will remain when running with an `overcommit_memory` settings value of 2, so this is now discouraged. Setting `overcommit_memory` to 0 or 1 (0 is the Linux kernel's default) fixes issues with increasing numbers of memory mappings for the arangod process (which may lead to an out-of-memory situation if the kernel's maximum number of mappings threshold is hit) and an increasing amount of memory that the kernel counts as "committed". With an `overcommit_memory` setting of 0 or 1, an arangod process may either be killed by the kernel's OOM killer or will die with a segfault when accessing memory it has allocated before but the kernel could not provide later on. This is still more acceptable than the kernel not providing any more memory to the process when there is still physical memory left, which may have occurred with an `overcommit_memory` setting of 2 after the arangod process had done lots of allocations. In summary, the recommendation for the `overcommit_memory` setting is now to set it to 0 or 1 (0 is kernel default) and not use 2. * force connection timeout to be 7 seconds to allow libcurl time to retry lost DNS queries. * increase maximum number of collections/shards in an AQL query from 256 to 2048 * don't rely on `_modules` collection being present and usable for arangod startup * optimizes the web ui's routing which could possibly led to unwanted events. * fixes some graph data parsing issues in the ui, e.g. cleaning up duplicate edges inside the graph viewer. * in a cluster environment, the arangod process now exits if wrong credentials are used during the startup process. * Fixed an AQL bug where the optimize-traversals rule was falsely applied to extensions with inline expressions and thereby ignoring them * fix side-effects of sorting larger arrays (>= 16 members) of constant literal values in AQL, when the array was not used only for IN-value filtering but also later in the query. The array values were sorted so the IN-value lookup could use a binary search instead of a linear search, but this did not take into account that the array could have been used elsewhere in the query, e.g. as a return value. The fix will create a copy of the array and sort the copy, leaving the original array untouched. * fixed a bug when cluster indexes were usable for queries, while still being built on db servers * fix move leader shard: wait until all but the old leader are in sync. This fixes some unstable tests. * cluster health features more elaborate agent records v3.3.19 (2018-10-20) -------------------- * fixes validation of allowed or not allowed foxx service mount paths within the Web UI * The single database or single coordinator statistics in a cluster environment within the Web UI sometimes got called way too often. This caused artifacts in the graphs, which is now fixed. * An aardvark statistics route could not collect and sum up the statistics of all coordinators if one of them was ahead and had more results than the others * upgraded arangodb starter version to 0.13.6 * turn on intermediate commits in replication applier in order to decrease the size of transactional operations on replication (issue #6821) * fixed issue #6770: document update: ignoreRevs parameter ignored * when returning memory to the OS, use the same memory protection flags as when initializing the memory this prevents "hole punching" and keeps the OS from splitting one memory mapping into multiple mappings with different memory protection settings * fix internal issue #2770: the Query Profiling modal dialog in the Web UI was slightly malformed. * fix internal issue #2035: the Web UI now updates its indices view to check whether new indices exist or not. * fix internal issue #6808: newly created databases within the Web UI did not appear when used Internet Explorer 11 as a browser. * fix internal issue #2688: the Web UI's graph viewer created malformed node labels if a node was expanded multiple times. * fix internal issue #2957: the Web UI was not able to display more than 1000 documents, even when it was set to a higher amount. * fix internal issue #2785: web ui's sort dialog sometimes got rendered, even if it should not. * fix internal issue #2764: the waitForSync property of a satellite collection could not be changed via the Web UI * improved logging in case of replication errors * recover short server id from agency after a restart of a cluster node this fixes problems with short server ids being set to 0 after a node restart, which then prevented cursor result load-forwarding between multiple coordinators to work properly this should fix arangojs#573 * increased default timeouts in replication this decreases the chances of followers not getting in sync with leaders because of replication operations timing out * fixed internal issue #1983: the Web UI was showing a deletion confirmation multiple times. * fixed agents busy looping gossip * handle missing `_frontend` collections gracefully the `_frontend` system collection is not required for normal ArangoDB operations, so if it is missing for whatever reason, ensure that normal operations can go on. v3.3.18 ------- * not released v3.3.17 (2018-10-04) -------------------- * upgraded arangosync version to 0.6.0 * added several advanced options for configuring and debugging LDAP connections. Please note that some of the following options are platform-specific and may not work on all platforms or with all LDAP servers reliably: - `--ldap.serialized`: whether or not calls into the underlying LDAP library should be serialized. This option can be used to work around thread-unsafe LDAP library functionality. - `--ldap.serialize-timeout`: sets the timeout value that is used when waiting to enter the LDAP library call serialization lock. This is only meaningful when `--ldap.serialized` has been set to `true`. - `--ldap.retries`: number of tries to attempt a connection. Setting this to values greater than one will make ArangoDB retry to contact the LDAP server in case no connection can be made initially. - `--ldap.restart`: whether or not the LDAP library should implicitly restart connections - `--ldap.referrals`: whether or not the LDAP library should implicitly chase referrals - `--ldap.debug`: turn on internal OpenLDAP library output (warning: will print to stdout). - `--ldap.timeout`: timeout value (in seconds) for synchronous LDAP API calls (a value of 0 means default timeout). - `--ldap.network-timeout`: timeout value (in seconds) after which network operations following the initial connection return in case of no activity (a value of 0 means default timeout). - `--ldap.async-connect`: whether or not the connection to the LDAP library will be done asynchronously. * fixed a shutdown race in ArangoDB's logger, which could have led to some buffered log messages being discarded on shutdown * display shard synchronization progress for collections outside of the `_system` database * fixed issue #6611: Properly display JSON properties of user defined foxx services configuration within the web UI * fixed issue #6583: Agency node segfaults if sent an authenticated HTTP request is sent to its port * when cleaning out a leader it could happen that it became follower instead of being removed completely * make synchronous replication detect more error cases when followers cannot apply the changes from the leader * fix some TLS errors that occurred when combining HTTPS/TLS transport with the VelocyStream protocol (VST) That combination could have led to spurious errors such as "TLS padding error" or "Tag mismatch" and connections being closed * agency endpoint updates now go through RAFT v3.3.16 (2018-09-19) -------------------- * fix undefined behavior in AQL query result cache * the query editor within the web ui is now catching http 501 responses properly * fixed issue #6495 (Document not found when removing records) * fixed undefined behavior in cluster plan-loading procedure that may have unintentionally modified a shared structure * reduce overhead of function initialization in AQL COLLECT aggregate functions, for functions COUNT/LENGTH, SUM and AVG this optimization will only be noticable when the COLLECT produces many groups and the "hash" COLLECT variant is used * fixed potential out-of-bounds access in admin log REST handler /_admin/log, which could have led to the server returning an HTTP 500 error * catch more exceptions in replication and handle them appropriately v3.3.15 (2018-09-10) -------------------- * fixed an issue in the "sorted" AQL COLLECT variant, that may have led to producing an incorrect number of results * upgraded arangodb starter version to 0.13.3 * fixed issue #5941 if using breadth-first search in traversals uniqueness checks on path (vertices and edges) have not been applied. In SmartGraphs the checks have been executed properly. * added more detailed progress output to arangorestore, showing the percentage of how much data is restored for bigger collections plus a set of overview statistics after each processed collection * added option `--rocksdb.use-file-logging` to enable writing of RocksDB's own informational LOG files into RocksDB's database directory. This option is turned off by default, but can be enabled for debugging RocksDB internals and performance. * improved error messages when managing Foxx services Install/replace/upgrade will now provide additional information when an error is encountered during setup. Errors encountered during a `require` call will also include information about the underlying cause in the error message. * fixed some Foxx script names being displayed incorrectly in web UI and Foxx CLI * added startup option `--query.optimizer-max-plans value` This option allows limiting the number of query execution plans created by the AQL optimizer for any incoming queries. The default value is `128`. By adjusting this value it can be controlled how many different query execution plans the AQL query optimizer will generate at most for any given AQL query. Normally the AQL query optimizer will generate a single execution plan per AQL query, but there are some cases in which it creates multiple competing plans. More plans can lead to better optimized queries, however, plan creation has its costs. The more plans are created and shipped through the optimization pipeline, the more time will be spent in the optimizer. Lowering this option's value will make the optimizer stop creating additional plans when it has already created enough plans. Note that this setting controls the default maximum number of plans to create. The value can still be adjusted on a per-query basis by setting the *maxNumberOfPlans* attribute when running a query. This change also lowers the default maximum number of query plans from 192 to 128. * bug fix: facilitate faster shutdown of coordinators and db servers * cluster nodes should retry registering in agency until successful * fixed some web ui action events related to Running Queries view and Slow Queries History view * Create a default pacing algorithm for arangoimport to avoid TimeoutErrors on VMs with limited disk throughput * backport PR 6150: establish unique function to indicate when application is terminating and therefore network retries should not occur * backport PR #5201: eliminate race scenario where handlePlanChange could run infinite times after an execution exceeded 7.4 second time span v3.3.14 (2018-08-15) -------------------- * upgraded arangodb starter version to 0.13.1 * Foxx HTTP API errors now log stacktraces * fixed issue #5736: Foxx HTTP API responds with 500 error when request body is too short * fixed issue #5831: custom queries in the ui could not be loaded if the user only has read access to the _system database. * fixed internal issue #2566: corrected web UI alignment of the nodes table * fixed internal issue #2869: when attaching a follower with global applier to an authenticated leader already existing users have not been replicated, all users created/modified later are replicated. * fixed internal issue #2865: dumping from an authenticated arangodb the users have not been included * fixed issue #5943: misplaced database ui icon and wrong cursor type were used * fixed issue #5354: updated the web UI JSON editor, improved usability * fixed issue #5648: fixed error message when saving unsupported document types * fixed issue #6076: Segmentation fault after AQL query This also fixes issues #6131 and #6174 * fixed issue #5884: Subquery nodes are no longer created on DBServers * fixed issue #6031: Broken LIMIT in nested list iterations * fixed internal issue #2812: Cluster fails to create many indexes in parallel * intermediate commits in the RocksDB engine are now only enabled in standalone AQL queries (not within a JS transaction), standalone truncate as well as for the "import" API * Bug fix: race condition could request data from Agency registry that did not exist yet. This caused a throw that would end the Supervision thread. All registry query APIs no longer throw exceptions. v3.3.13 (2018-07-26) -------------------- * fixed internal issue #2567: the Web UI was showing the possibility to move a shard from a follower to the current leader * fixed issue #5977: Unexpected execution plan when subquery contains COLLECT * Bugfix: The AQL syntax variants `UPDATE/REPLACE k WITH d` now correctly take _rev from k instead of d (when ignoreRevs is false) and ignore d._rev. * put an upper bound on the number of documents to be scanned when using `db..any()` in the RocksDB storage engine previous versions of ArangoDB did a scan of a random amount of documents in the collection, up to the total number of documents available. this produced a random selection with a good quality, but needed to scan half the number of documents in the collection on average. The new version will only scan up to 500 documents, so it produces a less random result, but will be a lot faster especially for large collections. The implementation of `any()` for the MMFiles engine remains unchanged. The MMFiles engine will pick a random document from the entire range of the in-memory primary index without performing scans. * return an empty result set instead of an "out of memory" exception when querying the geo index with invalid (out of range) coordinates * added load balancer support and user-restriction to cursor API. If a cursor is accessed on a different coordinator than where it was created, the requests will be forwarded to the correct coordinator. If a cursor is accessed by a different user than the one who created it, the request will be denied. * keep failed follower in followers list in Plan. This increases the changes of a failed follower getting back into sync if the follower comes back after a short time. In this case the follower can try to get in sync again, which normally takes less time than seeding a completely new follower. * fix assertion failure and undefined behavior in Unix domain socket connections, introduced by 3.3.12 * added configuration option `--rocksdb.sync-interval` This option specifies interval (in milliseconds) that ArangoDB will use to automatically synchronize data in RocksDB's write-ahead log (WAL) files to disk. Automatic syncs will only be performed for not-yet synchronized data, and only for operations that have been executed without the *waitForSync* attribute. Automatic synchronization is performed by a background thread. The default sync interval is 0, meaning the automatic background syncing is turned off. Background syncing in 3.3 is opt-in, whereas in ArangoDB 3.4 the default sync interval will be 100 milliseconds. Note: this option is not supported on Windows platforms. Setting the sync interval to a value greater 0 will produce a startup warning. * fixed graph creation sometimes failing with 'edge collection already used in edge def' when the edge definition contained multiple vertex collections, despite the edge definitions being identical * inception could get caught in a trap, where agent configuration version and timeout multiplier lead to incapacitated agency * fixed issue #5827: Batch request handling incompatible with .NET's default ContentType format * fixed agency's log compaction for internal issue #2249 * inspector collects additionally disk data size and storage engine statistics v3.3.12 (2018-07-12) -------------------- * issue #5854: RocksDB engine would frequently request a new DelayToken. This caused excessive write delay on the next Put() call. Alternate approach taken. * fixed graph creation under some circumstances failing with 'edge collection already used in edge def' despite the edge definitions being identical * fixed issue #5727: Edge document with user provided key is inserted as many times as the number of shards, violating the primary index * fixed internal issue #2658: AQL modification queries did not allow `_rev` checking. There is now a new option `ignoreRevs` which can be set to `false` in order to force AQL modification queries to match revision ids before doing any modifications * fixed issue #5679: Replication applier restrictions will crash synchronisation after initial sync * fixed potential issue in RETURN DISTINCT CollectBlock implementation that led to the block producing an empty result * changed communication tasks to use boost strands instead of locks, this fixes a race condition with parallel VST communication over SSL * fixed agency restart from compaction without data * fixed for agent coming back to agency with changed endpoint and total data loss * more patient agency tests to allow for ASAN tests to successfully finish v3.3.11 (2018-06-26) -------------------- * upgraded arangosync version to 0.5.3 * upgraded arangodb starter version to 0.12.0 * fixed internal issue #2559: "unexpected document key" error when custom shard keys are used and the "allowUserKeys" key generator option is set to false * fixed AQL DOCUMENT lookup function for documents for sharded collections with more than a single shard and using a custom shard key (i.e. some shard key attribute other than `_key`). The previous implementation of DOCUMENT restricted to lookup to a single shard in all cases, though this restriction was invalid. That lead to `DOCUMENT` not finding documents in cases the wrong shard was contacted. The fixed implementation in 3.3.11 will reach out to all shards to find the document, meaning it will produce the correct result, but will cause more cluster-internal traffic. This increase in traffic may be high if the number of shards is also high, because each invocation of `DOCUMENT` will have to contact all shards. There will be no performance difference for non-sharded collections or collections that are sharded by `_key` or that only have a single shard. * reimplemented replication view in web UI * fixed internal issue #2256: ui, document id not showing up when deleting a document * fixed internal issue #2163: wrong labels within foxx validation of service input parameters * fixed internal issue #2160: fixed misplaced tooltips in indices view * added new arangoinspect client tool, to help users and customers easily collect information of any ArangoDB server setup, and facilitate troubleshooting for the ArangoDB Support Team v3.3.10 (2018-06-04) -------------------- * make optimizer rule "remove-filter-covered-by-index" not stop after removing a sub-condition from a FILTER statement, but pass the optimized FILTER statement again into the optimizer rule for further optimizations. This allows optimizing away some more FILTER conditions than before. * allow accessing /_admin/status URL on followers too in active failover setup * fix cluster COLLECT optimization for attributes that were in "sorted" variant of COLLECT and that were provided by a sorted index on the collected attribute * apply fulltext index optimization rule for multiple fulltext searches in the same query this fixes https://stackoverflow.com/questions/50496274/two-fulltext-searches-on-arangodb-cluster-v8-is-involved * validate `_from` and `_to` values of edges on updates consistently * fixed issue #5400: Unexpected AQL Result * fixed issue #5429: Frequent 'updated local foxx repository' messages * fixed issue #5252: Empty result if FULLTEXT() is used together with LIMIT offset * fixed issue #5035: fixed a vulnerability issue within the web ui's index view * inception was ignoring leader's configuration v3.3.9 (2018-05-17) ------------------- * added `/_admin/repair/distributeShardsLike` that repairs collections with distributeShardsLike where the shards aren't actually distributed like in the prototype collection, as could happen due to internal issue #1770 * fixed Foxx queues bug when queues are created in a request handler with an ArangoDB authentication header * upgraded arangosync version to 0.5.1 * upgraded arangodb starter version to 0.11.3 * fix cluster upgrading issue introduced in 3.3.8 the issue made arangod crash when starting a DB server with option `--database.auto-upgrade true` * fix C++ implementation of AQL ZIP function to return each distinct attribute name only once. The previous implementation added non-unique attribute names multiple times, which led to follow-up issues. Now if an attribute name occurs multiple times in the input list of attribute names, it will only be incorporated once into the result object, with the value that corresponds to the first occurrence. This fix also changes the V8 implementation of the ZIP function, which now will always return the first value for non-unique attribute names and not the last occurring value. * self heal during a Foxx service install, upgrade or replace no longer breaks the respective operation * make /_api/index, /_api/database and /_api/user REST handlers use the scheduler's internal queue, so they do not run in an I/O handling thread * fixed issue #4919: C++ implementation of LIKE function now matches the old and correct behavior of the JavaScript implementation. * added REST API endpoint /_admin/server/availability for monitoring purposes * UI: fixed an unreasonable event bug within the modal view engine * fixed issue #3811: gharial api is now checking existence of _from and _to vertices during edge creation * fixed internal issue #2149: number of documents in the UI is not adjusted after moving them * fixed internal issue #2150: UI - loading a saved query does not update the list of bind parameters * fixed internal issue #2147 - fixed database filter in UI * fixed issue #4934: Wrong used GeoIndex depending on FILTER order * added `query` and `aql.literal` helpers to `@arangodb` module. * remove post-sort from GatherNode in cluster AQL queries that do use indexes for filtering but that do not require a sorted result This optimization can speed up gathering data from multiple shards, because it allows to remove a merge sort of the individual shards' results. * extend the already existing "reduce-extraction-to-projection" AQL optimizer rule for RocksDB to provide projections of up to 5 document attributes. The previous implementation only supported a projection for a single document attribute. The new implementation will extract up to 5 document attributes from a document while scanning a collection via an EnumerateCollectionNode. Additionally the new version of the optimizer rule can also produce projections when scanning an index via an IndexNode. The optimization is beneficial especially for huge documents because it will copy out only the projected attributes from the document instead of copying the entire document data from the storage engine. When applied, the explainer will show the projected attributes in a `projections` remark for an EnumerateCollectionNode or IndexNode. The optimization is limited to the RocksDB storage engine. * added index-only optimization for AQL queries that can satisfy the retrieval of all required document attributes directly from an index. This optimization will be triggered for the RocksDB engine if an index is used that covers all required attributes of the document used later on in the query. If applied, it will save retrieving the actual document data (which would require an extra lookup in RocksDB), but will instead build the document data solely from the index values found. It will only be applied when using up to 5 attributes from the document, and only if the rest of the document data is not used later on in the query. The optimization is currently available for the RocksDB engine for the index types primary, edge, hash, skiplist and persistent. If the optimization is applied, it will show up as "index only" in an AQL query's execution plan for an IndexNode. * added scan-only optimization for AQL queries that iterate over collections or indexes and that do not need to return the actual document values. Not fetching the document values from the storage engine will provide a considerable speedup when using the RocksDB engine, but may also help a bit in case of the MMFiles engine. The optimization will only be applied when full-scanning or index-scanning a collection without referring to any of its documents later on, and, for an IndexNode, if all filter conditions for the documents of the collection are covered by the index. If the optimization is applied, it will show up as "scan only" in an AQL query's execution plan for an EnumerateCollectionNode or an IndexNode. * extend existing "collect-in-cluster" optimizer rule to run grouping, counting and deduplication on the DB servers in several cases, so that the coordinator will only need to sum up the potentially smaller results from the individual shards. The following types of COLLECT queries are covered now: - RETURN DISTINCT expr - COLLECT WITH COUNT INTO ... - COLLECT var1 = expr1, ..., varn = exprn (WITH COUNT INTO ...), without INTO or KEEP - COLLECT var1 = expr1, ..., varn = exprn AGGREGATE ..., without INTO or KEEP, for aggregate functions COUNT/LENGTH, SUM, MIN and MAX. * honor specified COLLECT method in AQL COLLECT options for example, when the user explicitly asks for the COLLECT method to be `sorted`, the optimizer will now not produce an alternative version of the plan using the hash method. additionally, if the user explicitly asks for the COLLECT method to be `hash`, the optimizer will now change the existing plan to use the hash method if possible instead of just creating an alternative plan. `COLLECT ... OPTIONS { method: 'sorted' }` => always use sorted method `COLLECT ... OPTIONS { method: 'hash' }` => use hash if this is technically possible `COLLECT ...` (no options) => create a plan using sorted, and another plan using hash method * added bulk document lookups for MMFiles engine, which will improve the performance of document lookups from an inside an index in case the index lookup produces many documents v3.3.8 (2018-04-24) ------------------- * included version of ArangoDB Starter (`arangodb` binary) updated to v0.10.11, see [Starter changelog](https://github.com/arangodb-helper/arangodb/blob/master/CHANGELOG.md) * added arangod startup option `--dump-options` to print all configuration parameters as a JSON object * fixed: (Enterprise Edition only) If you restore a SmartGraph where the collections are still existing and are supposed to be dropped on restore we ended up in duplicate name error. This is now gone and the SmartGraph is correctly restored. * fix lookups by `_id` in smart graph edge collections * improve startup resilience in case there are datafile errors (MMFiles) also allow repairing broken VERSION files automatically on startup by specifying the option `--database.ignore-datafile-errors true` * fix issue #4582: UI query editor now supports usage of empty string as bind parameter value * fixed internal issue #2148: Number of documents found by filter is misleading in web UI * added startup option `--database.required-directory-state` using this option it is possible to require the database directory to be in a specific state on startup. the options for this value are: - non-existing: database directory must not exist - existing: database directory must exist - empty: database directory must exist but be empty - populated: database directory must exist and contain specific files already - any: any state allowed * field "$schema" in Foxx manifest.json files no longer produce warnings * added `@arangodb/locals` module to expose the Foxx service context as an alternative to using `module.context` directly. * `db._executeTransaction` now accepts collection objects as collections. * supervision can be put into maintenance mode v3.3.7 (2018-04-11) ------------------- * added hidden option `--query.registry-ttl` to control the lifetime of cluster AQL query parts * fixed internal issue #2237: AQL queries on collections with replicationFactor: "satellite" crashed arangod in single server mode * fixed restore of satellite collections: replicationFactor was set to 1 during restore * fixed dump and restore of smart graphs: a) The dump will not include the hidden shadow collections anymore, they were dumped accidentally and only contain duplicated data. b) Restore will now ignore hidden shadow collections as all data is contained in the smart-edge collection. You can manually include these collections from an old dump (3.3.5 or earlier) by using `--force`. c) Restore of a smart-graph will now create smart collections properly instead of getting into `TIMEOUT_IN_CLUSTER_OPERATION` * fixed issue in AQL query optimizer rule "restrict-to-single-shard", which may have sent documents to a wrong shard in AQL INSERT queries that specified the value for `_key` using an expression (and not a constant value) Important: if you were affected by this bug in v3.3.5 it is required that you recreate your dataset in v3.3.6 (i.e. dumping and restoring) instead of doing a simple binary upgrade * added /_admin/status HTTP API for debugging purposes * added ArangoShell helper function for packaging all information about an AQL query so it can be run and analyzed elsewhere: query = "FOR doc IN mycollection FILTER doc.value > 42 RETURN doc"; require("@arangodb/aql/explainer").debugDump("/tmp/query-debug-info", query); Entitled users can send the generated file to the ArangoDB support to facilitate reproduction and debugging. * added hidden option `--server.ask-jwt-secret`. This is an internal option for debugging and should not be exposed to end-users. * fix for internal issue #2215. supervision will now wait for agent to fully prepare before adding 10 second grace period after leadership change * fixed internal issue #2215's FailedLeader timeout bug v3.3.5 (2018-03-28) ------------------- * fixed issue #4934: Wrong used GeoIndex depending on FILTER order * make build id appear in startup log message alongside with other version info * make AQL data modification operations that are sent to all shards and that are supposed to return values (i.e. `RETURN OLD` or `RETURN NEW`) not return fake empty result rows if the document to be updated/replaced/removed was not present on the target shard * added AQL optimizer rule `restrict-to-single-shard` This rule will kick in if a collection operation (index lookup or data modification operation) will only affect a single shard, and the operation can be restricted to the single shard and is not applied for all shards. This optimization can be applied for queries that access a collection only once in the query, and that do not use traversals, shortest path queries and that do not access collection data dynamically using the `DOCUMENT`, `FULLTEXT`, `NEAR` or `WITHIN` AQL functions. Additionally, the optimizer will only pull off this optimization if can safely determine the values of all the collection's shard keys from the query, and when the shard keys are covered by a single index (this is always true if the shard key is the default `_key`) * display missing attributes of GatherNodes in AQL explain output * make AQL optimizer rule `undistribute-remove-after-enum-coll` fire in a few more cases in which it is possible * slightly improve index selection for the RocksDB engine when there are multiple competing indexes with the same attribute prefixes, but different amount of attributes covered. In this case, the more specialized index will be preferred now * fix issue #4924: removeFollower now prefers to remove the last follower(s) * added "collect-in-cluster" optimizer rule to have COLLECT WITH COUNT queries without grouping being executed on the DB servers and the coordinator only summing up the counts from the individual shards * fixed issue #4900: Nested FOR query uses index but ignores other filters * properly exit v8::Context in one place where it was missing before * added hidden option `--cluster.index-create-timeout` for controlling the default value of the index creation timeout in cluster under normal circumstances, this option does not need to be adjusted * increase default timeout for index creation in cluster to 3600s * fixed issue #4843: Query-Result has more Docs than the Collection itself * fixed the behavior of ClusterInfo when waiting for current to catch up with plan in create collection. * fixed issue #4827: COLLECT on edge _to field doesn't group distinct values as expected (MMFiles) v3.3.4 (2018-03-01) ------------------- * fix AQL `fullCount` result value in some cluster cases when it was off a bit * fix issue #4651: Simple query taking forever until a request timeout error * fix issue #4657: fixed incomplete content type header * Vastly improved the Foxx Store UI * fix issue #4677: AQL WITH with bind parameters results in "access after data-modification" for two independent UPSERTs * remove unused startup option `--ldap.permissions-attribute-name` * fix issue #4457: create /var/tmp/arangod with correct user in supervisor mode * remove long dysfunctional admin/long_echo handler * fixed Foxx API: * PUT /_api/foxx/service: Respect force flag * PATCH /_api/foxx/service: Check whether a service under given mount exists * internal issue #1726: supervision failed to remove multiple servers from health monitoring at once. * more information from inception, why agent is activated * fixed a bug where supervision tried to deal with shards of virtual collections * fix internal issue #1770: collection creation using distributeShardsLike yields errors and did not distribute shards correctly in the following cases: 1. If numberOfShards * replicationFactor % nrDBServers != 0 (shards * replication is not divisible by DBServers). 2. If there was failover / move shard case on the leading collection and creating the follower collection afterwards. * fix timeout issues in replication client expiration * added missing edge filter to neighbors-only traversals in case a filter condition was moved into the traverser and the traversal was executed in breadth-first mode and was returning each visited vertex exactly once, and there was a filter on the edges of the path and the resulting vertices and edges were not used later, the edge filter was not applied * fixed issue #4160: Run arangod with "--database.auto-upgrade" option always crash silently without error log * fix internal issue #1848: AQL optimizer was trying to resolve attribute accesses to attributes of constant object values at query compile time, but only did so far the very first attribute in each object this fixes https://stackoverflow.com/questions/48648737/beginner-bug-in-for-loops-from-objects * fix inconvenience: If we want to start server with a non-existing --javascript.app-path it will now be created (if possible) * fixed: REST API `POST _api/foxx` now returns HTTP code 201 on success, as documented. returned 200 before. * fixed: REST API `PATCH _api/foxx/dependencies` now updates the existing dependencies instead of replacing them. * fixed: Foxx upload of single javascript file. You now can upload via http-url pointing to a javascript file. * fixed issue #4395: If your foxx app includes an `APP` folder it got accidently removed by selfhealing this is not the case anymore. * fixed internal issue #1969 - command apt-get purge/remove arangodb3e was failing v3.3.3 (2018-01-16) ------------------- * fix issue #4272: VERSION file keeps disappearing * fix internal issue #81: quotation marks disappeared when switching table/json editor in the query editor ui * added option `--rocksdb.throttle` to control whether write-throttling is enabled Write-throttling is turned on by default, to reduce chances of compactions getting too far behind and blocking incoming writes. * fixed issue #4308: Crash when getter for error.name throws an error (on Windows) * UI: fixed a query editor caching and parsing issue * Fixed internal issue #1683: fixes an UI issue where a collection name gets wrongly cached within the documents overview of a collection. * Fixed an issue with the index estimates in RocksDB in the case a transaction is aborted. Former the index estimates were modified if the transaction commited or not. Now they will only be modified if the transaction commited successfully. * UI: optimized login view for very small screen sizes * Truncate in RocksDB will now do intermediate commits every 10.000 documents if truncate fails or the server crashes during this operation all deletes that have been commited so far are persisted. * make the default value of `--rocksdb.block-cache-shard-bits` use the RocksDB default value. This will mostly mean the default number block cache shard bits is lower than before, allowing each shard to store more data and cause less evictions from block cache * issue #4222: Permission error preventing AQL query import / export on webui * UI: optimized error messages for invalid query bind parameter * UI: upgraded swagger ui to version 3.9.0 * issue #3504: added option `--force-same-database` for arangorestore with this option set to true, it is possible to make any arangorestore attempt fail if the specified target database does not match the database name specified in the source dump's "dump.json" file. it can thus be used to prevent restoring data into the "wrong" database The option is set to `false` by default to ensure backwards-compatibility * make the default value of `--rocksdb.block-cache-shard-bits` use the RocksDB default value. This will mostly mean the default number block cache shard bits is lower than before, allowing each shard to store more data and cause less evictions from block cache * fixed issue #4255: AQL SORT consuming too much memory * fixed incorrect persistence of RAFT vote and term v3.3.2 (2018-01-04) ------------------- * fixed issue #4199: Internal failure: JavaScript exception in file 'arangosh.js' at 98,7: ArangoError 4: Expecting type String * fixed issue in agency supervision with a good server being left in failedServers * distinguish isReady and allInSync in clusterInventory * fixed issue #4197: AQL statement not working in 3.3.1 when upgraded from 3.2.10 * do not reuse collection ids when restoring collections from a dump, but assign new collection ids, this should prevent collection id conflicts v3.3.1 (2017-12-28) ------------------- * UI: displayed wrong wfs property for a collection when using RocksDB as storage engine * added `--ignore-missing` option to arangoimp this option allows importing lines with less fields than specified in the CSV header line * changed misleading error message from "no leader" to "not a leader" * optimize usage of AQL FULLTEXT index function to a FOR loop with index usage in some cases When the optimization is applied, this especially speeds up fulltext index queries in the cluster * UI: improved the behavior during collection creation in a cluster environment * Agency lockup fixes for very small machines. * Agency performance improvement by finer grained locking. * Use steady_clock in agency wherever possible. * Agency prevent Supervision thread crash. * Fix agency integer overflow in timeout calculation. v3.3.0 (2017-12-14) ------------------- * release version * added a missing try/catch block in the supervision thread v3.3.rc8 (2017-12-12) --------------------- * UI: fixed broken Foxx configuration keys. Some valid configuration values could not be edited via the ui. * UI: pressing the return key inside a select2 box no longer triggers the modal's success function * UI: coordinators and db servers are now in sorted order (ascending) v3.3.rc7 (2017-12-07) --------------------- * fixed issue #3741: fix terminal color output in Windows * UI: fixed issue #3822: disabled name input field for system collections * fixed issue #3640: limit in subquery * fixed issue #3745: Invalid result when using OLD object with array attribute in UPSERT statement * UI: edge collections were wrongly added to from and to vertices select box during graph creation * UI: added not found views for documents and collections * UI: using default user database api during database creation now * UI: the graph viewer backend now picks one random start vertex of the first 1000 documents instead of calling any(). The implementation of "any" is known to scale bad on huge collections with RocksDB. * UI: fixed disappearing of the navigation label in some case special case * UI: the graph viewer now displays updated label values correctly. Additionally the included node/edge editor now closes automatically after a successful node/edge update. * fixed issue #3917: traversals with high maximal depth take extremely long in planning phase. v3.3.rc4 (2017-11-28) --------------------- * minor bug-fixes v3.3.rc3 (2017-11-24) --------------------- * bug-fixes v3.3.rc2 (2017-11-22) --------------------- * UI: document/edge editor now remembering their modes (e.g. code or tree) * UI: optimized error messages for invalid graph definitions. Also fixed a graph renderer cleanup error. * UI: added a delay within the graph viewer while changing the colors of the graph. Necessary due different browser behavior. * added options `--encryption.keyfile` and `--encryption.key-generator` to arangodump and arangorestore * UI: the graph viewer now displays updated label values correctly. Additionally the included node/edge editor now closes automatically after a successful node/edge update. * removed `--recycle-ids` option for arangorestore using that option could have led to problems on the restore, with potential id conflicts between the originating server (the source dump server) and the target server (the restore server) v3.3.rc1 (2017-11-17) --------------------- * add readonly mode REST API * allow compilation of ArangoDB source code with g++ 7 * upgrade minimum required g++ compiler version to g++ 5.4 That means ArangoDB source code will not compile with g++ 4.x or g++ < 5.4 anymore. * AQL: during a traversal if a vertex is not found. It will not print an ERROR to the log and continue with a NULL value, but will register a warning at the query and continue with a NULL value. The situation is not desired as an ERROR as ArangoDB can store edges pointing to non-existing vertex which is perfectly valid, but it may be a n issue on the data model, so users can directly see it on the query now and do not "by accident" have to check the LOG output. * introduce `enforceReplicationFactor` attribute for creating collections: this optional parameter controls if the coordinator should bail out during collection creation if there are not enough DBServers available for the desired `replicationFactor`. * fixed issue #3516: Show execution time in arangosh this change adds more dynamic prompt components for arangosh The following components are now available for dynamic prompts, settable via the `--console.prompt` option in arangosh: - '%t': current time as timestamp - '%a': elapsed time since ArangoShell start in seconds - '%p': duration of last command in seconds - '%d': name of current database - '%e': current endpoint - '%E': current endpoint without protocol - '%u': current user The time a command takes can be displayed easily by starting arangosh with `--console.prompt "%p> "`. * make the ArangoShell refill its collection cache when a yet-unknown collection is first accessed. This fixes the following problem: arangosh1> db._collections(); // shell1 lists all collections arangosh2> db._create("test"); // shell2 now creates a new collection 'test' arangosh1> db.test.insert({}); // shell1 is not aware of the collection created // in shell2, so the insert will fail * make AQL `DISTINCT` not change the order of the results it is applied on * incremental transfer of initial collection data now can handle partial responses for a chunk, allowing the leader/master to send smaller chunks (in terms of HTTP response size) and limit memory usage this optimization is only active if client applications send the "offset" parameter in their requests to PUT `/_api/replication/keys/?type=docs` * initial creation of shards for cluster collections is now faster with `replicationFactor` values bigger than 1. this is achieved by an optimization for the case when the collection on the leader is still empty * potential fix for issue #3517: several "filesystem full" errors in logs while there's a lot of disk space * added C++ implementations for AQL function `SUBSTRING()`, `LEFT()`, `RIGHT()` and `TRIM()` * show C++ function name of call site in ArangoDB log output this requires option `--log.line-number` to be set to *true* * UI: added word wrapping to query editor * UI: fixed wrong user attribute name validation, issue #3228 * make AQL return a proper error message in case of a unique key constraint violation. previously it only returned the generic "unique constraint violated" error message but omitted the details about which index caused the problem. This addresses https://stackoverflow.com/questions/46427126/arangodb-3-2-unique-constraint-violation-id-or-key * added option `--server.local-authentication` * UI: added user roles * added config option `--log.color` to toggle colorful logging to terminal * added config option `--log.thread-name` to additionally log thread names * usernames must not start with `:role:`, added new options: --server.authentication-timeout --ldap.roles-attribute-name --ldap.roles-transformation --ldap.roles-search --ldap.superuser-role --ldap.roles-include --ldap.roles-exclude * performance improvements for full collection scans and a few other operations in MMFiles engine * added `--rocksdb.encryption-key-generator` for enterprise * removed `--compat28` parameter from arangodump and replication API older ArangoDB versions will no longer be supported by these tools. * increase the recommended value for `/proc/sys/vm/max_map_count` to a value eight times as high as the previous recommended value. Increasing the values helps to prevent an ArangoDB server from running out of memory mappings. The raised minimum recommended value may lead to ArangoDB showing some startup warnings as follows: WARNING {memory} maximum number of memory mappings per process is 65530, which seems too low. it is recommended to set it to at least 512000 WARNING {memory} execute 'sudo sysctl -w "vm.max_map_count=512000"' * Foxx now warns about malformed configuration/dependency names and aliases in the manifest. v3.2.17 (XXXX-XX-XX) -------------------- * added missing virtual destructor for MMFiles transaction data context object * make synchronous replication detect more error cases when followers cannot apply the changes from the leader * fixed undefined behavior in cluster plan-loading procedure that may have unintentionally modified a shared structure * cluster nodes should retry registering in agency until successful * fixed issue #5354: updated the ui json editor, improved usability * fixed issue #5648: fixed error message when saving unsupported document types * fixed issue #5943: misplaced database ui icon and wrong cursor type were used v3.2.16 (2018-07-12) -------------------- * upgraded arangodb starter version to 0.12.0 * make edge cache initialization and invalidation more portable by avoiding memset on non-POD types * fixed internal issue #2256: ui, document id not showing up when deleting a document * fixed issue #5400: Unexpected AQL Result * Fixed issue #5035: fixed a vulnerability issue within the web ui's index view * issue one HTTP call less per cluster AQL query * self heal during a Foxx service install, upgrade or replace no longer breaks the respective operation * inception was ignoring leader's configuration * inception could get caught in a trap, where agent configuration version and timeout multiplier lead to incapacitated agency * more patient agency tests to allow for ASAN tests to successfully finish * fixed for agent coming back to agency with changed endpoint and total data loss * fixed agency restart from compaction without data v3.2.15 (2018-05-13) -------------------- * upgraded arangodb starter version to 0.11.2 * make /_api/index and /_api/database REST handlers use the scheduler's internal queue, so they do not run in an I/O handling thread * fixed issue #3811: gharial api is now checking existence of _from and _to vertices during edge creation v3.2.14 (2018-04-20) -------------------- * field "$schema" in Foxx manifest.json files no longer produce warnings * added `@arangodb/locals` module to expose the Foxx service context as an alternative to using `module.context` directly. * the internal implementation of REST API `/_api/simple/by-example` now uses C++ instead of JavaScript * supervision can be switched to maintenance mode f.e. for rolling upgrades v3.2.13 (2018-04-13) -------------------- * improve startup resilience in case there are datafile errors (MMFiles) also allow repairing broken VERSION files automatically on startup by specifying the option `--database.ignore-datafile-errors true` * fix issue #4582: UI query editor now supports usage of empty string as bind parameter value * fix issue #4924: removeFollower now prefers to remove the last follower(s) * fixed issue #4934: Wrong used GeoIndex depending on FILTER order * fixed the behavior of clusterinfo when waiting for current to catch up with plan in create collection. * fix for internal issue #2215. supervision will now wait for agent to fully prepare before adding 10 second grace period after leadership change * fixed internal issue #2215 FailedLeader timeout bug v3.2.12 (2018-02-27) -------------------- * remove long dysfunctional admin/long_echo handler * fixed Foxx API: * PUT /_api/foxx/service: Respect force flag * PATCH /_api/foxx/service: Check whether a service under given mount exists * fix issue #4457: create /var/tmp/arangod with correct user in supervisor mode * fix internal issue #1848 AQL optimizer was trying to resolve attribute accesses to attributes of constant object values at query compile time, but only did so far the very first attribute in each object this fixes https://stackoverflow.com/questions/48648737/beginner-bug-in-for-loops-from-objects * fix inconvenience: If we want to start server with a non-existing --javascript.app-path it will now be created (if possible) * fixed: REST API `POST _api/foxx` now returns HTTP code 201 on success, as documented. returned 200 before. * fixed: REST API `PATCH _api/foxx/dependencies` now updates the existing dependencies instead of replacing them. * fixed: Foxx upload of single javascript file. You now can upload via http-url pointing to a javascript file. * fixed issue #4395: If your foxx app includes an `APP` folder it got accidently removed by selfhealing this is not the case anymore. * fix internal issue 1770: collection creation using distributeShardsLike yields errors and did not distribute shards correctly in the following cases: 1. If numberOfShards * replicationFactor % nrDBServers != 0 (shards * replication is not divisible by DBServers). 2. If there was failover / move shard case on the leading collection and creating the follower collection afterwards. * fix timeout issues in replication client expiration + fix some inconsistencies in replication for RocksDB engine that could have led to some operations not being shipped from master to slave servers * fix issue #4272: VERSION file keeps disappearing * fix internal issue #81: quotation marks disappeared when switching table/json editor in the query editor ui * make the default value of `--rocksdb.block-cache-shard-bits` use the RocksDB default value. This will mostly mean the default number block cache shard bits is lower than before, allowing each shard to store more data and cause less evictions from block cache * fix issue #4393: broken handling of unix domain sockets in JS_Download * fix internal bug #1726: supervision failed to remove multiple removed servers from health UI * fixed internal issue #1969 - command apt-get purge/remove arangodb3e was failing * fixed a bug where supervision tried to deal with shards of virtual collections v3.2.11 (2018-01-17) -------------------- * Fixed an issue with the index estimates in RocksDB in the case a transaction is aborted. Former the index estimates were modified if the transaction commited or not. Now they will only be modified if the transaction commited successfully. * Truncate in RocksDB will now do intermediate commits every 10.000 documents if truncate fails or the server crashes during this operation all deletes that have been commited so far are persisted. * fixed issue #4308: Crash when getter for error.name throws an error (on Windows) * UI: fixed a query editor caching and parsing issue for arrays and objects * Fixed internal issue #1684: Web UI: saving arrays/objects as bind parameters faulty * Fixed internal issue #1683: fixes an UI issue where a collection name gets wrongly cached within the documents overview of a collection. * issue #4222: Permission error preventing AQL query import / export on webui * UI: optimized login view for very small screen sizes * UI: Shard distribution view now has an accordion view instead of displaying all shards of all collections at once. * UI: optimized error messages for invalid query bind parameter * fixed missing transaction events in RocksDB asynchronous replication * fixed issue #4255: AQL SORT consuming too much memory * fixed issue #4199: Internal failure: JavaScript exception in file 'arangosh.js' at 98,7: ArangoError 4: Expecting type String * fixed issue #3818: Foxx configuration keys cannot contain spaces (will not save) * UI: displayed wrong "waitForSync" property for a collection when using RocksDB as storage engine * prevent binding to the same combination of IP and port on Windows * fixed incorrect persistence of RAFT vote and term v3.2.10 (2017-12-22) -------------------- * replication: more robust initial sync * fixed a bug in the RocksDB engine that would prevent recalculated collection counts to be actually stored * fixed issue #4095: Inconsistent query execution plan * fixed issue #4056: Executing empty query causes crash * fixed issue #4045: Out of memory in `arangorestore` when no access rights to dump files * fixed issue #3031: New Graph: Edge definitions with edges in fromCollections and toCollections * fixed issue #2668: UI: when following wrong link from edge to vertex in nonexisting collection misleading error is printed * UI: improved the behavior during collection creation in a cluster environment * UI: the graph viewer backend now picks one random start vertex of the first 1000 documents instead of calling any(). The implementation of any is known to scale bad on huge collections with rocksdb. * fixed snapshots becoming potentially invalid after intermediate commits in the RocksDB engine * backport agency inquire API changes * fixed issue #3822: Field validation error in ArangoDB UI - Minor * UI: fixed disappearing of the navigation label in some cases * UI: fixed broken foxx configuration keys. Some valid configuration values could not be edited via the ui. * fixed issue #3640: limit in subquery * UI: edge collections were wrongly added to from and to vertices select box during graph creation * fixed issue #3741: fix terminal color output in Windows * fixed issue #3917: traversals with high maximal depth take extremely long in planning phase. * fix equality comparison for MMFiles documents in AQL functions UNIQUE and UNION_DISTINCT v3.2.9 (2017-12-04) ------------------- * under certain conditions, replication could stop. Now fixed by adding an equality check for requireFromPresent tick value * fixed locking for replication context info in RocksDB engine this fixes undefined behavior when parallel requests are made to the same replication context * UI: added not found views for documents and collections * fixed issue #3858: Foxx queues stuck in 'progress' status * allow compilation of ArangoDB source code with g++ 7 * fixed issue #3224: Issue in the Foxx microservices examples * fixed a deadlock in user privilege/permission change routine * fixed a deadlock on server shutdown * fixed some collection locking issues in MMFiles engine * properly report commit errors in AQL write queries to the caller for the RocksDB engine * UI: optimized error messages for invalid graph definitions. Also fixed a graph renderer cleanrenderer cleanup error. * UI: document/edge editor now remembering their modes (e.g. code or tree) * UI: added a delay within the graph viewer while changing the colors of the graph. Necessary due different browser behavior. * fix removal of failed cluster nodes via web interface * back port of ClusterComm::wait fix in devel among other things this fixes too eager dropping of other followers in case one of the followers does not respond in time * transact interface in agency should not be inquired as of now * inquiry tests and blocking of inquiry on AgencyGeneralTransaction v3.2.8 (2017-11-18) ------------------- * fixed a race condition occurring when upgrading via linux package manager * fixed authentication issue during replication v3.2.7 (2017-11-13) ------------------- * Cluster customers, which have upgraded from 3.1 to 3.2 need to upgrade to 3.2.7. The cluster supervision is otherwise not operational. * Fixed issue #3597: AQL with path filters returns unexpected results In some cases breadth first search in combination with vertex filters yields wrong result, the filter was not applied correctly. * fixed some undefined behavior in some internal value caches for AQL GatherNodes and SortNodes, which could have led to sorted results being effectively not correctly sorted. * make the replication applier for the RocksDB engine start automatically after a restart of the server if the applier was configured with its `autoStart` property set to `true`. previously the replication appliers were only automatically restarted at server start for the MMFiles engine. * fixed arangodump batch size adaptivity in cluster mode and upped default batch size for arangodump these changes speed up arangodump in cluster context * smart graphs now return a proper inventory in response to replication inventory requests * fixed issue #3618: Inconsistent behavior of OR statement with object bind parameters * only users with read/write rights on the "_system" database can now execute "_admin/shutdown" as well as modify properties of the write-ahead log (WAL) * increase default maximum number of V8 contexts to at least 16 if not explicitly configured otherwise. the procedure for determining the actual maximum value of V8 contexts is unchanged apart from the value `16` and works as follows: - if explicitly set, the value of the configuration option `--javascript.v8-contexts` is used as the maximum number of V8 contexts - when the option is not set, the maximum number of V8 contexts is determined by the configuration option `--server.threads` if that option is set. if `--server.threads` is not set, then the maximum number of V8 contexts is the server's reported hardware concurrency (number of processors visible to the arangod process). if that would result in a maximum value of less than 16 in any of these two cases, then the maximum value will be increased to 16. * fixed issue #3447: ArangoError 1202: AQL: NotFound: (while executing) when updating collection * potential fix for issue #3581: Unexpected "rocksdb unique constraint violated" with unique hash index * fixed geo index optimizer rule for geo indexes with a single (array of coordinates) attribute. * improved the speed of the shards overview in cluster (API endpoint /_api/cluster/shardDistribution API) It is now guaranteed to return after ~2 seconds even if the entire cluster is unresponsive. * fix agency precondition check for complex objects this fixes issues with several CAS operations in the agency * several fixes for agency restart and shutdown * the cluster-internal representation of planned collection objects is now more lightweight than before, using less memory and not allocating any cache for indexes etc. * fixed issue #3403: How to kill long running AQL queries with the browser console's AQL (display issue) * fixed issue #3549: server reading ENGINE config file fails on common standard newline character * UI: fixed error notifications for collection modifications * several improvements for the truncate operation on collections: * the timeout for the truncate operation was increased in cluster mode in order to prevent too frequent "could not truncate collection" errors * after a truncate operation, collections in MMFiles still used disk space. to reclaim disk space used by truncated collection, the truncate actions in the web interface and from the ArangoShell now issue an extra WAL flush command (in cluster mode, this command is also propagated to all servers). the WAL flush allows all servers to write out any pending operations into the datafiles of the truncated collection. afterwards, a final journal rotate command is sent, which enables the compaction to entirely remove all datafiles and journals for the truncated collection, so that all disk space can be reclaimed * for MMFiles a special method will be called after a truncate operation so that all indexes of the collection can free most of their memory. previously some indexes (hash and skiplist indexes) partially kept already allocated memory in order to avoid future memory allocations * after a truncate operation in the RocksDB engine, an additional compaction will be triggered for the truncated collection. this compaction removes all deletions from the key space so that follow-up scans over the collection's key range do not have to filter out lots of already-removed values These changes make truncate operations potentially more time-consuming than before, but allow for memory/disk space savings afterwards. * enable JEMalloc background threads for purging and returning unused memory back to the operating system (Linux only) JEMalloc will create its background threads on demand. The number of background threads is capped by the number of CPUs or active arenas. The background threads run periodically and purge unused memory pages, allowing memory to be returned to the operating system. This change will make the arangod process create several additional threads. It is accompanied by an increased `TasksMax` value in the systemd service configuration file for the arangodb3 service. * upgraded bundled V8 engine to bugfix version v5.7.492.77 this upgrade fixes a memory leak in upstream V8 described in https://bugs.chromium.org/p/v8/issues/detail?id=5945 that will result in memory chunks only getting uncommitted but not unmapped v3.2.6 (2017-10-26) ------------------- * UI: fixed event cleanup in cluster shards view * UI: reduced cluster dashboard api calls * fixed a permission problem that prevented collection contents to be displayed in the web interface * removed posix_fadvise call from RocksDB's PosixSequentialFile::Read(). This is consistent with Facebook PR 2573 (#3505) this fix should improve the performance of the replication with the RocksDB storage engine * allow changing of collection replication factor for existing collections * UI: replicationFactor of a collection is now changeable in a cluster environment * several fixes for the cluster agency * fixed undefined behavior in the RocksDB-based geo index * fixed Foxxmaster failover * purging or removing the Debian/Ubuntu arangodb3 packages now properly stops the arangod instance before actually purging or removing v3.2.5 (2017-10-16) ------------------- * general-graph module and _api/gharial now accept cluster options for collection creation. It is now possible to set replicationFactor and numberOfShards for all collections created via this graph object. So adding a new collection will not result in a singleShard and no replication anymore. * fixed issue #3408: Hard crash in query for pagination * minimum number of V8 contexts in console mode must be 2, not 1. this is required to ensure the console gets one dedicated V8 context and all other operations have at least one extra context. This requirement was not enforced anymore. * fixed issue #3395: AQL: cannot instantiate CollectBlock with undetermined aggregation method * UI: fixed wrong user attribute name validation, issue #3228 * fix potential overflow in CRC marker check when a corrupted CRC marker is found at the very beginning of an MMFiles datafile * UI: fixed unresponsive events in cluster shards view * Add statistics about the V8 context counts and number of available/active/busy threads we expose through the server statistics interface. v3.2.4 (2017-09-26) ------------------- * UI: no default index selected during index creation * UI: added replicationFactor option during SmartGraph creation * make the MMFiles compactor perform less writes during normal compaction operation This partially fixes issue #3144 * make the MMFiles compactor configurable The following options have been added: * `--compaction.db-sleep-time`: sleep interval between two compaction runs (in s) * `--compaction.min-interval"`: minimum sleep time between two compaction runs (in s) * `--compaction.min-small-data-file-size`: minimal filesize threshold original datafiles have to be below for a compaction * `--compaction.dead-documents-threshold`: minimum unused count of documents in a datafile * `--compaction.dead-size-threshold`: how many bytes of the source data file are allowed to be unused at most * `--compaction.dead-size-percent-threshold`: how many percent of the source datafile should be unused at least * `--compaction.max-files`: Maximum number of files to merge to one file * `--compaction.max-result-file-size`: how large may the compaction result file become (in bytes) * `--compaction.max-file-size-factor`: how large the resulting file may be in comparison to the collection's `--database.maximal-journal-size' setting` * fix downwards-incompatibility in /_api/explain REST handler * fix Windows implementation for fs.getTempPath() to also create a sub-directory as we do on linux * fixed a multi-threading issue in cluster-internal communication * performance improvements for traversals and edge lookups * removed internal memory zone handling code. the memory zones were a leftover from the early ArangoDB days and did not provide any value in the current implementation. * (Enterprise Edition only) added `skipInaccessibleCollections` option for AQL queries: if set, AQL queries (especially graph traversals) will treat collections to which a user has no access rights to as if these collections were empty. * adjusted scheduler thread handling to start and stop less threads in normal operations * leader-follower replication catchup code has been rewritten in C++ * early stage AQL optimization now also uses the C++ implementations of AQL functions if present. Previously it always referred to the JavaScript implementations and ignored the C++ implementations. This change gives more flexibility to the AQL optimizer. * ArangoDB tty log output is now colored for log messages with levels FATAL, ERR and WARN. * changed the return values of AQL functions `REGEX_TEST` and `REGEX_REPLACE` to `null` when the input regex is invalid. Previous versions of ArangoDB partly returned `false` for invalid regexes and partly `null`. * added `--log.role` option for arangod When set to `true`, this option will make the ArangoDB logger print a single character with the server's role into each logged message. The roles are: - U: undefined/unclear (used at startup) - S: single server - C: coordinator - P: primary - A: agent The default value for this option is `false`, so no roles will be logged. v3.2.3 (2017-09-07) ------------------- * fixed issue #3106: orphan collections could not be registered in general-graph module * fixed wrong selection of the database inside the internal cluster js api * added startup option `--server.check-max-memory-mappings` to make arangod check the number of memory mappings currently used by the process and compare it with the maximum number of allowed mappings as determined by /proc/sys/vm/max_map_count The default value is `true`, so the checks will be performed. When the current number of mappings exceeds 90% of the maximum number of mappings, the creation of further V8 contexts will be deferred. Note that this option is effective on Linux systems only. * arangoimp now has a `--remove-attribute` option * added V8 context lifetime control options `--javascript.v8-contexts-max-invocations` and `--javascript.v8-contexts-max-age` These options allow specifying after how many invocations a used V8 context is disposed, or after what time a V8 context is disposed automatically after its creation. If either of the two thresholds is reached, an idl V8 context will be disposed. The default value of `--javascript.v8-contexts-max-invocations` is 0, meaning that the maximum number of invocations per context is unlimited. The default value for `--javascript.v8-contexts-max-age` is 60 seconds. * fixed wrong UI cluster health information * fixed issue #3070: Add index in _jobs collection * fixed issue #3125: HTTP Foxx API JSON parsing * fixed issue #3120: Foxx queue: job isn't running when server.authentication = true * fixed supervision failure detection and handling, which happened with simultaneous agency leadership change v3.2.2 (2017-08-23) ------------------- * make "Rebalance shards" button work in selected database only, and not make it rebalance the shards of all databases * fixed issue #2847: adjust the response of the DELETE `/_api/users/database/*` calls * fixed issue #3075: Error when upgrading arangoDB on linux ubuntu 16.04 * fixed a buffer overrun in linenoise console input library for long input strings * increase size of the linenoise input buffer to 8 KB * abort compilation if the detected GCC or CLANG isn't in the range of compilers we support * fixed spurious cluster hangups by always sending AQL-query related requests to the correct servers, even after failover or when a follower drops The problem with the previous shard-based approach was that responsibilities for shards may change from one server to another at runtime, after the query was already instantiated. The coordinator and other parts of the query then sent further requests for the query to the servers now responsible for the shards. However, an AQL query must send all further requests to the same servers on which the query was originally instantiated, even in case of failover. Otherwise this would potentially send requests to servers that do not know about the query, and would also send query shutdown requests to the wrong servers, leading to abandoned queries piling up and using resources until they automatically time out. * fixed issue with RocksDB engine acquiring the collection count values too early, leading to the collection count values potentially being slightly off even in exclusive transactions (for which the exclusive access should provide an always-correct count value) * fixed some issues in leader-follower catch-up code, specifically for the RocksDB engine * make V8 log fatal errors to syslog before it terminates the process. This change is effective on Linux only. * fixed issue with MMFiles engine creating superfluous collection journals on shutdown * fixed issue #3067: Upgrade from 3.2 to 3.2.1 reset autoincrement keys * fixed issue #3044: ArangoDB server shutdown unexpectedly * fixed issue #3039: Incorrect filter interpretation * fixed issue #3037: Foxx, internal server error when I try to add a new service * improved MMFiles fulltext index document removal performance and fulltext index query performance for bigger result sets * ui: fixed a display bug within the slow and running queries view * ui: fixed a bug when success event triggers twice in a modal * ui: fixed the appearance of the documents filter * ui: graph vertex collections not restricted to 10 anymore * fixed issue #2835: UI detection of JWT token in case of server restart or upgrade * upgrade jemalloc version to 5.0.1 This fixes problems with the memory allocator returning "out of memory" when calling munmap to free memory in order to return it to the OS. It seems that calling munmap on Linux can increase the number of mappings, at least when a region is partially unmapped. This can lead to the process exceeding its maximum number of mappings, and munmap and future calls to mmap returning errors. jemalloc version 5.0.1 does not have the `--enable-munmap` configure option anymore, so the problem is avoided. To return memory to the OS eventually, jemalloc 5's background purge threads are used on Linux. * fixed issue #2978: log something more obvious when you log a Buffer * fixed issue #2982: AQL parse error? * fixed issue #3125: HTTP Foxx API Json parsing v3.2.1 (2017-08-09) ------------------- * added C++ implementations for AQL functions `LEFT()`, `RIGHT()` and `TRIM()` * fixed docs for issue #2968: Collection _key autoincrement value increases on error * fixed issue #3011: Optimizer rule reduce-extraction-to-projection breaks queries * Now allowing to restore users in a sharded environment as well It is still not possible to restore collections that are sharded differently than by _key. * fixed an issue with restoring of system collections and user rights. It was not possible to restore users into an authenticated server. * fixed issue #2977: Documentation for db._createDatabase is wrong * ui: added bind parameters to slow query history view * fixed issue #1751: Slow Query API should provide bind parameters, webui should display them * ui: fixed a bug when moving multiple documents was not possible * fixed docs for issue #2968: Collection _key autoincrement value increases on error * AQL CHAR_LENGTH(null) returns now 0. Since AQL TO_STRING(null) is '' (string of length 0) * ui: now supports single js file upload for Foxx services in addition to zip files * fixed a multi-threading issue in the agency when callElection was called while the Supervision was calling updateSnapshot * added startup option `--query.tracking-with-bindvars` This option controls whether the list of currently running queries and the list of slow queries should contain the bind variables used in the queries or not. The option can be changed at runtime using the commands // enables tracking of bind variables // set to false to turn tracking of bind variables off var value = true; require("@arangodb/aql/queries").properties({ trackBindVars: value }); * index selectivity estimates are now available in the cluster as well * fixed issue #2943: loadIndexesIntoMemory not returning the same structure as the rest of the collection APIs * fixed issue #2949: ArangoError 1208: illegal name * fixed issue #2874: Collection properties do not return `isVolatile` attribute * potential fix for issue #2939: Segmentation fault when starting coordinator node * fixed issue #2810: out of memory error when running UPDATE/REPLACE on medium-size collection * fix potential deadlock errors in collector thread * disallow the usage of volatile collections in the RocksDB engine by throwing an error when a collection is created with attribute `isVolatile` set to `true`. Volatile collections are unsupported by the RocksDB engine, so creating them should not succeed and silently create a non-volatile collection * prevent V8 from issuing SIGILL instructions when it runs out of memory Now arangod will attempt to log a FATAL error into its logfile in case V8 runs out of memory. In case V8 runs out of memory, it will still terminate the entire process. But at least there should be something in the ArangoDB logs indicating what the problem was. Apart from that, the arangod process should now be exited with SIGABRT rather than SIGILL as it shouldn't return into the V8 code that aborted the process with `__builtin_trap`. this potentially fixes issue #2920: DBServer crashing automatically post upgrade to 3.2 * Foxx queues and tasks now ensure that the scripts in them run with the same permissions as the Foxx code who started the task / queue * fixed issue #2928: Offset problems * fixed issue #2876: wrong skiplist index usage in edge collection * fixed issue #2868: cname missing from logger-follow results in rocksdb * fixed issue #2889: Traversal query using incorrect collection id * fixed issue #2884: AQL traversal uniqueness constraints "propagating" to other traversals? Weird results * arangoexport: added `--query` option for passing an AQL query to export the result * fixed issue #2879: No result when querying for the last record of a query * ui: allows now to edit default access level for collections in database _system for all users except the root user. * The _users collection is no longer accessible outside the arangod process, _queues is always read-only * added new option "--rocksdb.max-background-jobs" * removed options "--rocksdb.max-background-compactions", "--rocksdb.base-background-compactions" and "--rocksdb.max-background-flushes" * option "--rocksdb.compaction-read-ahead-size" now defaults to 2MB * change Windows build so that RocksDB doesn't enforce AVX optimizations by default This fixes startup crashes on servers that do not have AVX CPU extensions * speed up RocksDB secondary index creation and dropping * removed RocksDB note in Geo index docs v3.2.0 (2017-07-20) ------------------- * fixed UI issues * fixed multi-threading issues in Pregel * fixed Foxx resilience * added command-line option `--javascript.allow-admin-execute` This option can be used to control whether user-defined JavaScript code is allowed to be executed on server by sending via HTTP to the API endpoint `/_admin/execute` with an authenticated user account. The default value is `false`, which disables the execution of user-defined code. This is also the recommended setting for production. In test environments, it may be convenient to turn the option on in order to send arbitrary setup or teardown commands for execution on the server. v3.2.beta6 (2017-07-18) ----------------------- * various bugfixes v3.2.beta5 (2017-07-16) ----------------------- * numerous bugfixes v3.2.beta4 (2017-07-04) ----------------------- * ui: fixed document view _from and _to linking issue for special characters * added function `db._parse(query)` for parsing an AQL query and returning information about it * fixed one medium priority and two low priority security user interface issues found by owasp zap. * ui: added index deduplicate options * ui: fixed renaming of collections for the rocksdb storage engine * documentation and js fixes for secondaries * RocksDB storage format was changed, users of the previous beta/alpha versions must delete the database directory and re-import their data * enabled permissions on database and collection level * added and changed some user related REST APIs * added `PUT /_api/user/{user}/database/{database}/{collection}` to change collection permission * added `GET /_api/user/{user}/database/{database}/{collection}` * added optional `full` parameter to the `GET /_api/user/{user}/database/` REST call * added user functions in the arangoshell `@arangodb/users` module * added `grantCollection` and `revokeCollection` functions * added `permission(user, database, collection)` to retrieve collection specific rights * added "deduplicate" attribute for array indexes, which controls whether inserting duplicate index values from the same document into a unique array index will lead to an error or not: // with deduplicate = true, which is the default value: db._create("test"); db.test.ensureIndex({ type: "hash", fields: ["tags[*]"], deduplicate: true }); db.test.insert({ tags: ["a", "b"] }); db.test.insert({ tags: ["c", "d", "c"] }); // will work, because deduplicate = true db.test.insert({ tags: ["a"] }); // will fail // with deduplicate = false db._create("test"); db.test.ensureIndex({ type: "hash", fields: ["tags[*]"], deduplicate: false }); db.test.insert({ tags: ["a", "b"] }); db.test.insert({ tags: ["c", "d", "c"] }); // will not work, because deduplicate = false db.test.insert({ tags: ["a"] }); // will fail The "deduplicate" attribute is now also accepted by the index creation HTTP API endpoint POST /_api/index and is returned by GET /_api/index. * added optimizer rule "remove-filters-covered-by-traversal" * Debian/Ubuntu installer: make messages about future package upgrades more clear * fix a hangup in VST The problem happened when the two first chunks of a VST message arrived together on a connection that was newly switched to VST. * fix deletion of outdated WAL files in RocksDB engine * make use of selectivity estimates in hash, skiplist and persistent indexes in RocksDB engine * changed VM overcommit recommendation for user-friendliness * fix a shutdown bug in the cluster: a destroyed query could still be active * do not terminate the entire server process if a temp file cannot be created (Windows only) * fix log output in the front-end, it stopped in case of too many messages v3.2.beta3 (2017-06-27) ----------------------- * numerous bugfixes v3.2.beta2 (2017-06-20) ----------------------- * potentially fixed issue #2559: Duplicate _key generated on insertion * fix invalid results (too many) when a skipping LIMIT was used for a traversal. `LIMIT x` or `LIMIT 0, x` were not affected, but `LIMIT s, x` may have returned too many results * fix races in SSL communication code * fix invalid locking in JWT authentication cache, which could have crashed the server * fix invalid first group results for sorted AQL COLLECT when LIMIT was used * fix potential race, which could make arangod hang on startup * removed `exception` field from transaction error result; users should throw explicit `Error` instances to return custom exceptions (addresses issue #2561) * fixed issue #2613: Reduce log level when Foxx manager tries to self heal missing database * add a read only mode for users and collection level authorization * removed `exception` field from transaction error result; users should throw explicit `Error` instances to return custom exceptions (addresses issue #2561) * fixed issue #2677: Foxx disabling development mode creates non-deterministic service bundle * fixed issue #2684: Legacy service UI not working v3.2.beta1 (2017-06-12) ----------------------- * provide more context for index errors (addresses issue #342) * arangod now validates several OS/environment settings on startup and warns if the settings are non-ideal. Most of the checks are executed on Linux systems only. * fixed issue #2515: The replace-or-with-in optimization rule might prevent use of indexes * added `REGEX_REPLACE` AQL function * the RocksDB storage format was changed, users of the previous alpha versions must delete the database directory and re-import their data * added server startup option `--query.fail-on-warning` setting this option to `true` will abort any AQL query with an exception if it causes a warning at runtime. The value can be overridden per query by setting the `failOnWarning` attribute in a query's options. * added --rocksdb.num-uncompressed-levels to adjust number of non-compressed levels * added checks for memory management and warn (i. e. if hugepages are enabled) * set default SSL cipher suite string to "HIGH:!EXPORT:!aNULL@STRENGTH" * fixed issue #2469: Authentication = true does not protect foxx-routes * fixed issue #2459: compile success but can not run with rocksdb * `--server.maximal-queue-size` is now an absolute maximum. If the queue is full, then 503 is returned. Setting it to 0 means "no limit". * (Enterprise Edition only) added authentication against an LDAP server * fixed issue #2083: Foxx services aren't distributed to all coordinators * fixed issue #2384: new coordinators don't pick up existing Foxx services * fixed issue #2408: Foxx service validation causes unintended side-effects * extended HTTP API with routes for managing Foxx services * added distinction between hasUser and authorized within Foxx (cluster internal requests are authorized requests but don't have a user) * arangoimp now has a `--threads` option to enable parallel imports of data * PR #2514: Foxx services that can't be fixed by self-healing now serve a 503 error * added `time` function to `@arangodb` module v3.2.alpha4 (2017-04-25) ------------------------ * fixed issue #2450: Bad optimization plan on simple query * fixed issue #2448: ArangoDB Web UI takes no action when Delete button is clicked * fixed issue #2442: Frontend shows already deleted databases during login * added 'x-content-type-options: nosniff' to avoid MSIE bug * set default value for `--ssl.protocol` from TLSv1 to TLSv1.2. * AQL breaking change in cluster: The SHORTEST_PATH statement using edge-collection names instead of a graph name now requires to explicitly name the vertex-collection names within the AQL query in the cluster. It can be done by adding `WITH ` at the beginning of the query. Example: ``` FOR v,e IN OUTBOUND SHORTEST_PATH @start TO @target edges [...] ``` Now has to be: ``` WITH vertices FOR v,e IN OUTBOUND SHORTEST_PATH @start TO @target edges [...] ``` This change is due to avoid dead-lock situations in clustered case. An error stating the above is included. * add implicit use of geo indexes when using SORT/FILTER in AQL, without the need to use the special-purpose geo AQL functions `NEAR` or `WITHIN`. the special purpose `NEAR` AQL function can now be substituted with the following AQL (provided there is a geo index present on the `doc.latitude` and `doc.longitude` attributes): FOR doc in geoSort SORT DISTANCE(doc.latitude, doc.longitude, 0, 0) LIMIT 5 RETURN doc `WITHIN` can be substituted with the following AQL: FOR doc in geoFilter FILTER DISTANCE(doc.latitude, doc.longitude, 0, 0) < 2000 RETURN doc Compared to using the special purpose AQL functions this approach has the advantage that it is more composable, and will also honor any `LIMIT` values used in the AQL query. * potential fix for shutdown hangs on OSX * added KB, MB, GB prefix for integer parameters, % for integer parameters with a base value * added JEMALLOC 4.5.0 * added `--vm.resident-limit` and `--vm.path` for file-backed memory mapping after reaching a configurable maximum RAM size * try recommended limit for file descriptors in case of unlimited hard limit * issue #2413: improve logging in case of lock timeout and deadlocks * added log topic attribute to /_admin/log api * removed internal build option `USE_DEV_TIMERS` Enabling this option activated some proprietary timers for only selected events in arangod. Instead better use `perf` to gather timings. v3.2.alpha3 (2017-03-22) ------------------------ * increase default collection lock timeout from 30 to 900 seconds * added function `db._engine()` for retrieval of storage engine information at server runtime There is also an HTTP REST handler at GET /_api/engine that returns engine information. * require at least cmake 3.2 for building ArangoDB * make arangod start with less V8 JavaScript contexts This speeds up the server start (a little bit) and makes it use less memory. Whenever a V8 context is needed by a Foxx action or some other operation and there is no usable V8 context, a new one will be created dynamically now. Up to `--javascript.v8-contexts` V8 contexts will be created, so this option will change its meaning. Previously as many V8 contexts as specified by this option were created at server start, and the number of V8 contexts did not change at runtime. Now up to this number of V8 contexts will be in use at the same time, but the actual number of V8 contexts is dynamic. The garbage collector thread will automatically delete unused V8 contexts after a while. The number of spare contexts will go down to as few as configured in the new option `--javascript.v8-contexts-minimum`. Actually that many V8 contexts are also created at server start. The first few requests in new V8 contexts will take longer than in contexts that have been there already. Performance may therefore suffer a bit for the initial requests sent to ArangoDB or when there are only few but performance- critical situations in which new V8 contexts will be created. If this is a concern, it can easily be fixed by setting `--javascript.v8-contexts-minimum` and `--javascript.v8-contexts` to a relatively high value, which will guarantee that many number of V8 contexts to be created at startup and kept around even when unused. Waiting for an unused V8 context will now also abort if no V8 context can be acquired/created after 120 seconds. * improved diagnostic messages written to logfiles by supervisor process * fixed issue #2367 * added "bindVars" to attributes of currently running and slow queries * added "jsonl" as input file type for arangoimp * upgraded version of bundled zlib library from 1.2.8 to 1.2.11 * added input file type `auto` for arangoimp so it can automatically detect the type of the input file from the filename extension * fixed variables parsing in GraphQL * added `--translate` option for arangoimp to translate attribute names from the input files to attribute names expected by ArangoDB The `--translate` option can be specified multiple times (once per translation to be executed). The following example renames the "id" column from the input file to "_key", and the "from" column to "_from", and the "to" column to "_to": arangoimp --type csv --file data.csv --translate "id=_key" --translate "from=_from" --translate "to=_to" `--translate` works for CSV and TSV inputs only. * changed default value for `--server.max-packet-size` from 128 MB to 256 MB * fixed issue #2350 * fixed issue #2349 * fixed issue #2346 * fixed issue #2342 * change default string truncation length from 80 characters to 256 characters for `print`/`printShell` functions in ArangoShell and arangod. This will emit longer prefixes of string values before truncating them with `...`, which is helpful for debugging. * always validate incoming JSON HTTP requests for duplicate attribute names Incoming JSON data with duplicate attribute names will now be rejected as invalid. Previous versions of ArangoDB only validated the uniqueness of attribute names inside incoming JSON for some API endpoints, but not consistently for all APIs. * don't let read-only transactions block the WAL collector * allow passing own `graphql-sync` module instance to Foxx GraphQL router * arangoexport can now export to csv format * arangoimp: fixed issue #2214 * Foxx: automatically add CORS response headers * added "OPTIONS" to CORS `access-control-allow-methods` header * Foxx: Fix arangoUser sometimes not being set correctly * fixed issue #1974 v3.2.alpha2 (2017-02-20) ------------------------ * ui: fixed issue #2065 * ui: fixed a dashboard related memory issue * Internal javascript rest actions will now hide their stack traces to the client unless maintainer mode is activated. Instead they will always log to the logfile * Removed undocumented internal HTTP API: * PUT _api/edges The documented GET _api/edges and the undocumented POST _api/edges remains unmodified. * updated V8 version to 5.7.0.0 * change undocumented behavior in case of invalid revision ids in If-Match and If-None-Match headers from 400 (BAD) to 412 (PRECONDITION FAILED). * change undocumented behavior in case of invalid revision ids in JavaScript document operations from 1239 ("illegal document revision") to 1200 ("conflict"). * added data export tool, arangoexport. arangoexport can be used to export collections to json, jsonl or xml and export a graph or collections to xgmml. * fixed a race condition when closing a connection * raised default hard limit on threads for very small to 64 * fixed negative counting of http connection in UI v3.2.alpha1 (2017-02-05) ------------------------ * added figure `httpRequests` to AQL query statistics * removed revisions cache intermediate layer implementation * obsoleted startup options `--database.revision-cache-chunk-size` and `--database.revision-cache-target-size` * fix potential port number over-/underruns * added startup option `--log.shorten-filenames` for controlling whether filenames in log messages should be shortened to just the filename with the absolute path * removed IndexThreadFeature, made `--database.index-threads` option obsolete * changed index filling to make it more parallel, dispatch tasks to boost::asio * more detailed stacktraces in Foxx apps * generated Foxx services now use swagger tags v3.1.24 (XXXX-XX-XX) -------------------- * fixed one more LIMIT issue in traversals v3.1.23 (2017-06-19) -------------------- * potentially fixed issue #2559: Duplicate _key generated on insertion * fix races in SSL communication code * fix invalid results (too many) when a skipping LIMIT was used for a traversal. `LIMIT x` or `LIMIT 0, x` were not affected, but `LIMIT s, x` may have returned too many results * fix invalid first group results for sorted AQL COLLECT when LIMIT was used * fix invalid locking in JWT authentication cache, which could have crashed the server * fix undefined behavior in traverser when traversals were used inside a FOR loop v3.1.22 (2017-06-07) -------------------- * fixed issue #2505: Problem with export + report of a bug * documented changed behavior of WITH * fixed ui glitch in aardvark * avoid agency compaction bug * fixed issue #2283: disabled proxy communication internally v3.1.21 (2017-05-22) -------------------- * fixed issue #2488: AQL operator IN error when data use base64 chars * more randomness in seeding RNG v3.1.20 (2016-05-16) -------------------- * fixed incorrect sorting for distributeShardsLike * improve reliability of AgencyComm communication with Agency * fixed shard numbering bug, where ids were erroneously incremented by 1 * remove an unnecessary precondition in createCollectionCoordinator * funny fail rotation fix * fix in SimpleHttpClient for correct advancement of readBufferOffset * forward SIG_HUP in supervisor process to the server process to fix log rotation You need to stop the remaining arangod server process manually for the upgrade to work. v3.1.19 (2017-04-28) -------------------- * Fixed a StackOverflow issue in Traversal and ShortestPath. Occurred if many (>1000) input values in a row do not return any result. Fixes issue: #2445 * fixed issue #2448 * fixed issue #2442 * added 'x-content-type-options: nosniff' to avoid MSIE bug * fixed issue #2441 * fixed issue #2440 * Fixed a StackOverflow issue in Traversal and ShortestPath. Occurred if many (>1000) input values in a row do not return any result. Fixes issue: #2445 * fix occasional hanging shutdowns on OS X v3.1.18 (2017-04-18) -------------------- * fixed error in continuous synchronization of collections * fixed spurious hangs on server shutdown * better error messages during restore collection * completely overhaul supervision. More detailed tests * Fixed a dead-lock situation in cluster traversers, it could happen in rare cases if the computation on one DBServer could be completed much earlier than the other server. It could also be restricted to SmartGraphs only. * (Enterprise Edition only) Fixed a bug in SmartGraph DepthFirstSearch. In some more complicated queries, the maxDepth limit of 1 was not considered strictly enough, causing the traverser to do unlimited depth searches. * fixed issue #2415 * fixed issue #2422 * fixed issue #1974 v3.1.17 (2017-04-04) -------------------- * (Enterprise Edition only) fixed a bug where replicationFactor was not correctly forwarded in SmartGraph creation. * fixed issue #2404 * fixed issue #2397 * ui - fixed smart graph option not appearing * fixed issue #2389 * fixed issue #2400 v3.1.16 (2017-03-27) -------------------- * fixed issue #2392 * try to raise file descriptors to at least 8192, warn otherwise * ui - aql editor improvements + updated ace editor version (memory leak) * fixed lost HTTP requests * ui - fixed some event issues * avoid name resolution when given connection string is a valid ip address * helps with issue #1842, bug in COLLECT statement in connection with LIMIT. * fix locking bug in cluster traversals * increase lock timeout defaults * increase various cluster timeouts * limit default target size for revision cache to 1GB, which is better for tight RAM situations (used to be 40% of (totalRAM - 1GB), use --database.revision-cache-target-size to get back the old behavior * fixed a bug with restarted servers indicating status as "STARTUP" rather that "SERVING" in Nodes UI. v3.1.15 (2017-03-20) -------------------- * add logrotate configuration as requested in #2355 * fixed issue #2376 * ui - changed document api due a chrome bug * ui - fixed a submenu bug * added endpoint /_api/cluster/endpoints in cluster case to get all coordinator endpoints * fix documentation of /_api/endpoint, declaring this API obsolete. * Foxx response objects now have a `type` method for manipulating the content-type header * Foxx tests now support `xunit` and `tap` reporters v3.1.14 (2017-03-13) -------------------- * ui - added feature request (multiple start nodes within graph viewer) #2317 * added missing locks to authentication cache methods * ui - added feature request (multiple start nodes within graph viewer) #2317 * ui - fixed wrong merge of statistics information from different coordinators * ui - fixed issue #2316 * ui - fixed wrong protocol usage within encrypted environment * fixed compile error on Mac Yosemite * minor UI fixes v3.1.13 (2017-03-06) -------------------- * fixed variables parsing in GraphQL * fixed issue #2214 * fixed issue #2342 * changed thread handling to queue only user requests on coordinator * use exponential backoff when waiting for collection locks * repair short name server lookup in cluster in the case of a removed server v3.1.12 (2017-02-28) -------------------- * disable shell color escape sequences on Windows * fixed issue #2326 * fixed issue #2320 * fixed issue #2315 * fixed a race condition when closing a connection * raised default hard limit on threads for very small to 64 * fixed negative counting of http connection in UI * fixed a race when renaming collections * fixed a race when dropping databases v3.1.11 (2017-02-17) -------------------- * fixed a race between connection closing and sending out last chunks of data to clients when the "Connection: close" HTTP header was set in requests * ui: optimized smart graph creation usability * ui: fixed #2308 * fixed a race in async task cancellation via `require("@arangodb/tasks").unregisterTask()` * fixed spuriously hanging threads in cluster AQL that could sit idle for a few minutes * fixed potential numeric overflow for big index ids in index deletion API * fixed sort issue in cluster, occurring when one of the local sort buffers of a GatherNode was empty * reduce number of HTTP requests made for certain kinds of join queries in cluster, leading to speedup of some join queries * supervision deals with demised coordinators correctly again * implement a timeout in TraverserEngineRegistry * agent communication reduced in large batches of append entries RPCs * inception no longer estimates RAFT timings * compaction in agents has been moved to a separate thread * replicated logs hold local timestamps * supervision jobs failed leader and failed follower revisited for function in precarious stability situations * fixed bug in random number generator for 64bit int v3.1.10 (2017-02-02) -------------------- * updated versions of bundled node modules: - joi: from 8.4.2 to 9.2.0 - joi-to-json-schema: from 2.2.0 to 2.3.0 - sinon: from 1.17.4 to 1.17.6 - lodash: from 4.13.1 to 4.16.6 * added shortcut for AQL ternary operator instead of `condition ? true-part : false-part` it is now possible to also use a shortcut variant `condition ? : false-part`, e.g. FOR doc IN docs RETURN doc.value ?: 'not present' instead of FOR doc IN docs RETURN doc.value ? doc.value : 'not present' * fixed wrong sorting order in cluster, if an index was used to sort with many shards. * added --replication-factor, --number-of-shards and --wait-for-sync to arangobench * turn on UTF-8 string validation for VelocyPack values received via VST connections * fixed issue #2257 * upgraded Boost version to 1.62.0 * added optional detail flag for db..count() setting the flag to `true` will make the count operation returned the per-shard counts for the collection: db._create("test", { numberOfShards: 10 }); for (i = 0; i < 1000; ++i) { db.test.insert({value: i}); } db.test.count(true); { "s100058" : 99, "s100057" : 103, "s100056" : 100, "s100050" : 94, "s100055" : 90, "s100054" : 122, "s100051" : 109, "s100059" : 99, "s100053" : 95, "s100052" : 89 } * added optional memory limit for AQL queries: db._query("FOR i IN 1..100000 SORT i RETURN i", {}, { options: { memoryLimit: 100000 } }); This option limits the default maximum amount of memory (in bytes) that a single AQL query can use. When a single AQL query reaches the specified limit value, the query will be aborted with a *resource limit exceeded* exception. In a cluster, the memory accounting is done per shard, so the limit value is effectively a memory limit per query per shard. The global limit value can be overriden per query by setting the *memoryLimit* option value for individual queries when running an AQL query. * added server startup option `--query.memory-limit` * added convenience function to create vertex-centric indexes. Usage: `db.collection.ensureVertexCentricIndex("label", {type: "hash", direction: "outbound"})` That will create an index that can be used on OUTBOUND with filtering on the edge attribute `label`. * change default log output for tools to stdout (instead of stderr) * added option -D to define a configuration file environment key=value * changed encoding behavior for URLs encoded in the C++ code of ArangoDB: previously the special characters `-`, `_`, `~` and `.` were returned as-is after URL-encoding, now `.` will be encoded to be `%2e`. This also changes the behavior of how incoming URIs are processed: previously occurrences of `..` in incoming request URIs were collapsed (e.g. `a/../b/` was collapsed to a plain `b/`). Now `..` in incoming request URIs are not collapsed. * Foxx request URL suffix is no longer unescaped * @arangodb/request option json now defaults to `true` if the response body is not empty and encoding is not explicitly set to `null` (binary). The option can still be set to `false` to avoid unnecessary attempts at parsing the response as JSON. * Foxx configuration values for unknown options will be discarded when saving the configuration in production mode using the web interface * module.context.dependencies is now immutable * process.stdout.isTTY now returns `true` in arangosh and when running arangod with the `--console` flag * add support for Swagger tags in Foxx v3.1.9 (XXXX-XX-XX) ------------------- * macos CLI package: store databases and apps in the users home directory * ui: fixed re-login issue within a non system db, when tab was closed * fixed a race in the VelocyStream Commtask implementation * fixed issue #2256 v3.1.8 (2017-01-09) ------------------- * add Windows silent installer * add handling of debug symbols during Linux & windows release builds. * fixed issue #2181 * fixed issue #2248: reduce V8 max old space size from 3 GB to 1 GB on 32 bit systems * upgraded Boost version to 1.62.0 * fixed issue #2238 * fixed issue #2234 * agents announce new endpoints in inception phase to leader * agency leadership accepts updated endpoints to given uuid * unified endpoints replace localhost with 127.0.0.1 * fix several problems within an authenticated cluster v3.1.7 (2016-12-29) ------------------- * fixed one too many elections in RAFT * new agency comm backported from devel v3.1.6 (2016-12-20) ------------------- * fixed issue #2227 * fixed issue #2220 * agency constituent/agent bug fixes in race conditions picking up leadership * supervision does not need waking up anymore as it is running regardless * agents challenge their leadership more rigorously v3.1.5 (2016-12-16) ------------------- * lowered default value of `--database.revision-cache-target-size` from 75% of RAM to less than 40% of RAM * fixed issue #2218 * fixed issue #2217 * Foxx router.get/post/etc handler argument can no longer accidentally omitted * fixed issue #2223 v3.1.4 (2016-12-08) ------------------- * fixed issue #2211 * fixed issue #2204 * at cluster start, coordinators wait until at least one DBserver is there, and either at least two DBservers are there or 15s have passed, before they initiate the bootstrap of system collections. * more robust agency startup from devel * supervision's AddFollower adds many followers at once * supervision has new FailedFollower job * agency's Node has new method getArray * agency RAFT timing estimates more conservative in waitForSync scenario * agency RAFT timing estimates capped at maximum 2.0/10.0 for low/high v3.1.3 (2016-12-02) ------------------- * fix a traversal bug when using skiplist indexes: if we have a skiplist of ["a", "unused", "_from"] and a traversal like: FOR v,e,p IN OUTBOUND @start @@edges FILTER p.edges[0].a == 'foo' RETURN v And the above index applied on "a" is considered better than EdgeIndex, than the executor got into undefined behavior. * fix endless loop when trying to create a collection with replicationFactor: -1 v3.1.2 (2016-11-24) ------------------- * added support for descriptions field in Foxx dependencies * (Enterprise Edition only) fixed a bug in the statistic report for SmartGraph traversals. Now they state correctly how many documents were fetched from the index and how many have been filtered. * Prevent uniform shard distribution when replicationFactor == numServers v3.1.1 (2016-11-15) ------------------- * fixed issue #2176 * fixed issue #2168 * display index usage of traversals in AQL explainer output (previously missing) * fixed issue #2163 * preserve last-used HLC value across server starts * allow more control over handling of pre-3.1 _rev values this changes the server startup option `--database.check-30-revisions` from a boolean (true/false) parameter to a string parameter with the following possible values: - "fail": will validate _rev values of 3.0 collections on collection loading and throw an exception when invalid _rev values are found. in this case collections with invalid _rev values are marked as corrupted and cannot be used in the ArangoDB 3.1 instance. the fix procedure for such collections is to export the collections from 3.0 database with arangodump and restore them in 3.1 with arangorestore. collections that do not contain invalid _rev values are marked as ok and will not be re-checked on following loads. collections that contain invalid _rev values will be re-checked on following loads. - "true": will validate _rev values of 3.0 collections on collection loading and print a warning when invalid _rev values are found. in this case collections with invalid _rev values can be used in the ArangoDB 3.1 instance. however, subsequent operations on documents with invalid _rev values may silently fail or fail with explicit errors. the fix procedure for such collections is to export the collections from 3.0 database with arangodump and restore them in 3.1 with arangorestore. collections that do not contain invalid _rev values are marked as ok and will not be re-checked on following loads. collections that contain invalid _rev values will be re-checked on following loads. - "false": will not validate _rev values on collection loading and not print warnings. no hint is given when invalid _rev values are found. subsequent operations on documents with invalid _rev values may silently fail or fail with explicit errors. this setting does not affect whether collections are re-checked later. collections will be re-checked on following loads if `--database.check-30-revisions` is later set to either `true` or `fail`. The change also suppresses warnings that were printed when collections were restored using arangorestore, and the restore data contained invalid _rev values. Now these warnings are suppressed, and new HLC _rev values are generated for these documents as before. * added missing functions to AQL syntax highlighter in web interface * fixed display of `ANY` direction in traversal explainer output (direction `ANY` was shown as either `INBOUND` or `OUTBOUND`) * changed behavior of toJSON() function when serializing an object before saving it in the database if an object provides a toJSON() function, this function is still called for serializing it. the change is that the result of toJSON() is not stringified anymore, but saved as is. previous versions of ArangoDB called toJSON() and after that additionally stringified its result. This change will affect the saving of JS Buffer objects, which will now be saved as arrays of bytes instead of a comma-separated string of the Buffer's byte contents. * allow creating unique indexes on more attributes than present in shardKeys The following combinations of shardKeys and indexKeys are allowed/not allowed: shardKeys indexKeys a a ok a b not ok a a b ok a b a not ok a b b not ok a b a b ok a b a b c ok a b c a b not ok a b c a b c ok * fixed wrong version in web interface login screen (EE only) * make web interface not display an exclamation mark next to ArangoDB version number 3.1 * fixed search for arbitrary document attributes in web interface in case multiple search values were used on different attribute names. in this case, the search always produced an empty result * disallow updating `_from` and `_to` values of edges in SmartGraphs. Updating these attributes would lead to potential redistribution of edges to other shards, which must be avoided. * fixed issue #2148 * updated graphql-sync dependency to 0.6.2 * fixed issue #2156 * fixed CRC4 assembly linkage v3.1.0 (2016-10-29) ------------------- * AQL breaking change in cluster: from ArangoDB 3.1 onwards `WITH` is required for traversals in a clustered environment in order to avoid deadlocks. Note that for queries that access only a single collection or that have all collection names specified somewhere else in the query string, there is no need to use *WITH*. *WITH* is only useful when the AQL query parser cannot automatically figure out which collections are going to be used by the query. *WITH* is only useful for queries that dynamically access collections, e.g. via traversals, shortest path operations or the *DOCUMENT()* function. more info can be found [here](https://github.com/arangodb/arangodb/blob/devel/Documentation/Books/AQL/Operations/With.md) * added AQL function `DISTANCE` to calculate the distance between two arbitrary coordinates (haversine formula) * fixed issue #2110 * added Auto-adaptation of RAFT timings as calculations only v3.1.rc2 (2016-10-10) --------------------- * second release candidate v3.1.rc1 (2016-09-30) --------------------- * first release candidate v3.1.alpha2 (2016-09-01) ------------------------ * added module.context.createDocumentationRouter to replace module.context.apiDocumentation * bug in RAFT implementation of reads. dethroned leader still answered requests in isolation * ui: added new graph viewer * ui: aql-editor added tabular & graph display * ui: aql-editor improved usability * ui: aql-editor: query profiling support * fixed issue #2109 * fixed issue #2111 * fixed issue #2075 * added AQL function `DISTANCE` to calculate the distance between two arbitrary coordinates (haversine formula) * rewrote scheduler and dispatcher based on boost::asio parameters changed: `--scheduler.threads` and `--server.threads` are now merged into a single one: `--server.threads` hidden `--server.extra-threads` has been removed hidden `--server.aql-threads` has been removed hidden `--server.backend` has been removed hidden `--server.show-backends` has been removed hidden `--server.thread-affinity` has been removed * fixed issue #2086 * fixed issue #2079 * fixed issue #2071 make the AQL query optimizer inject filter condition expressions referred to by variables during filter condition aggregation. For example, in the following query FOR doc IN collection LET cond1 = (doc.value == 1) LET cond2 = (doc.value == 2) FILTER cond1 || cond2 RETURN { doc, cond1, cond2 } the optimizer will now inject the conditions for `cond1` and `cond2` into the filter condition `cond1 || cond2`, expanding it to `(doc.value == 1) || (doc.value == 2)` and making these conditions available for index searching. Note that the optimizer previously already injected some conditions into other conditions, but only if the variable that defined the condition was not used elsewhere. For example, the filter condition in the query FOR doc IN collection LET cond = (doc.value == 1) FILTER cond RETURN { doc } already got optimized before because `cond` was only used once in the query and the optimizer decided to inject it into the place where it was used. This only worked for variables that were referred to once in the query. When a variable was used multiple times, the condition was not injected as in the following query: FOR doc IN collection LET cond = (doc.value == 1) FILTER cond RETURN { doc, cond } The fix for #2070 now will enable this optimization so that the query can use an index on `doc.value` if available. * changed behavior of AQL array comparison operators for empty arrays: * `ALL` and `ANY` now always return `false` when the left-hand operand is an empty array. The behavior for non-empty arrays does not change: * `[] ALL == 1` will return `false` * `[1] ALL == 1` will return `true` * `[1, 2] ALL == 1` will return `false` * `[2, 2] ALL == 1` will return `false` * `[] ANY == 1` will return `false` * `[1] ANY == 1` will return `true` * `[1, 2] ANY == 1` will return `true` * `[2, 2] ANY == 1` will return `false` * `NONE` now always returns `true` when the left-hand operand is an empty array. The behavior for non-empty arrays does not change: * `[] NONE == 1` will return `true` * `[1] NONE == 1` will return `false` * `[1, 2] NONE == 1` will return `false` * `[2, 2] NONE == 1` will return `true` * added experimental AQL functions `JSON_STRINGIFY` and `JSON_PARSE` * added experimental support for incoming gzip-compressed requests * added HTTP REST APIs for online log level adjustments: - GET `/_admin/log/level` returns the current log level settings - PUT `/_admin/log/level` modifies the current log level settings * PATCH /_api/gharial/{graph-name}/vertex/{collection-name}/{vertex-key} - changed default value for keepNull to true * PATCH /_api/gharial/{graph-name}/edge/{collection-name}/{edge-key} - changed default value for keepNull to true * renamed `maximalSize` attribute in parameter.json files to `journalSize` The `maximalSize` attribute will still be picked up from collections that have not been adjusted. Responses from the replication API will now also use `journalSize` instead of `maximalSize`. * added `--cluster.system-replication-factor` in order to adjust the replication factor for new system collections * fixed issue #2012 * added a memory expection in case V8 memory gets too low * added Optimizer Rule for other indexes in Traversals this allows AQL traversals to use other indexes than the edge index. So traversals with filters on edges can now make use of more specific indexes, e.g. FOR v, e, p IN 2 OUTBOUND @start @@edge FILTER p.edges[0].foo == "bar" will prefer a Hash Index on [_from, foo] above the EdgeIndex. * fixed epoch computation in hybrid logical clock * fixed thread affinity * replaced require("internal").db by require("@arangodb").db * added option `--skip-lines` for arangoimp this allows skipping the first few lines from the import file in case the CSV or TSV import are used * fixed periodic jobs: there should be only one instance running - even if it runs longer than the period * improved performance of primary index and edge index lookups * optimizations for AQL `[*]` operator in case no filter, no projection and no offset/limit are used * added AQL function `OUTERSECTION` to return the symmetric difference of its input arguments * Foxx manifests of installed services are now saved to disk with indentation * Foxx tests and scripts in development mode should now always respect updated files instead of loading stale modules * When disabling Foxx development mode the setup script is now re-run * Foxx now provides an easy way to directly serve GraphQL requests using the `@arangodb/foxx/graphql` module and the bundled `graphql-sync` dependency * Foxx OAuth2 module now correctly passes the `access_token` to the OAuth2 server * added iconv-lite and timezone modules * web interface now allows installing GitHub and zip services in legacy mode * added module.context.createDocumentationRouter to replace module.context.apiDocumentation * bug in RAFT implementation of reads. dethroned leader still answered requests in isolation * all lambdas in ClusterInfo might have been left with dangling references. * Agency bug fix for handling of empty json objects as values. * Foxx tests no longer support the Mocha QUnit interface as this resulted in weird inconsistencies in the BDD and TDD interfaces. This fixes the TDD interface as well as out-of-sequence problems when using the BDD before/after functions. * updated bundled JavaScript modules to latest versions; joi has been updated from 8.4 to 9.2 (see [joi 9.0.0 release notes](https://github.com/hapijs/joi/issues/920) for information on breaking changes and new features) * fixed issue #2139 * updated graphql-sync dependency to 0.6.2 * fixed issue #2156 v3.0.13 (XXXX-XX-XX) -------------------- * fixed issue #2315 * fixed issue #2210 v3.0.12 (2016-11-23) -------------------- * fixed issue #2176 * fixed issue #2168 * fixed issues #2149, #2159 * fixed error reporting for issue #2158 * fixed assembly linkage bug in CRC4 module * added support for descriptions field in Foxx dependencies v3.0.11 (2016-11-08) -------------------- * fixed issue #2140: supervisor dies instead of respawning child * fixed issue #2131: use shard key value entered by user in web interface * fixed issue #2129: cannot kill a long-run query * fixed issue #2110 * fixed issue #2081 * fixed issue #2038 * changes to Foxx service configuration or dependencies should now be stored correctly when options are cleared or omitted * Foxx tests no longer support the Mocha QUnit interface as this resulted in weird inconsistencies in the BDD and TDD interfaces. This fixes the TDD interface as well as out-of-sequence problems when using the BDD before/after functions. * fixed issue #2148 v3.0.10 (2016-09-26) -------------------- * fixed issue #2072 * fixed issue #2070 * fixed slow cluster startup issues. supervision will demonstrate more patience with db servers v3.0.9 (2016-09-21) ------------------- * fixed issue #2064 * fixed issue #2060 * speed up `collection.any()` and skiplist index creation * fixed multiple issues where ClusterInfo bug hung agency in limbo timing out on multiple collection and database callbacks v3.0.8 (2016-09-14) ------------------- * fixed issue #2052 * fixed issue #2005 * fixed issue #2039 * fixed multiple issues where ClusterInfo bug hung agency in limbo timing out on multiple collection and database callbacks v3.0.7 (2016-09-05) ------------------- * new supervision job handles db server failure during collection creation. v3.0.6 (2016-09-02) ------------------- * fixed issue #2026 * slightly better error diagnostics for AQL query compilation and replication * fixed issue #2018 * fixed issue #2015 * fixed issue #2012 * fixed wrong default value for arangoimp's `--on-duplicate` value * fix execution of AQL traversal expressions when there are multiple conditions that refer to variables set outside the traversal * properly return HTTP 503 in JS actions when backend is gone * supervision creates new key in agency for failed servers * new shards will not be allocated on failed or cleaned servers v3.0.5 (2016-08-18) ------------------- * execute AQL ternary operator via C++ if possible * fixed issue #1977 * fixed extraction of _id attribute in AQL traversal conditions * fix SSL agency endpoint * Minimum RAFT timeout was one order of magnitude to short. * Optimized RAFT RPCs from leader to followers for efficiency. * Optimized RAFT RPC handling on followers with respect to compaction. * Fixed bug in handling of duplicates and overlapping logs * Fixed bug in supervision take over after leadership change. v3.0.4 (2016-08-01) ------------------- * added missing lock for periodic jobs access * fix multiple Foxx related cluster issues * fix handling of empty AQL query strings * fixed issue in `INTERSECTION` AQL function with duplicate elements in the source arrays * fixed issue #1970 * fixed issue #1968 * fixed issue #1967 * fixed issue #1962 * fixed issue #1959 * replaced require("internal").db by require("@arangodb").db * fixed issue #1954 * fixed issue #1953 * fixed issue #1950 * fixed issue #1949 * fixed issue #1943 * fixed segfault in V8, by backporting https://bugs.chromium.org/p/v8/issues/detail?id=5033 * Foxx OAuth2 module now correctly passes the `access_token` to the OAuth2 server * fixed credentialed CORS requests properly respecting --http.trusted-origin * fixed a crash in V8Periodic task (forgotten lock) * fixed two bugs in synchronous replication (syncCollectionFinalize) v3.0.3 (2016-07-17) ------------------- * fixed issue #1942 * fixed issue #1941 * fixed array index batch insertion issues for hash indexes that caused problems when no elements remained for insertion * fixed AQL MERGE() function with External objects originating from traversals * fixed some logfile recovery errors with error message "document not found" * fixed issue #1937 * fixed issue #1936 * improved performance of arangorestore in clusters with synchronous replication * Foxx tests and scripts in development mode should now always respect updated files instead of loading stale modules * When disabling Foxx development mode the setup script is now re-run * Foxx manifests of installed services are now saved to disk with indentation v3.0.2 (2016-07-09) ------------------- * fixed assertion failure in case multiple remove operations were used in the same query * fixed upsert behavior in case upsert was used in a loop with the same document example * fixed issue #1930 * don't expose local file paths in Foxx error messages. * fixed issue #1929 * make arangodump dump the attribute `isSystem` when dumping the structure of a collection, additionally make arangorestore not fail when the attribute is missing * fixed "Could not extract custom attribute" issue when using COLLECT with MIN/MAX functions in some contexts * honor presence of persistent index for sorting * make AQL query optimizer not skip "use-indexes-rule", even if enough plans have been created already * make AQL optimizer not skip "use-indexes-rule", even if enough execution plans have been created already * fix double precision value loss in VelocyPack JSON parser * added missing SSL support for arangorestore * improved cluster import performance * fix Foxx thumbnails on DC/OS * fix Foxx configuration not being saved * fix Foxx app access from within the frontend on DC/OS * add option --default-replication-factor to arangorestore and simplify the control over the number of shards when restoring * fix a bug in the VPack -> V8 conversion if special attributes _key, _id, _rev, _from and _to had non-string values, which is allowed below the top level * fix malloc_usable_size for darwin v3.0.1 (2016-06-30) ------------------- * fixed periodic jobs: there should be only one instance running - even if it runs longer than the period * increase max. number of collections in AQL queries from 32 to 256 * fixed issue #1916: header "authorization" is required" when opening services page * fixed issue #1915: Explain: member out of range * fixed issue #1914: fix unterminated buffer * don't remove lockfile if we are the same (now stale) pid fixes docker setups (our pid will always be 1) * do not use revision id comparisons in compaction for determining whether a revision is obsolete, but marker memory addresses this ensures revision ids don't matter when compacting documents * escape Unicode characters in JSON HTTP responses this converts UTF-8 characters in HTTP responses of arangod into `\uXXXX` escape sequences. This makes the HTTP responses fit into the 7 bit ASCII character range, which speeds up HTTP response parsing for some clients, namely node.js/v8 * add write before read collections when starting a user transaction this allows specifying the same collection in both read and write mode without unintended side effects * fixed buffer overrun that occurred when building very large result sets * index lookup optimizations for primary index and edge index * fixed "collection is a nullptr" issue when starting a traversal from a transaction * enable /_api/import on coordinator servers v3.0.0 (2016-06-22) ------------------- * minor GUI fixes * fix for replication and nonces v3.0.0-rc3 (2016-06-19) ----------------------- * renamed various Foxx errors to no longer refer to Foxx services as apps * adjusted various error messages in Foxx to be more informative * specifying "files" in a Foxx manifest to be mounted at the service root no longer results in 404s when trying to access non-file routes * undeclared path parameters in Foxx no longer break the service * trusted reverse proxy support is now handled more consistently * ArangoDB request compatibility and user are now exposed in Foxx * all bundled NPM modules have been upgraded to their latest versions v3.0.0-rc2 (2016-06-12) ----------------------- * added option `--server.max-packet-size` for client tools * renamed option `--server.ssl-protocol` to `--ssl.protocol` in client tools (was already done for arangod, but overlooked for client tools) * fix handling of `--ssl.protocol` value 5 (TLS v1.2) in client tools, which claimed to support it but didn't * config file can use '@include' to include a different config file as base v3.0.0-rc1 (2016-06-10) ----------------------- * the user management has changed: it now has users that are independent of databases. A user can have one or more database assigned to the user. * forward ported V8 Comparator bugfix for inline heuristics from https://github.com/v8/v8/commit/5ff7901e24c2c6029114567de5a08ed0f1494c81 * changed to-string conversion for AQL objects and arrays, used by the AQL function `TO_STRING()` and implicit to-string casts in AQL - arrays are now converted into their JSON-stringify equivalents, e.g. - `[ ]` is now converted to `[]` - `[ 1, 2, 3 ]` is now converted to `[1,2,3]` - `[ "test", 1, 2 ] is now converted to `["test",1,2]` Previous versions of ArangoDB converted arrays with no members into the empty string, and non-empty arrays into a comma-separated list of member values, without the surrounding angular brackets. Additionally, string array members were not enclosed in quotes in the result string: - `[ ]` was converted to `` - `[ 1, 2, 3 ]` was converted to `1,2,3` - `[ "test", 1, 2 ] was converted to `test,1,2` - objects are now converted to their JSON-stringify equivalents, e.g. - `{ }` is converted to `{}` - `{ a: 1, b: 2 }` is converted to `{"a":1,"b":2}` - `{ "test" : "foobar" }` is converted to `{"test":"foobar"}` Previous versions of ArangoDB always converted objects into the string `[object Object]` This change affects also the AQL functions `CONCAT()` and `CONCAT_SEPARATOR()` which treated array values differently in previous versions. Previous versions of ArangoDB automatically flattened array values on the first level of the array, e.g. `CONCAT([1, 2, 3, [ 4, 5, 6 ]])` produced `1,2,3,4,5,6`. Now this will produce `[1,2,3,[4,5,6]]`. To flatten array members on the top level, you can now use the more explicit `CONCAT(FLATTEN([1, 2, 3, [4, 5, 6]], 1))`. * added C++ implementations for AQL functions `SLICE()`, `CONTAINS()` and `RANDOM_TOKEN()` * as a consequence of the upgrade to V8 version 5, the implementation of the JavaScript `Buffer` object had to be changed. JavaScript `Buffer` objects in ArangoDB now always store their data on the heap. There is no shared pool for small Buffer values, and no pointing into existing Buffer data when extracting slices. This change may increase the cost of creating Buffers with short contents or when peeking into existing Buffers, but was required for safer memory management and to prevent leaks. * the `db` object's function `_listDatabases()` was renamed to just `_databases()` in order to make it more consistent with the existing `_collections()` function. Additionally the `db` object's `_listEndpoints()` function was renamed to just `_endpoints()`. * changed default value of `--server.authentication` from `false` to `true` in configuration files etc/relative/arangod.conf and etc/arangodb/arangod.conf.in. This means the server will be started with authentication enabled by default, requiring all client connections to provide authentication data when connecting to ArangoDB. Authentication can still be turned off via setting the value of `--server.authentication` to `false` in ArangoDB's configuration files or by specifying the option on the command-line. * Changed result format for querying all collections via the API GET `/_api/collection`. Previous versions of ArangoDB returned an object with an attribute named `collections` and an attribute named `names`. Both contained all available collections, but `collections` contained the collections as an array, and `names` contained the collections again, contained in an object in which the attribute names were the collection names, e.g. ``` { "collections": [ {"id":"5874437","name":"test","isSystem":false,"status":3,"type":2}, {"id":"17343237","name":"something","isSystem":false,"status":3,"type":2}, ... ], "names": { "test": {"id":"5874437","name":"test","isSystem":false,"status":3,"type":2}, "something": {"id":"17343237","name":"something","isSystem":false,"status":3,"type":2}, ... } } ``` This result structure was redundant, and therefore has been simplified to just ``` { "result": [ {"id":"5874437","name":"test","isSystem":false,"status":3,"type":2}, {"id":"17343237","name":"something","isSystem":false,"status":3,"type":2}, ... ] } ``` in ArangoDB 3.0. * added AQL functions `TYPENAME()` and `HASH()` * renamed arangob tool to arangobench * added AQL string comparison operator `LIKE` The operator can be used to compare strings like this: value LIKE search The operator is currently implemented by calling the already existing AQL function `LIKE`. This change also makes `LIKE` an AQL keyword. Using `LIKE` in either case as an attribute or collection name in AQL thus requires quoting. * make AQL optimizer rule "remove-unnecessary-calculations" fire in more cases The rule will now remove calculations that are used exactly once in other expressions (e.g. `LET a = doc RETURN a.value`) and calculations, or calculations that are just references (e.g. `LET a = b`). * renamed AQL optimizer rule "merge-traversal-filter" to "optimize-traversals" Additionally, the optimizer rule will remove unused edge and path result variables from the traversal in case they are specified in the `FOR` section of the traversal, but not referenced later in the query. This saves constructing edges and paths results. * added AQL optimizer rule "inline-subqueries" This rule can pull out certain subqueries that are used as an operand to a `FOR` loop one level higher, eliminating the subquery completely. For example, the query FOR i IN (FOR j IN [1,2,3] RETURN j) RETURN i will be transformed by the rule to: FOR i IN [1,2,3] RETURN i The query FOR name IN (FOR doc IN _users FILTER doc.status == 1 RETURN doc.name) LIMIT 2 RETURN name will be transformed into FOR tmp IN _users FILTER tmp.status == 1 LIMIT 2 RETURN tmp.name The rule will only fire when the subquery is used as an operand to a `FOR` loop, and if the subquery does not contain a `COLLECT` with an `INTO` variable. * added new endpoint "srv://" for DNS service records * The result order of the AQL functions VALUES and ATTRIBUTES has never been guaranteed and it only had the "correct" ordering by accident when iterating over objects that were not loaded from the database. This accidental behavior is now changed by introduction of VelocyPack. No ordering is guaranteed unless you specify the sort parameter. * removed configure option `--enable-logger` * added AQL array comparison operators All AQL comparison operators now also exist in an array variant. In the array variant, the operator is preceded with one of the keywords *ALL*, *ANY* or *NONE*. Using one of these keywords changes the operator behavior to execute the comparison operation for all, any, or none of its left hand argument values. It is therefore expected that the left hand argument of an array operator is an array. Examples: [ 1, 2, 3 ] ALL IN [ 2, 3, 4 ] // false [ 1, 2, 3 ] ALL IN [ 1, 2, 3 ] // true [ 1, 2, 3 ] NONE IN [ 3 ] // false [ 1, 2, 3 ] NONE IN [ 23, 42 ] // true [ 1, 2, 3 ] ANY IN [ 4, 5, 6 ] // false [ 1, 2, 3 ] ANY IN [ 1, 42 ] // true [ 1, 2, 3 ] ANY == 2 // true [ 1, 2, 3 ] ANY == 4 // false [ 1, 2, 3 ] ANY > 0 // true [ 1, 2, 3 ] ANY <= 1 // true [ 1, 2, 3 ] NONE < 99 // false [ 1, 2, 3 ] NONE > 10 // true [ 1, 2, 3 ] ALL > 2 // false [ 1, 2, 3 ] ALL > 0 // true [ 1, 2, 3 ] ALL >= 3 // false ["foo", "bar"] ALL != "moo" // true ["foo", "bar"] NONE == "bar" // false ["foo", "bar"] ANY == "foo" // true * improved AQL optimizer to remove unnecessary sort operations in more cases * allow enclosing AQL identifiers in forward ticks in addition to using backward ticks This allows for convenient writing of AQL queries in JavaScript template strings (which are delimited with backticks themselves), e.g. var q = `FOR doc IN ´collection´ RETURN doc.´name´`; * allow to set `print.limitString` to configure the number of characters to output before truncating * make logging configurable per log "topic" `--log.level ` sets the global log level to , e.g. `info`, `debug`, `trace`. `--log.level topic=` sets the log level for a specific topic. Currently, the following topics exist: `collector`, `compactor`, `mmap`, `performance`, `queries`, and `requests`. `performance` and `requests` are set to FATAL by default. `queries` is set to info. All others are set to the global level by default. The new log option `--log.output ` allows directing the global or per-topic log output to different outputs. The output definition "" can be one of "-" for stdin "+" for stderr "syslog://" "syslog:///" "file://" The option can be specified multiple times in order to configure the output for different log topics. To set up a per-topic output configuration, use `--log.output =`, e.g. queries=file://queries.txt logs all queries to the file "queries.txt". * the option `--log.requests-file` is now deprecated. Instead use `--log.level requests=info` `--log.output requests=file://requests.txt` * the option `--log.facility` is now deprecated. Instead use `--log.output requests=syslog://facility` * the option `--log.performance` is now deprecated. Instead use `--log.level performance=trace` * removed option `--log.source-filter` * removed configure option `--enable-logger` * change collection directory names to include a random id component at the end The new pattern is `collection--`, where `` is the collection id and `` is a random number. Previous versions of ArangoDB used a pattern `collection-` without the random number. ArangoDB 3.0 understands both the old and name directory name patterns. * removed mostly unused internal spin-lock implementation * removed support for pre-Windows 7-style locks. This removes compatibility for Windows versions older than Windows 7 (e.g. Windows Vista, Windows XP) and Windows 2008R2 (e.g. Windows 2008). * changed names of sub-threads started by arangod * added option `--default-number-of-shards` to arangorestore, allowing creating collections with a specifiable number of shards from a non-cluster dump * removed support for CoffeeScript source files * removed undocumented SleepAndRequeue * added WorkMonitor to inspect server threads * when downloading a Foxx service from the web interface the suggested filename is now based on the service's mount path instead of simply "app.zip" * the `@arangodb/request` response object now stores the parsed JSON response body in a property `json` instead of `body` when the request was made using the `json` option. The `body` instead contains the response body as a string. * the Foxx API has changed significantly, 2.8 services are still supported using a backwards-compatible "legacy mode" v2.8.12 (XXXX-XX-XX) -------------------- * issue #2091: decrease connect timeout to 5 seconds on startup * fixed issue #2072 * slightly better error diagnostics for some replication errors * fixed issue #1977 * fixed issue in `INTERSECTION` AQL function with duplicate elements in the source arrays * fixed issue #1962 * fixed issue #1959 * export aqlQuery template handler as require('org/arangodb').aql for forwards-compatibility v2.8.11 (2016-07-13) -------------------- * fixed array index batch insertion issues for hash indexes that caused problems when no elements remained for insertion * fixed issue #1937 v2.8.10 (2016-07-01) -------------------- * make sure next local _rev value used for a document is at least as high as the _rev value supplied by external sources such as replication * make adding a collection in both read- and write-mode to a transaction behave as expected (write includes read). This prevents the `unregister collection used in transaction` error * fixed sometimes invalid result for `byExample(...).count()` when an index plus post-filtering was used * fixed "collection is a nullptr" issue when starting a traversal from a transaction * honor the value of startup option `--database.wait-for-sync` (that is used to control whether new collections are created with `waitForSync` set to `true` by default) also when creating collections via the HTTP API (and thus the ArangoShell). When creating a collection via these mechanisms, the option was ignored so far, which was inconsistent. * fixed issue #1826: arangosh --javascript.execute: internal error (geo index issue) * fixed issue #1823: Arango crashed hard executing very simple query on windows v2.8.9 (2016-05-13) ------------------- * fixed escaping and quoting of extra parameters for executables in Mac OS X App * added "waiting for" status variable to web interface collection figures view * fixed undefined behavior in query cache invalidation * fixed access to /_admin/statistics API in case statistics are disable via option `--server.disable-statistics` * Foxx manager will no longer fail hard when Foxx store is unreachable unless installing a service from the Foxx store (e.g. when behind a firewall or GitHub is unreachable). v2.8.8 (2016-04-19) ------------------- * fixed issue #1805: Query: internal error (location: arangod/Aql/AqlValue.cpp:182). Please report this error to arangodb.com (while executing) * allow specifying collection name prefixes for `_from` and `_to` in arangoimp: To avoid specifying complete document ids (consisting of collection names and document keys) for *_from* and *_to* values when importing edges with arangoimp, there are now the options *--from-collection-prefix* and *--to-collection-prefix*. If specified, these values will be automatically prepended to each value in *_from* (or *_to* resp.). This allows specifying only document keys inside *_from* and/or *_to*. *Example* > arangoimp --from-collection-prefix users --to-collection-prefix products ... Importing the following document will then create an edge between *users/1234* and *products/4321*: ```js { "_from" : "1234", "_to" : "4321", "desc" : "users/1234 is connected to products/4321" } ``` * requests made with the interactive system API documentation in the web interface (Swagger) will now respect the active database instead of always using `_system` v2.8.7 (2016-04-07) ------------------- * optimized primary=>secondary failover * fix to-boolean conversion for documents in AQL * expose the User-Agent HTTP header from the ArangoShell since GitHub seems to require it now, and we use the ArangoShell for fetching Foxx repositories from GitHub * work with http servers that only send * fixed potential race condition between compactor and collector threads * fix removal of temporary directories on arangosh exit * javadoc-style comments in Foxx services are no longer interpreted as Foxx comments outside of controller/script/exports files (#1748) * removed remaining references to class syntax for Foxx Model and Repository from the documentation * added a safe-guard for corrupted master-pointer v2.8.6 (2016-03-23) ------------------- * arangosh can now execute JavaScript script files that contain a shebang in the first line of the file. This allows executing script files directly. Provided there is a script file `/path/to/script.js` with the shebang `#!arangosh --javascript.execute`: > cat /path/to/script.js #!arangosh --javascript.execute print("hello from script.js"); If the script file is made executable > chmod a+x /path/to/script.js it can be invoked on the shell directly and use arangosh for its execution: > /path/to/script.js hello from script.js This did not work in previous versions of ArangoDB, as the whole script contents (including the shebang) were treated as JavaScript code. Now shebangs in script files will now be ignored for all files passed to arangosh's `--javascript.execute` parameter. The alternative way of executing a JavaScript file with arangosh still works: > arangosh --javascript.execute /path/to/script.js hello from script.js * added missing reset of traversal state for nested traversals. The state of nested traversals (a traversal in an AQL query that was located in a repeatedly executed subquery or inside another FOR loop) was not reset properly, so that multiple invocations of the same nested traversal with different start vertices led to the nested traversal always using the start vertex provided on the first invocation. * fixed issue #1781: ArangoDB startup time increased tremendously * fixed issue #1783: SIGHUP should rotate the log v2.8.5 (2016-03-11) ------------------- * Add OpenSSL handler for TLS V1.2 as suggested by kurtkincaid in #1771 * fixed issue #1765 (The webinterface should display the correct query time) and #1770 (Display ACTUAL query time in aardvark's AQL editor) * Windows: the unhandled exception handler now calls the windows logging facilities directly without locks. This fixes lockups on crashes from the logging framework. * improve nullptr handling in logger. * added new endpoint "srv://" for DNS service records * `org/arangodb/request` no longer sets the content-type header to the string "undefined" when no content-type header should be sent (issue #1776) v2.8.4 (2016-03-01) ------------------- * global modules are no longer incorrectly resolved outside the ArangoDB JavaScript directory or the Foxx service's root directory (issue #1577) * improved error messages from Foxx and JavaScript (issues #1564, #1565, #1744) v2.8.3 (2016-02-22) ------------------- * fixed AQL filter condition collapsing for deeply-nested cases, potentially enabling usage of indexes in some dedicated cases * added parentheses in AQL explain command output to correctly display precedence of logical and arithmetic operators * Foxx Model event listeners defined on the model are now correctly invoked by the Repository methods (issue #1665) * Deleting a Foxx service in the frontend should now always succeed even if the files no longer exist on the file system (issue #1358) * Routing actions loaded from the database no longer throw exceptions when trying to load other modules using "require" * The `org/arangodb/request` response object now sets a property `json` to the parsed JSON response body in addition to overwriting the `body` property when the request was made using the `json` option. * Improved Windows stability * Fixed a bug in the interactive API documentation that would escape slashes in document-handle fields. Document handles are now provided as separate fields for collection name and document key. v2.8.2 (2016-02-09) ------------------- * the continuous replication applier will now prevent the master's WAL logfiles from being removed if they are still needed by the applier on the slave. This should help slaves that suffered from masters garbage collection WAL logfiles which would have been needed by the slave later. The initial synchronization will block removal of still needed WAL logfiles on the master for 10 minutes initially, and will extend this period when further requests are made to the master. Initial synchronization hands over its handle for blocking logfile removal to the continuous replication when started via the *setupReplication* function. In this case, continuous replication will extend the logfile removal blocking period for the required WAL logfiles when the slave makes additional requests. All handles that block logfile removal will time out automatically after at most 5 minutes should a master not be contacted by the slave anymore (e.g. in case the slave's replication is turned off, the slaves loses the connection to the master or the slave goes down). * added all-in-one function *setupReplication* to synchronize data from master to slave and start the continuous replication: require("@arangodb/replication").setupReplication(configuration); The command will return when the initial synchronization is finished and the continuous replication has been started, or in case the initial synchronization has failed. If the initial synchronization is successful, the command will store the given configuration on the slave. It also configures the continuous replication to start automatically if the slave is restarted, i.e. *autoStart* is set to *true*. If the command is run while the slave's replication applier is already running, it will first stop the running applier, drop its configuration and do a resynchronization of data with the master. It will then use the provided configuration, overwriting any previously existing replication configuration on the slave. The following example demonstrates how to use the command for setting up replication for the *_system* database. Note that it should be run on the slave and not the master: db._useDatabase("_system"); require("@arangodb/replication").setupReplication({ endpoint: "tcp://master.domain.org:8529", username: "myuser", password: "mypasswd", verbose: false, includeSystem: false, incremental: true, autoResync: true }); * the *sync* and *syncCollection* functions now always start the data synchronization as an asynchronous server job. The call to *sync* or *syncCollection* will block until synchronization is either complete or has failed with an error. The functions will automatically poll the slave periodically for status updates. The main benefit is that the connection to the slave does not need to stay open permanently and is thus not affected by timeout issues. Additionally the caller does not need to query the synchronization status from the slave manually as this is now performed automatically by these functions. * fixed undefined behavior when explaining some types of AQL traversals, fixed display of some types of traversals in AQL explain output v2.8.1 (2016-01-29) ------------------- * Improved AQL Pattern matching by allowing to specify a different traversal direction for one or many of the edge collections. FOR v, e, p IN OUTBOUND @start @@ec1, INBOUND @@ec2, @@ec3 will traverse *ec1* and *ec3* in the OUTBOUND direction and for *ec2* it will use the INBOUND direction. These directions can be combined in arbitrary ways, the direction defined after *IN [steps]* will we used as default direction and can be overriden for specific collections. This feature is only available for collection lists, it is not possible to combine it with graph names. * detect more types of transaction deadlocks early * fixed display of relational operators in traversal explain output * fixed undefined behavior in AQL function `PARSE_IDENTIFIER` * added "engines" field to Foxx services generated in the admin interface * added AQL function `IS_SAME_COLLECTION`: *IS_SAME_COLLECTION(collection, document)*: Return true if *document* has the same collection id as the collection specified in *collection*. *document* can either be a [document handle](../Glossary/README.md#document-handle) string, or a document with an *_id* attribute. The function does not validate whether the collection actually contains the specified document, but only compares the name of the specified collection with the collection name part of the specified document. If *document* is neither an object with an *id* attribute nor a *string* value, the function will return *null* and raise a warning. /* true */ IS_SAME_COLLECTION('_users', '_users/my-user') IS_SAME_COLLECTION('_users', { _id: '_users/my-user' }) /* false */ IS_SAME_COLLECTION('_users', 'foobar/baz') IS_SAME_COLLECTION('_users', { _id: 'something/else' }) v2.8.0 (2016-01-25) ------------------- * avoid recursive locking v2.8.0-beta8 (2016-01-19) ------------------------- * improved internal datafile statistics for compaction and compaction triggering conditions, preventing excessive growth of collection datafiles under some workloads. This should also fix issue #1596. * renamed AQL optimizer rule `remove-collect-into` to `remove-collect-variables` * fixed primary and edge index lookups prematurely aborting searches when the specified id search value contained a different collection than the collection the index was created for v2.8.0-beta7 (2016-01-06) ------------------------- * added vm.runInThisContext * added AQL keyword `AGGREGATE` for use in AQL `COLLECT` statement Using `AGGREGATE` allows more efficient aggregation (incrementally while building the groups) than previous versions of AQL, which built group aggregates afterwards from the total of all group values. `AGGREGATE` can be used inside a `COLLECT` statement only. If used, it must follow the declaration of grouping keys: FOR doc IN collection COLLECT gender = doc.gender AGGREGATE minAge = MIN(doc.age), maxAge = MAX(doc.age) RETURN { gender, minAge, maxAge } or, if no grouping keys are used, it can follow the `COLLECT` keyword: FOR doc IN collection COLLECT AGGREGATE minAge = MIN(doc.age), maxAge = MAX(doc.age) RETURN { minAge, maxAge } Only specific expressions are allowed on the right-hand side of each `AGGREGATE` assignment: - on the top level the expression must be a call to one of the supported aggregation functions `LENGTH`, `MIN`, `MAX`, `SUM`, `AVERAGE`, `STDDEV_POPULATION`, `STDDEV_SAMPLE`, `VARIANCE_POPULATION`, or `VARIANCE_SAMPLE` - the expression must not refer to variables introduced in the `COLLECT` itself * Foxx: mocha test paths with wildcard characters (asterisks) now work on Windows * reserved AQL keyword `NONE` for future use * web interface: fixed a graph display bug concerning dashboard view * web interface: fixed several bugs during the dashboard initialize process * web interface: included several bugfixes: #1597, #1611, #1623 * AQL query optimizer now converts `LENGTH(collection-name)` to an optimized expression that returns the number of documents in a collection * adjusted the behavior of the expansion (`[*]`) operator in AQL for non-array values In ArangoDB 2.8, calling the expansion operator on a non-array value will always return an empty array. Previous versions of ArangoDB expanded non-array values by calling the `TO_ARRAY()` function for the value, which for example returned an array with a single value for boolean, numeric and string input values, and an array with the object's values for an object input value. This behavior was inconsistent with how the expansion operator works for the array indexes in 2.8, so the behavior is now unified: - if the left-hand side operand of `[*]` is an array, the array will be returned as is when calling `[*]` on it - if the left-hand side operand of `[*]` is not an array, an empty array will be returned by `[*]` AQL queries that rely on the old behavior can be changed by either calling `TO_ARRAY` explicitly or by using the `[*]` at the correct position. The following example query will change its result in 2.8 compared to 2.7: LET values = "foo" RETURN values[*] In 2.7 the query has returned the array `[ "foo" ]`, but in 2.8 it will return an empty array `[ ]`. To make it return the array `[ "foo" ]` again, an explicit `TO_ARRAY` function call is needed in 2.8 (which in this case allows the removal of the `[*]` operator altogether). This also works in 2.7: LET values = "foo" RETURN TO_ARRAY(values) Another example: LET values = [ { name: "foo" }, { name: "bar" } ] RETURN values[*].name[*] The above returned `[ [ "foo" ], [ "bar" ] ] in 2.7. In 2.8 it will return `[ [ ], [ ] ]`, because the value of `name` is not an array. To change the results to the 2.7 style, the query can be changed to LET values = [ { name: "foo" }, { name: "bar" } ] RETURN values[* RETURN TO_ARRAY(CURRENT.name)] The above also works in 2.7. The following types of queries won't change: LET values = [ 1, 2, 3 ] RETURN values[*] LET values = [ { name: "foo" }, { name: "bar" } ] RETURN values[*].name LET values = [ { names: [ "foo", "bar" ] }, { names: [ "baz" ] } ] RETURN values[*].names[*] LET values = [ { names: [ "foo", "bar" ] }, { names: [ "baz" ] } ] RETURN values[*].names[**] * slightly adjusted V8 garbage collection strategy so that collection eventually happens in all contexts that hold V8 external references to documents and collections. also adjusted default value of `--javascript.gc-frequency` from 10 seconds to 15 seconds, as less internal operations are carried out in JavaScript. * fixes for AQL optimizer and traversal * added `--create-collection-type` option to arangoimp This allows specifying the type of the collection to be created when `--create-collection` is set to `true`. * Foxx export cache should no longer break if a broken app is loaded in the web admin interface. v2.8.0-beta2 (2015-12-16) ------------------------- * added AQL query optimizer rule "sort-in-values" This rule pre-sorts the right-hand side operand of the `IN` and `NOT IN` operators so the operation can use a binary search with logarithmic complexity instead of a linear search. The rule is applied when the right-hand side operand of an `IN` or `NOT IN` operator in a filter condition is a variable that is defined in a different loop/scope than the operator itself. Additionally, the filter condition must consist of solely the `IN` or `NOT IN` operation in order to avoid any side-effects. * changed collection status terminology in web interface for collections for which an unload request has been issued from `in the process of being unloaded` to `will be unloaded`. * unloading a collection via the web interface will now trigger garbage collection in all v8 contexts and force a WAL flush. This increases the chances of performing the unload faster. * added the following attributes to the result of `collection.figures()` and the corresponding HTTP API at `PUT /_api/collection//figures`: - `documentReferences`: The number of references to documents in datafiles that JavaScript code currently holds. This information can be used for debugging compaction and unload issues. - `waitingFor`: An optional string value that contains information about which object type is at the head of the collection's cleanup queue. This information can be used for debugging compaction and unload issues. - `compactionStatus.time`: The point in time the compaction for the collection was last executed. This information can be used for debugging compaction issues. - `compactionStatus.message`: The action that was performed when the compaction was last run for the collection. This information can be used for debugging compaction issues. Note: `waitingFor` and `compactionStatus` may be empty when called on a coordinator in a cluster. * the compaction will now provide queryable status info that can be used to track its progress. The compaction status is displayed in the web interface, too. * better error reporting for arangodump and arangorestore * arangodump will now fail by default when trying to dump edges that refer to already dropped collections. This can be circumvented by specifying the option `--force true` when invoking arangodump * fixed cluster upgrade procedure * the AQL functions `NEAR` and `WITHIN` now have stricter validations for their input parameters `limit`, `radius` and `distance`. They may now throw exceptions when invalid parameters are passed that may have not led to exceptions in previous versions. * deprecation warnings now log stack traces * Foxx: improved backwards compatibility with 2.5 and 2.6 - reverted Model and Repository back to non-ES6 "classes" because of compatibility issues when using the extend method with a constructor - removed deprecation warnings for extend and controller.del - restored deprecated method Model.toJSONSchema - restored deprecated `type`, `jwt` and `sessionStorageApp` options in Controller#activateSessions * Fixed a deadlock problem in the cluster v2.8.0-beta1 (2015-12-06) ------------------------- * added AQL function `IS_DATESTRING(value)` Returns true if *value* is a string that can be used in a date function. This includes partial dates such as *2015* or *2015-10* and strings containing invalid dates such as *2015-02-31*. The function will return false for all non-string values, even if some of them may be usable in date functions. v2.8.0-alpha1 (2015-12-03) -------------------------- * added AQL keywords `GRAPH`, `OUTBOUND`, `INBOUND` and `ANY` for use in graph traversals, reserved AQL keyword `ALL` for future use Usage of these keywords as collection names, variable names or attribute names in AQL queries will not be possible without quoting. For example, the following AQL query will still work as it uses a quoted collection name and a quoted attribute name: FOR doc IN `OUTBOUND` RETURN doc.`any` * issue #1593: added AQL `POW` function for exponentiation * added cluster execution site info in explain output for AQL queries * replication improvements: - added `autoResync` configuration parameter for continuous replication. When set to `true`, a replication slave will automatically trigger a full data re-synchronization with the master when the master cannot provide the log data the slave had asked for. Note that `autoResync` will only work when the option `requireFromPresent` is also set to `true` for the continuous replication, or when the continuous syncer is started and detects that no start tick is present. Automatic re-synchronization may transfer a lot of data from the master to the slave and may be expensive. It is therefore turned off by default. When turned off, the slave will never perform an automatic re-synchronization with the master. - added `idleMinWaitTime` and `idleMaxWaitTime` configuration parameters for continuous replication. These parameters can be used to control the minimum and maximum wait time the slave will (intentionally) idle and not poll for master log changes in case the master had sent the full logs already. The `idleMaxWaitTime` value will only be used when `adaptivePolling` is set to `true`. When `adaptivePolling` is disable, only `idleMinWaitTime` will be used as a constant time span in which the slave will not poll the master for further changes. The default values are 0.5 seconds for `idleMinWaitTime` and 2.5 seconds for `idleMaxWaitTime`, which correspond to the hard-coded values used in previous versions of ArangoDB. - added `initialSyncMaxWaitTime` configuration parameter for initial and continuous replication This option controls the maximum wait time (in seconds) that the initial synchronization will wait for a response from the master when fetching initial collection data. If no response is received within this time period, the initial synchronization will give up and fail. This option is also relevant for continuous replication in case *autoResync* is set to *true*, as then the continuous replication may trigger a full data re-synchronization in case the master cannot the log data the slave had asked for. - HTTP requests sent from the slave to the master during initial synchronization will now be retried if they fail with connection problems. - the initial synchronization now logs its progress so it can be queried using the regular replication status check APIs. - added `async` attribute for `sync` and `syncCollection` operations called from the ArangoShell. Setting this attribute to `true` will make the synchronization job on the server go into the background, so that the shell does not block. The status of the started asynchronous synchronization job can be queried from the ArangoShell like this: /* starts initial synchronization */ var replication = require("@arangodb/replication"); var id = replication.sync({ endpoint: "tcp://master.domain.org:8529", username: "myuser", password: "mypasswd", async: true }); /* now query the id of the returned async job and print the status */ print(replication.getSyncResult(id)); The result of `getSyncResult()` will be `false` while the server-side job has not completed, and different to `false` if it has completed. When it has completed, all job result details will be returned by the call to `getSyncResult()`. * fixed non-deterministic query results in some cluster queries * fixed issue #1589 * return HTTP status code 410 (gone) instead of HTTP 408 (request timeout) for server-side operations that are canceled / killed. Sending 410 instead of 408 prevents clients from re-starting the same (canceled) operation. Google Chrome for example sends the HTTP request again in case it is responded with an HTTP 408, and this is exactly the opposite of the desired behavior when an operation is canceled / killed by the user. * web interface: queries in AQL editor now cancelable * web interface: dashboard - added replication information * web interface: AQL editor now supports bind parameters * added startup option `--server.hide-product-header` to make the server not send the HTTP response header `"Server: ArangoDB"` in its HTTP responses. By default, the option is turned off so the header is still sent as usual. * added new AQL function `UNSET_RECURSIVE` to recursively unset attritutes from objects/documents * switched command-line editor in ArangoShell and arangod to linenoise-ng * added automatic deadlock detection for transactions In case a deadlock is detected, a multi-collection operation may be rolled back automatically and fail with error 29 (`deadlock detected`). Client code for operations containing more than one collection should be aware of this potential error and handle it accordingly, either by giving up or retrying the transaction. * Added C++ implementations for the AQL arithmetic operations and the following AQL functions: - ABS - APPEND - COLLECTIONS - CURRENT_DATABASE - DOCUMENT - EDGES - FIRST - FIRST_DOCUMENT - FIRST_LIST - FLATTEN - FLOOR - FULLTEXT - LAST - MEDIAN - MERGE_RECURSIVE - MINUS - NEAR - NOT_NULL - NTH - PARSE_IDENTIFIER - PERCENTILE - POP - POSITION - PUSH - RAND - RANGE - REMOVE_NTH - REMOVE_VALUE - REMOVE_VALUES - ROUND - SHIFT - SQRT - STDDEV_POPULATION - STDDEV_SAMPLE - UNSHIFT - VARIANCE_POPULATION - VARIANCE_SAMPLE - WITHIN - ZIP * improved performance of skipping over many documents in an AQL query when no indexes and no filters are used, e.g. FOR doc IN collection LIMIT 1000000, 10 RETURN doc * Added array indexes Hash indexes and skiplist indexes can now optionally be defined for array values so they index individual array members. To define an index for array values, the attribute name is extended with the expansion operator `[*]` in the index definition: arangosh> db.colName.ensureHashIndex("tags[*]"); When given the following document { tags: [ "AQL", "ArangoDB", "Index" ] } the index will now contain the individual values `"AQL"`, `"ArangoDB"` and `"Index"`. Now the index can be used for finding all documents having `"ArangoDB"` somewhere in their tags array using the following AQL query: FOR doc IN colName FILTER "ArangoDB" IN doc.tags[*] RETURN doc * rewrote AQL query optimizer rule `use-index-range` and renamed it to `use-indexes`. The name change affects rule names in the optimizer's output. * rewrote AQL execution node `IndexRangeNode` and renamed it to `IndexNode`. The name change affects node names in the optimizer's explain output. * added convenience function `db._explain(query)` for human-readable explanation of AQL queries * module resolution as used by `require` now behaves more like in node.js * the `org/arangodb/request` module now returns response bodies for error responses by default. The old behavior of not returning bodies for error responses can be re-enabled by explicitly setting the option `returnBodyOnError` to `false` (#1437) v2.7.6 (2016-01-30) ------------------- * detect more types of transaction deadlocks early v2.7.5 (2016-01-22) ------------------- * backported added automatic deadlock detection for transactions In case a deadlock is detected, a multi-collection operation may be rolled back automatically and fail with error 29 (`deadlock detected`). Client code for operations containing more than one collection should be aware of this potential error and handle it accordingly, either by giving up or retrying the transaction. * improved internal datafile statistics for compaction and compaction triggering conditions, preventing excessive growth of collection datafiles under some workloads. This should also fix issue #1596. * Foxx export cache should no longer break if a broken app is loaded in the web admin interface. * Foxx: removed some incorrect deprecation warnings. * Foxx: mocha test paths with wildcard characters (asterisks) now work on Windows v2.7.4 (2015-12-21) ------------------- * slightly adjusted V8 garbage collection strategy so that collection eventually happens in all contexts that hold V8 external references to documents and collections. * added the following attributes to the result of `collection.figures()` and the corresponding HTTP API at `PUT /_api/collection//figures`: - `documentReferences`: The number of references to documents in datafiles that JavaScript code currently holds. This information can be used for debugging compaction and unload issues. - `waitingFor`: An optional string value that contains information about which object type is at the head of the collection's cleanup queue. This information can be used for debugging compaction and unload issues. - `compactionStatus.time`: The point in time the compaction for the collection was last executed. This information can be used for debugging compaction issues. - `compactionStatus.message`: The action that was performed when the compaction was last run for the collection. This information can be used for debugging compaction issues. Note: `waitingFor` and `compactionStatus` may be empty when called on a coordinator in a cluster. * the compaction will now provide queryable status info that can be used to track its progress. The compaction status is displayed in the web interface, too. v2.7.3 (2015-12-17) ------------------- * fixed some replication value conversion issues when replication applier properties were set via ArangoShell * fixed disappearing of documents for collections transferred via `sync` or `syncCollection` if the collection was dropped right before synchronization and drop and (re-)create collection markers were located in the same WAL file * fixed an issue where overwriting the system sessions collection would break the web interface when authentication is enabled v2.7.2 (2015-12-01) ------------------- * replication improvements: - added `autoResync` configuration parameter for continuous replication. When set to `true`, a replication slave will automatically trigger a full data re-synchronization with the master when the master cannot provide the log data the slave had asked for. Note that `autoResync` will only work when the option `requireFromPresent` is also set to `true` for the continuous replication, or when the continuous syncer is started and detects that no start tick is present. Automatic re-synchronization may transfer a lot of data from the master to the slave and may be expensive. It is therefore turned off by default. When turned off, the slave will never perform an automatic re-synchronization with the master. - added `idleMinWaitTime` and `idleMaxWaitTime` configuration parameters for continuous replication. These parameters can be used to control the minimum and maximum wait time the slave will (intentionally) idle and not poll for master log changes in case the master had sent the full logs already. The `idleMaxWaitTime` value will only be used when `adaptivePolling` is set to `true`. When `adaptivePolling` is disable, only `idleMinWaitTime` will be used as a constant time span in which the slave will not poll the master for further changes. The default values are 0.5 seconds for `idleMinWaitTime` and 2.5 seconds for `idleMaxWaitTime`, which correspond to the hard-coded values used in previous versions of ArangoDB. - added `initialSyncMaxWaitTime` configuration parameter for initial and continuous replication This option controls the maximum wait time (in seconds) that the initial synchronization will wait for a response from the master when fetching initial collection data. If no response is received within this time period, the initial synchronization will give up and fail. This option is also relevant for continuous replication in case *autoResync* is set to *true*, as then the continuous replication may trigger a full data re-synchronization in case the master cannot the log data the slave had asked for. - HTTP requests sent from the slave to the master during initial synchronization will now be retried if they fail with connection problems. - the initial synchronization now logs its progress so it can be queried using the regular replication status check APIs. * fixed non-deterministic query results in some cluster queries * added missing lock instruction for primary index in compactor size calculation * fixed issue #1589 * fixed issue #1583 * fixed undefined behavior when accessing the top level of a document with the `[*]` operator * fixed potentially invalid pointer access in shaper when the currently accessed document got re-located by the WAL collector at the very same time * Foxx: optional configuration options no longer log validation errors when assigned empty values (#1495) * Foxx: constructors provided to Repository and Model sub-classes via extend are now correctly called (#1592) v2.7.1 (2015-11-07) ------------------- * switch to linenoise next generation * exclude `_apps` collection from replication The slave has its own `_apps` collection which it populates on server start. When replicating data from the master to the slave, the data from the master may clash with the slave's own data in the `_apps` collection. Excluding the `_apps` collection from replication avoids this. * disable replication appliers when starting in modes `--upgrade`, `--no-server` and `--check-upgrade` * more detailed output in arango-dfdb * fixed "no start tick" issue in replication applier This error could occur after restarting a slave server after a shutdown when no data was ever transferred from the master to the slave via the continuous replication * fixed problem during SSL client connection abort that led to scheduler thread staying at 100% CPU saturation * fixed potential segfault in AQL `NEIGHBORS` function implementation when C++ function variant was used and collection names were passed as strings * removed duplicate target for some frontend JavaScript files from the Makefile * make AQL function `MERGE()` work on a single array parameter, too. This allows combining the attributes of multiple objects from an array into a single object, e.g. RETURN MERGE([ { foo: 'bar' }, { quux: 'quetzalcoatl', ruled: true }, { bar: 'baz', foo: 'done' } ]) will now return: { "foo": "done", "quux": "quetzalcoatl", "ruled": true, "bar": "baz" } * fixed potential deadlock in collection status changing on Windows * fixed hard-coded `incremental` parameter in shell implementation of `syncCollection` function in replication module * fix for GCC5: added check for '-stdlib' option v2.7.0 (2015-10-09) ------------------- * fixed request statistics aggregation When arangod was started in supervisor mode, the request statistics always showed 0 requests, as the statistics aggregation thread did not run then. * read server configuration files before dropping privileges. this ensures that the SSL keyfile specified in the configuration can be read with the server's start privileges (i.e. root when using a standard ArangoDB package). * fixed replication with a 2.6 replication configuration and issues with a 2.6 master * raised default value of `--server.descriptors-minimum` to 1024 * allow Foxx apps to be installed underneath URL path `/_open/`, so they can be (intentionally) accessed without authentication. * added *allowImplicit* sub-attribute in collections declaration of transactions. The *allowImplicit* attributes allows making transactions fail should they read-access a collection that was not explicitly declared in the *collections* array of the transaction. * added "special" password ARANGODB_DEFAULT_ROOT_PASSWORD. If you pass ARANGODB_DEFAULT_ROOT_PASSWORD as password, it will read the password from the environment variable ARANGODB_DEFAULT_ROOT_PASSWORD v2.7.0-rc2 (2015-09-22) ----------------------- * fix over-eager datafile compaction This should reduce the need to compact directly after loading a collection when a collection datafile contained many insertions and updates for the same documents. It should also prevent from re-compacting already merged datafiles in case not many changes were made. Compaction will also make fewer index lookups than before. * added `syncCollection()` function in module `org/arangodb/replication` This allows synchronizing the data of a single collection from a master to a slave server. Synchronization can either restore the whole collection by transferring all documents from the master to the slave, or incrementally by only transferring documents that differ. This is done by partitioning the collection's entire key space into smaller chunks and comparing the data chunk-wise between master and slave. Only chunks that are different will be re-transferred. The `syncCollection()` function can be used as follows: require("org/arangodb/replication").syncCollection(collectionName, options); e.g. require("org/arangodb/replication").syncCollection("myCollection", { endpoint: "tcp://127.0.0.1:8529", /* master */ username: "root", /* username for master */ password: "secret", /* password for master */ incremental: true /* use incremental mode */ }); * additionally allow the following characters in document keys: `(` `)` `+` `,` `=` `;` `$` `!` `*` `'` `%` v2.7.0-rc1 (2015-09-17) ----------------------- * removed undocumented server-side-only collection functions: * collection.OFFSET() * collection.NTH() * collection.NTH2() * collection.NTH3() * upgraded Swagger to version 2.0 for the Documentation This gives the user better prepared test request structures. More conversions will follow so finally client libraries can be auto-generated. * added extra AQL functions for date and time calculation and manipulation. These functions were contributed by GitHub users @CoDEmanX and @friday. A big thanks for their work! The following extra date functions are available from 2.7 on: * `DATE_DAYOFYEAR(date)`: Returns the day of year number of *date*. The return values range from 1 to 365, or 366 in a leap year respectively. * `DATE_ISOWEEK(date)`: Returns the ISO week date of *date*. The return values range from 1 to 53. Monday is considered the first day of the week. There are no fractional weeks, thus the last days in December may belong to the first week of the next year, and the first days in January may be part of the previous year's last week. * `DATE_LEAPYEAR(date)`: Returns whether the year of *date* is a leap year. * `DATE_QUARTER(date)`: Returns the quarter of the given date (1-based): * 1: January, February, March * 2: April, May, June * 3: July, August, September * 4: October, November, December - *DATE_DAYS_IN_MONTH(date)*: Returns the number of days in *date*'s month (28..31). * `DATE_ADD(date, amount, unit)`: Adds *amount* given in *unit* to *date* and returns the calculated date. *unit* can be either of the following to specify the time unit to add or subtract (case-insensitive): - y, year, years - m, month, months - w, week, weeks - d, day, days - h, hour, hours - i, minute, minutes - s, second, seconds - f, millisecond, milliseconds *amount* is the number of *unit*s to add (positive value) or subtract (negative value). * `DATE_SUBTRACT(date, amount, unit)`: Subtracts *amount* given in *unit* from *date* and returns the calculated date. It works the same as `DATE_ADD()`, except that it subtracts. It is equivalent to calling `DATE_ADD()` with a negative amount, except that `DATE_SUBTRACT()` can also subtract ISO durations. Note that negative ISO durations are not supported (i.e. starting with `-P`, like `-P1Y`). * `DATE_DIFF(date1, date2, unit, asFloat)`: Calculate the difference between two dates in given time *unit*, optionally with decimal places. Returns a negative value if *date1* is greater than *date2*. * `DATE_COMPARE(date1, date2, unitRangeStart, unitRangeEnd)`: Compare two partial dates and return true if they match, false otherwise. The parts to compare are defined by a range of time units. The full range is: years, months, days, hours, minutes, seconds, milliseconds. Pass the unit to start from as *unitRangeStart*, and the unit to end with as *unitRangeEnd*. All units in between will be compared. Leave out *unitRangeEnd* to only compare *unitRangeStart*. * `DATE_FORMAT(date, format)`: Format a date according to the given format string. It supports the following placeholders (case-insensitive): - %t: timestamp, in milliseconds since midnight 1970-01-01 - %z: ISO date (0000-00-00T00:00:00.000Z) - %w: day of week (0..6) - %y: year (0..9999) - %yy: year (00..99), abbreviated (last two digits) - %yyyy: year (0000..9999), padded to length of 4 - %yyyyyy: year (-009999 .. +009999), with sign prefix and padded to length of 6 - %m: month (1..12) - %mm: month (01..12), padded to length of 2 - %d: day (1..31) - %dd: day (01..31), padded to length of 2 - %h: hour (0..23) - %hh: hour (00..23), padded to length of 2 - %i: minute (0..59) - %ii: minute (00..59), padded to length of 2 - %s: second (0..59) - %ss: second (00..59), padded to length of 2 - %f: millisecond (0..999) - %fff: millisecond (000..999), padded to length of 3 - %x: day of year (1..366) - %xxx: day of year (001..366), padded to length of 3 - %k: ISO week date (1..53) - %kk: ISO week date (01..53), padded to length of 2 - %l: leap year (0 or 1) - %q: quarter (1..4) - %a: days in month (28..31) - %mmm: abbreviated English name of month (Jan..Dec) - %mmmm: English name of month (January..December) - %www: abbreviated English name of weekday (Sun..Sat) - %wwww: English name of weekday (Sunday..Saturday) - %&: special escape sequence for rare occasions - %%: literal % - %: ignored * new WAL logfiles and datafiles are now created non-sparse This prevents SIGBUS signals being raised when memory of a sparse datafile is accessed and the disk is full and the accessed file part is not actually disk-backed. In this case the mapped memory region is not necessarily backed by physical memory, and accessing the memory may raise SIGBUS and crash arangod. * the `internal.download()` function and the module `org/arangodb/request` used some internal library function that handled the sending of HTTP requests from inside of ArangoDB. This library unconditionally set an HTTP header `Accept-Encoding: gzip` in all outgoing HTTP requests. This has been fixed in 2.7, so `Accept-Encoding: gzip` is not set automatically anymore. Additionally, the header `User-Agent: ArangoDB` is not set automatically either. If client applications desire to send these headers, they are free to add it when constructing the requests using the `download` function or the request module. * fixed issue #1436: org/arangodb/request advertises deflate without supporting it * added template string generator function `aqlQuery` for generating AQL queries This can be used to generate safe AQL queries with JavaScript parameter variables or expressions easily: var name = 'test'; var attributeName = '_key'; var query = aqlQuery`FOR u IN users FILTER u.name == ${name} RETURN u.${attributeName}`; db._query(query); * report memory usage for document header data (revision id, pointer to data etc.) in `db.collection.figures()`. The memory used for document headers will now show up in the already existing attribute `indexes.size`. Due to that, the index sizes reported by `figures()` in 2.7 will be higher than those reported by 2.6, but the 2.7 values are more accurate. * IMPORTANT CHANGE: the filenames in dumps created by arangodump now contain not only the name of the dumped collection, but also an additional 32-digit hash value. This is done to prevent overwriting dump files in case-insensitive file systems when there exist multiple collections with the same name (but with different cases). For example, if a database has two collections: `test` and `Test`, previous versions of ArangoDB created the files * `test.structure.json` and `test.data.json` for collection `test` * `Test.structure.json` and `Test.data.json` for collection `Test` This did not work for case-insensitive filesystems, because the files for the second collection would have overwritten the files of the first. arangodump in 2.7 will create the following filenames instead: * `test_098f6bcd4621d373cade4e832627b4f6.structure.json` and `test_098f6bcd4621d373cade4e832627b4f6.data.json` * `Test_0cbc6611f5540bd0809a388dc95a615b.structure.json` and `Test_0cbc6611f5540bd0809a388dc95a615b.data.json` These filenames will be unambiguous even in case-insensitive filesystems. * IMPORTANT CHANGE: make arangod actually close lingering client connections when idle for at least the duration specified via `--server.keep-alive-timeout`. In previous versions of ArangoDB, connections were not closed by the server when the timeout was reached and the client was still connected. Now the connection is properly closed by the server in case of timeout. Client applications relying on the old behavior may now need to reconnect to the server when their idle connections time out and get closed (note: connections being idle for a long time may be closed by the OS or firewalls anyway - client applications should be aware of that and try to reconnect). * IMPORTANT CHANGE: when starting arangod, the server will drop the process privileges to the specified values in options `--server.uid` and `--server.gid` instantly after parsing the startup options. That means when either `--server.uid` or `--server.gid` are set, the privilege change will happen earlier. This may prevent binding the server to an endpoint with a port number lower than 1024 if the arangodb user has no privileges for that. Previous versions of ArangoDB changed the privileges later, so some startup actions were still carried out under the invoking user (i.e. likely *root* when started via init.d or system scripts) and especially binding to low port numbers was still possible there. The default privileges for user *arangodb* will not be sufficient for binding to port numbers lower than 1024. To have an ArangoDB 2.7 bind to a port number lower than 1024, it needs to be started with either a different privileged user, or the privileges of the *arangodb* user have to raised manually beforehand. * added AQL optimizer rule `patch-update-statements` * Linux startup scripts and systemd configuration for arangod now try to adjust the NOFILE (number of open files) limits for the process. The limit value is set to 131072 (128k) when ArangoDB is started via start/stop commands * When ArangoDB is started/stopped manually via the start/stop commands, the main process will wait for up to 10 seconds after it forks the supervisor and arangod child processes. If the startup fails within that period, the start/stop script will fail with an exit code other than zero. If the startup of the supervisor or arangod is still ongoing after 10 seconds, the main program will still return with exit code 0. The limit of 10 seconds is arbitrary because the time required for a startup is not known in advance. * added startup option `--database.throw-collection-not-loaded-error` Accessing a not-yet loaded collection will automatically load a collection on first access. This flag controls what happens in case an operation would need to wait for another thread to finalize loading a collection. If set to *true*, then the first operation that accesses an unloaded collection will load it. Further threads that try to access the same collection while it is still loading immediately fail with an error (1238, *collection not loaded*). This is to prevent all server threads from being blocked while waiting on the same collection to finish loading. When the first thread has completed loading the collection, the collection becomes regularly available, and all operations from that point on can be carried out normally, and error 1238 will not be thrown anymore for that collection. If set to *false*, the first thread that accesses a not-yet loaded collection will still load it. Other threads that try to access the collection while loading will not fail with error 1238 but instead block until the collection is fully loaded. This configuration might lead to all server threads being blocked because they are all waiting for the same collection to complete loading. Setting the option to *true* will prevent this from happening, but requires clients to catch error 1238 and react on it (maybe by scheduling a retry for later). The default value is *false*. * added better control-C support in arangosh When CTRL-C is pressed in arangosh, it will now print a `^C` first. Pressing CTRL-C again will reset the prompt if something was entered before, or quit arangosh if no command was entered directly before. This affects the arangosh version build with Readline-support only (Linux and MacOS). The MacOS version of ArangoDB for Homebrew now depends on Readline, too. The Homebrew formula has been changed accordingly. When self-compiling ArangoDB on MacOS without Homebrew, Readline now is a prerequisite. * increased default value for collection-specific `indexBuckets` value from 1 to 8 Collections created from 2.7 on will use the new default value of `8` if not overridden on collection creation or later using `collection.properties({ indexBuckets: ... })`. The `indexBuckets` value determines the number of buckets to use for indexes of type `primary`, `hash` and `edge`. Having multiple index buckets allows splitting an index into smaller components, which can be filled in parallel when a collection is loading. Additionally, resizing and reallocation of indexes are faster and less intrusive if the index uses multiple buckets, because resize and reallocation will affect only data in a single bucket instead of all index values. The index buckets will be filled in parallel when loading a collection if the collection has an `indexBuckets` value greater than 1 and the collection contains a significant amount of documents/edges (the current threshold is 256K documents but this value may change in future versions of ArangoDB). * changed HTTP client to use poll instead of select on Linux and MacOS This affects the ArangoShell and user-defined JavaScript code running inside arangod that initiates its own HTTP calls. Using poll instead of select allows using arbitrary high file descriptors (bigger than the compiled in FD_SETSIZE). Server connections are still handled using epoll, which has never been affected by FD_SETSIZE. * implemented AQL `LIKE` function using ICU regexes * added `RETURN DISTINCT` for AQL queries to return unique results: FOR doc IN collection RETURN DISTINCT doc.status This change also introduces `DISTINCT` as an AQL keyword. * removed `createNamedQueue()` and `addJob()` functions from org/arangodb/tasks * use less locks and more atomic variables in the internal dispatcher and V8 context handling implementations. This leads to improved throughput in some ArangoDB internals and allows for higher HTTP request throughput for many operations. A short overview of the improvements can be found here: https://www.arangodb.com/2015/08/throughput-enhancements/ * added shorthand notation for attribute names in AQL object literals: LET name = "Peter" LET age = 42 RETURN { name, age } The above is the shorthand equivalent of the generic form LET name = "Peter" LET age = 42 RETURN { name : name, age : age } * removed configure option `--enable-timings` This option did not have any effect. * removed configure option `--enable-figures` This option previously controlled whether HTTP request statistics code was compiled into ArangoDB or not. The previous default value was `true` so statistics code was available in official packages. Setting the option to `false` led to compile errors so it is doubtful the default value was ever changed. By removing the option some internal statistics code was also simplified. * removed run-time manipulation methods for server endpoints: * `db._removeEndpoint()` * `db._configureEndpoint()` * HTTP POST `/_api/endpoint` * HTTP DELETE `/_api/endpoint` * AQL query result cache The query result cache can optionally cache the complete results of all or selected AQL queries. It can be operated in the following modes: * `off`: the cache is disabled. No query results will be stored * `on`: the cache will store the results of all AQL queries unless their `cache` attribute flag is set to `false` * `demand`: the cache will store the results of AQL queries that have their `cache` attribute set to `true`, but will ignore all others The mode can be set at server startup using the `--database.query-cache-mode` configuration option and later changed at runtime. The following HTTP REST APIs have been added for controlling the query cache: * HTTP GET `/_api/query-cache/properties`: returns the global query cache configuration * HTTP PUT `/_api/query-cache/properties`: modifies the global query cache configuration * HTTP DELETE `/_api/query-cache`: invalidates all results in the query cache The following JavaScript functions have been added for controlling the query cache: * `require("org/arangodb/aql/cache").properties()`: returns the global query cache configuration * `require("org/arangodb/aql/cache").properties(properties)`: modifies the global query cache configuration * `require("org/arangodb/aql/cache").clear()`: invalidates all results in the query cache * do not link arangoimp against V8 * AQL function call arguments optimization This will lead to arguments in function calls inside AQL queries not being copied but passed by reference. This may speed up calls to functions with bigger argument values or queries that call functions a lot of times. * upgraded V8 version to 4.3.61 * removed deprecated AQL `SKIPLIST` function. This function was introduced in older versions of ArangoDB with a less powerful query optimizer to retrieve data from a skiplist index using a `LIMIT` clause. It was marked as deprecated in ArangoDB 2.6. Since ArangoDB 2.3 the behavior of the `SKIPLIST` function can be emulated using regular AQL constructs, e.g. FOR doc IN @@collection FILTER doc.value >= @value SORT doc.value DESC LIMIT 1 RETURN doc * the `skip()` function for simple queries does not accept negative input any longer. This feature was deprecated in 2.6.0. * fix exception handling In some cases JavaScript exceptions would re-throw without information of the original problem. Now the original exception is logged for failure analysis. * based REST API method PUT `/_api/simple/all` on the cursor API and make it use AQL internally. The change speeds up this REST API method and will lead to additional query information being returned by the REST API. Clients can use this extra information or ignore it. * Foxx Queue job success/failure handlers arguments have changed from `(jobId, jobData, result, jobFailures)` to `(result, jobData, job)`. * added Foxx Queue job options `repeatTimes`, `repeatUntil` and `repeatDelay` to automatically re-schedule jobs when they are completed. * added Foxx manifest configuration type `password` to mask values in the web interface. * fixed default values in Foxx manifest configurations sometimes not being used as defaults. * fixed optional parameters in Foxx manifest configurations sometimes not being cleared correctly. * Foxx dependencies can now be marked as optional using a slightly more verbose syntax in your manifest file. * converted Foxx constructors to ES6 classes so you can extend them using class syntax. * updated aqb to 2.0. * updated chai to 3.0. * Use more madvise calls to speed up things when memory is tight, in particular at load time but also for random accesses later. * Overhauled web interface The web interface now has a new design. The API documentation for ArangoDB has been moved from "Tools" to "Links" in the web interface. The "Applications" tab in the web interfaces has been renamed to "Services". v2.6.12 (2015-12-02) -------------------- * fixed disappearing of documents for collections transferred via `sync` if the the collection was dropped right before synchronization and drop and (re-)create collection markers were located in the same WAL file * added missing lock instruction for primary index in compactor size calculation * fixed issue #1589 * fixed issue #1583 * Foxx: optional configuration options no longer log validation errors when assigned empty values (#1495) v2.6.11 (2015-11-18) -------------------- * fixed potentially invalid pointer access in shaper when the currently accessed document got re-located by the WAL collector at the very same time v2.6.10 (2015-11-10) -------------------- * disable replication appliers when starting in modes `--upgrade`, `--no-server` and `--check-upgrade` * more detailed output in arango-dfdb * fixed potential deadlock in collection status changing on Windows * issue #1521: Can't dump/restore with user and password v2.6.9 (2015-09-29) ------------------- * added "special" password ARANGODB_DEFAULT_ROOT_PASSWORD. If you pass ARANGODB_DEFAULT_ROOT_PASSWORD as password, it will read the password from the environment variable ARANGODB_DEFAULT_ROOT_PASSWORD * fixed failing AQL skiplist, sort and limit combination When using a Skiplist index on an attribute (say "a") and then using sort and skip on this attribute caused the result to be empty e.g.: require("internal").db.test.ensureSkiplist("a"); require("internal").db._query("FOR x IN test SORT x.a LIMIT 10, 10"); Was always empty no matter how many documents are stored in test. This is now fixed. v2.6.8 (2015-09-09) ------------------- * ARM only: The ArangoDB packages for ARM require the kernel to allow unaligned memory access. How the kernel handles unaligned memory access is configurable at runtime by checking and adjusting the contents `/proc/cpu/alignment`. In order to operate on ARM, ArangoDB requires the bit 1 to be set. This will make the kernel trap and adjust unaligned memory accesses. If this bit is not set, the kernel may send a SIGBUS signal to ArangoDB and terminate it. To set bit 1 in `/proc/cpu/alignment` use the following command as a privileged user (e.g. root): echo "2" > /proc/cpu/alignment Note that this setting affects all user processes and not just ArangoDB. Setting the alignment with the above command will also not make the setting permanent, so it will be lost after a restart of the system. In order to make the setting permanent, it should be executed during system startup or before starting arangod. The ArangoDB start/stop scripts do not adjust the alignment setting, but rely on the environment to have the correct alignment setting already. The reason for this is that the alignment settings also affect all other user processes (which ArangoDB is not aware of) and thus may have side-effects outside of ArangoDB. It is therefore more reasonable to have the system administrator carry out the change. v2.6.7 (2015-08-25) ------------------- * improved AssocMulti index performance when resizing. This makes the edge index perform less I/O when under memory pressure. v2.6.6 (2015-08-23) ------------------- * added startup option `--server.additional-threads` to create separate queues for slow requests. v2.6.5 (2015-08-17) ------------------- * added startup option `--database.throw-collection-not-loaded-error` Accessing a not-yet loaded collection will automatically load a collection on first access. This flag controls what happens in case an operation would need to wait for another thread to finalize loading a collection. If set to *true*, then the first operation that accesses an unloaded collection will load it. Further threads that try to access the same collection while it is still loading immediately fail with an error (1238, *collection not loaded*). This is to prevent all server threads from being blocked while waiting on the same collection to finish loading. When the first thread has completed loading the collection, the collection becomes regularly available, and all operations from that point on can be carried out normally, and error 1238 will not be thrown anymore for that collection. If set to *false*, the first thread that accesses a not-yet loaded collection will still load it. Other threads that try to access the collection while loading will not fail with error 1238 but instead block until the collection is fully loaded. This configuration might lead to all server threads being blocked because they are all waiting for the same collection to complete loading. Setting the option to *true* will prevent this from happening, but requires clients to catch error 1238 and react on it (maybe by scheduling a retry for later). The default value is *false*. * fixed busy wait loop in scheduler threads that sometimes consumed 100% CPU while waiting for events on connections closed unexpectedly by the client side * handle attribute `indexBuckets` when restoring collections via arangorestore. Previously the `indexBuckets` attribute value from the dump was ignored, and the server default value for `indexBuckets` was used when restoring a collection. * fixed "EscapeValue already set error" crash in V8 actions that might have occurred when canceling V8-based operations. v2.6.4 (2015-08-01) ------------------- * V8: Upgrade to version 4.1.0.27 - this is intended to be the stable V8 version. * fixed issue #1424: Arango shell should not processing arrows pushing on keyboard v2.6.3 (2015-07-21) ------------------- * issue #1409: Document values with null character truncated v2.6.2 (2015-07-04) ------------------- * fixed issue #1383: bindVars for HTTP API doesn't work with empty string * fixed handling of default values in Foxx manifest configurations * fixed handling of optional parameters in Foxx manifest configurations * fixed a reference error being thrown in Foxx queues when a function-based job type is used that is not available and no options object is passed to queue.push v2.6.1 (2015-06-24) ------------------- * Add missing swagger files to cmake build. fixes #1368 * fixed documentation errors v2.6.0 (2015-06-20) ------------------- * using negative values for `SimpleQuery.skip()` is deprecated. This functionality will be removed in future versions of ArangoDB. * The following simple query functions are now deprecated: * collection.near * collection.within * collection.geo * collection.fulltext * collection.range * collection.closedRange This also lead to the following REST API methods being deprecated from now on: * PUT /_api/simple/near * PUT /_api/simple/within * PUT /_api/simple/fulltext * PUT /_api/simple/range It is recommended to replace calls to these functions or APIs with equivalent AQL queries, which are more flexible because they can be combined with other operations: FOR doc IN NEAR(@@collection, @latitude, @longitude, @limit) RETURN doc FOR doc IN WITHIN(@@collection, @latitude, @longitude, @radius, @distanceAttributeName) RETURN doc FOR doc IN FULLTEXT(@@collection, @attributeName, @queryString, @limit) RETURN doc FOR doc IN @@collection FILTER doc.value >= @left && doc.value < @right LIMIT @skip, @limit RETURN doc` The above simple query functions and REST API methods may be removed in future versions of ArangoDB. * deprecated now-obsolete AQL `SKIPLIST` function The function was introduced in older versions of ArangoDB with a less powerful query optimizer to retrieve data from a skiplist index using a `LIMIT` clause. Since 2.3 the same goal can be achieved by using regular AQL constructs, e.g. FOR doc IN collection FILTER doc.value >= @value SORT doc.value DESC LIMIT 1 RETURN doc * fixed issues when switching the database inside tasks and during shutdown of database cursors These features were added during 2.6 alpha stage so the fixes affect devel/2.6-alpha builds only * issue #1360: improved foxx-manager help * added `--enable-tcmalloc` configure option. When this option is set, arangod and the client tools will be linked against tcmalloc, which replaces the system allocator. When the option is set, a tcmalloc library must be present on the system under one of the names `libtcmalloc`, `libtcmalloc_minimal` or `libtcmalloc_debug`. As this is a configure option, it is supported for manual builds on Linux-like systems only. tcmalloc support is currently experimental. * issue #1353: Windows: HTTP API - incorrect path in errorMessage * issue #1347: added option `--create-database` for arangorestore. Setting this option to `true` will now create the target database if it does not exist. When creating the target database, the username and passwords passed to arangorestore will be used to create an initial user for the new database. * issue #1345: advanced debug information for User Functions * issue #1341: Can't use bindvars in UPSERT * fixed vulnerability in JWT implementation. * changed default value of option `--database.ignore-datafile-errors` from `true` to `false` If the new default value of `false` is used, then arangod will refuse loading collections that contain datafiles with CRC mismatches or other errors. A collection with datafile errors will then become unavailable. This prevents follow up errors from happening. The only way to access such collection is to use the datafile debugger (arango-dfdb) and try to repair or truncate the datafile with it. If `--database.ignore-datafile-errors` is set to `true`, then collections will become available even if parts of their data cannot be loaded. This helps availability, but may cause (partial) data loss and follow up errors. * added server startup option `--server.session-timeout` for controlling the timeout of user sessions in the web interface * add sessions and cookie authentication for ArangoDB's web interface ArangoDB's built-in web interface now uses sessions. Session information ids are stored in cookies, so clients using the web interface must accept cookies in order to use it * web interface: display query execution time in AQL editor * web interface: renamed AQL query *submit* button to *execute* * web interface: added query explain feature in AQL editor * web interface: demo page added. only working if demo data is available, hidden otherwise * web interface: added support for custom app scripts with optional arguments and results * web interface: mounted apps that need to be configured are now indicated in the app overview * web interface: added button for running tests to app details * web interface: added button for configuring app dependencies to app details * web interface: upgraded API documentation to use Swagger 2 * INCOMPATIBLE CHANGE removed startup option `--log.severity` The docs for `--log.severity` mentioned lots of severities (e.g. `exception`, `technical`, `functional`, `development`) but only a few severities (e.g. `all`, `human`) were actually used, with `human` being the default and `all` enabling the additional logging of requests. So the option pretended to control a lot of things which it actually didn't. Additionally, the option `--log.requests-file` was around for a long time already, also controlling request logging. Because the `--log.severity` option effectively did not control that much, it was removed. A side effect of removing the option is that 2.5 installations which used `--log.severity all` will not log requests after the upgrade to 2.6. This can be adjusted by setting the `--log.requests-file` option. * add backtrace to fatal log events * added optional `limit` parameter for AQL function `FULLTEXT` * make fulltext index also index text values contained in direct sub-objects of the indexed attribute. Previous versions of ArangoDB only indexed the attribute value if it was a string. Sub-attributes of the index attribute were ignored when fulltext indexing. Now, if the index attribute value is an object, the object's values will each be included in the fulltext index if they are strings. If the index attribute value is an array, the array's values will each be included in the fulltext index if they are strings. For example, with a fulltext index present on the `translations` attribute, the following text values will now be indexed: var c = db._create("example"); c.ensureFulltextIndex("translations"); c.insert({ translations: { en: "fox", de: "Fuchs", fr: "renard", ru: "лиса" } }); c.insert({ translations: "Fox is the English translation of the German word Fuchs" }); c.insert({ translations: [ "ArangoDB", "document", "database", "Foxx" ] }); c.fulltext("translations", "лиса").toArray(); // returns only first document c.fulltext("translations", "Fox").toArray(); // returns first and second documents c.fulltext("translations", "prefix:Fox").toArray(); // returns all three documents * added batch document removal and lookup commands: collection.lookupByKeys(keys) collection.removeByKeys(keys) These commands can be used to perform multi-document lookup and removal operations efficiently from the ArangoShell. The argument to these operations is an array of document keys. Also added HTTP APIs for batch document commands: * PUT /_api/simple/lookup-by-keys * PUT /_api/simple/remove-by-keys * properly prefix document address URLs with the current database name for calls to the REST API method GET `/_api/document?collection=...` (that method will return partial URLs to all documents in the collection). Previous versions of ArangoDB returned the URLs starting with `/_api/` but without the current database name, e.g. `/_api/document/mycollection/mykey`. Starting with 2.6, the response URLs will include the database name as well, e.g. `/_db/_system/_api/document/mycollection/mykey`. * added dedicated collection export HTTP REST API ArangoDB now provides a dedicated collection export API, which can take snapshots of entire collections more efficiently than the general-purpose cursor API. The export API is useful to transfer the contents of an entire collection to a client application. It provides optional filtering on specific attributes. The export API is available at endpoint `POST /_api/export?collection=...`. The API has the same return value structure as the already established cursor API (`POST /_api/cursor`). An introduction to the export API is given in this blog post: http://jsteemann.github.io/blog/2015/04/04/more-efficient-data-exports/ * subquery optimizations for AQL queries This optimization avoids copying intermediate results into subqueries that are not required by the subquery. A brief description can be found here: http://jsteemann.github.io/blog/2015/05/04/subquery-optimizations/ * return value optimization for AQL queries This optimization avoids copying the final query result inside the query's main `ReturnNode`. A brief description can be found here: http://jsteemann.github.io/blog/2015/05/04/return-value-optimization-for-aql/ * speed up AQL queries containing big `IN` lists for index lookups `IN` lists used for index lookups had performance issues in previous versions of ArangoDB. These issues have been addressed in 2.6 so using bigger `IN` lists for filtering is much faster. A brief description can be found here: http://jsteemann.github.io/blog/2015/05/07/in-list-improvements/ * allow `@` and `.` characters in document keys, too This change also leads to document keys being URL-encoded when returned in HTTP `location` response headers. * added alternative implementation for AQL COLLECT The alternative method uses a hash table for grouping and does not require its input elements to be sorted. It will be taken into account by the optimizer for `COLLECT` statements that do not use an `INTO` clause. In case a `COLLECT` statement can use the hash table variant, the optimizer will create an extra plan for it at the beginning of the planning phase. In this plan, no extra `SORT` node will be added in front of the `COLLECT` because the hash table variant of `COLLECT` does not require sorted input. Instead, a `SORT` node will be added after it to sort its output. This `SORT` node may be optimized away again in later stages. If the sort order of the result is irrelevant to the user, adding an extra `SORT null` after a hash `COLLECT` operation will allow the optimizer to remove the sorts altogether. In addition to the hash table variant of `COLLECT`, the optimizer will modify the original plan to use the regular `COLLECT` implementation. As this implementation requires sorted input, the optimizer will insert a `SORT` node in front of the `COLLECT`. This `SORT` node may be optimized away in later stages. The created plans will then be shipped through the regular optimization pipeline. In the end, the optimizer will pick the plan with the lowest estimated total cost as usual. The hash table variant does not require an up-front sort of the input, and will thus be preferred over the regular `COLLECT` if the optimizer estimates many input elements for the `COLLECT` node and cannot use an index to sort them. The optimizer can be explicitly told to use the regular *sorted* variant of `COLLECT` by suffixing a `COLLECT` statement with `OPTIONS { "method" : "sorted" }`. This will override the optimizer guesswork and only produce the *sorted* variant of `COLLECT`. A blog post on the new `COLLECT` implementation can be found here: http://jsteemann.github.io/blog/2015/04/22/collecting-with-a-hash-table/ * refactored HTTP REST API for cursors The HTTP REST API for cursors (`/_api/cursor`) has been refactored to improve its performance and use less memory. A post showing some of the performance improvements can be found here: http://jsteemann.github.io/blog/2015/04/01/improvements-for-the-cursor-api/ * simplified return value syntax for data-modification AQL queries ArangoDB 2.4 since version allows to return results from data-modification AQL queries. The syntax for this was quite limited and verbose: FOR i IN 1..10 INSERT { value: i } IN test LET inserted = NEW RETURN inserted The `LET inserted = NEW RETURN inserted` was required literally to return the inserted documents. No calculations could be made using the inserted documents. This is now more flexible. After a data-modification clause (e.g. `INSERT`, `UPDATE`, `REPLACE`, `REMOVE`, `UPSERT`) there can follow any number of `LET` calculations. These calculations can refer to the pseudo-values `OLD` and `NEW` that are created by the data-modification statements. This allows returning projections of inserted or updated documents, e.g.: FOR i IN 1..10 INSERT { value: i } IN test RETURN { _key: NEW._key, value: i } Still not every construct is allowed after a data-modification clause. For example, no functions can be called that may access documents. More information can be found here: http://jsteemann.github.io/blog/2015/03/27/improvements-for-data-modification-queries/ * added AQL `UPSERT` statement This adds an `UPSERT` statement to AQL that is a combination of both `INSERT` and `UPDATE` / `REPLACE`. The `UPSERT` will search for a matching document using a user-provided example. If no document matches the example, the *insert* part of the `UPSERT` statement will be executed. If there is a match, the *update* / *replace* part will be carried out: UPSERT { page: 'index.html' } /* search example */ INSERT { page: 'index.html', pageViews: 1 } /* insert part */ UPDATE { pageViews: OLD.pageViews + 1 } /* update part */ IN pageViews `UPSERT` can be used with an `UPDATE` or `REPLACE` clause. The `UPDATE` clause will perform a partial update of the found document, whereas the `REPLACE` clause will replace the found document entirely. The `UPDATE` or `REPLACE` parts can refer to the pseudo-value `OLD`, which contains all attributes of the found document. `UPSERT` statements can optionally return values. In the following query, the return attribute `found` will return the found document before the `UPDATE` was applied. If no document was found, `found` will contain a value of `null`. The `updated` result attribute will contain the inserted / updated document: UPSERT { page: 'index.html' } /* search example */ INSERT { page: 'index.html', pageViews: 1 } /* insert part */ UPDATE { pageViews: OLD.pageViews + 1 } /* update part */ IN pageViews RETURN { found: OLD, updated: NEW } A more detailed description of `UPSERT` can be found here: http://jsteemann.github.io/blog/2015/03/27/preview-of-the-upsert-command/ * adjusted default configuration value for `--server.backlog-size` from 10 to 64. * issue #1231: bug xor feature in AQL: LENGTH(null) == 4 This changes the behavior of the AQL `LENGTH` function as follows: - if the single argument to `LENGTH()` is `null`, then the result will now be `0`. In previous versions of ArangoDB, the result of `LENGTH(null)` was `4`. - if the single argument to `LENGTH()` is `true`, then the result will now be `1`. In previous versions of ArangoDB, the result of `LENGTH(true)` was `4`. - if the single argument to `LENGTH()` is `false`, then the result will now be `0`. In previous versions of ArangoDB, the result of `LENGTH(false)` was `5`. The results of `LENGTH()` with string, numeric, array object argument values do not change. * issue #1298: Bulk import if data already exists (#1298) This change extends the HTTP REST API for bulk imports as follows: When documents are imported and the `_key` attribute is specified for them, the import can be used for inserting and updating/replacing documents. Previously, the import could be used for inserting new documents only, and re-inserting a document with an existing key would have failed with a *unique key constraint violated* error. The above behavior is still the default. However, the API now allows controlling the behavior in case of a unique key constraint error via the optional URL parameter `onDuplicate`. This parameter can have one of the following values: - `error`: when a unique key constraint error occurs, do not import or update the document but report an error. This is the default. - `update`: when a unique key constraint error occurs, try to (partially) update the existing document with the data specified in the import. This may still fail if the document would violate secondary unique indexes. Only the attributes present in the import data will be updated and other attributes already present will be preserved. The number of updated documents will be reported in the `updated` attribute of the HTTP API result. - `replace`: when a unique key constraint error occurs, try to fully replace the existing document with the data specified in the import. This may still fail if the document would violate secondary unique indexes. The number of replaced documents will be reported in the `updated` attribute of the HTTP API result. - `ignore`: when a unique key constraint error occurs, ignore this error. There will be no insert, update or replace for the particular document. Ignored documents will be reported separately in the `ignored` attribute of the HTTP API result. The result of the HTTP import API will now contain the attributes `ignored` and `updated`, which contain the number of ignored and updated documents respectively. These attributes will contain a value of zero unless the `onDuplicate` URL parameter is set to either `update` or `replace` (in this case the `updated` attribute may contain non-zero values) or `ignore` (in this case the `ignored` attribute may contain a non-zero value). To support the feature, arangoimp also has a new command line option `--on-duplicate` which can have one of the values `error`, `update`, `replace`, `ignore`. The default value is `error`. A few examples for using arangoimp with the `--on-duplicate` option can be found here: http://jsteemann.github.io/blog/2015/04/14/updating-documents-with-arangoimp/ * changed behavior of `db._query()` in the ArangoShell: if the command's result is printed in the shell, the first 10 results will be printed. Previously only a basic description of the underlying query result cursor was printed. Additionally, if the cursor result contains more than 10 results, the cursor is assigned to a global variable `more`, which can be used to iterate over the cursor result. Example: arangosh [_system]> db._query("FOR i IN 1..15 RETURN i") [object ArangoQueryCursor, count: 15, hasMore: true] [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ] type 'more' to show more documents arangosh [_system]> more [object ArangoQueryCursor, count: 15, hasMore: false] [ 11, 12, 13, 14, 15 ] * Disallow batchSize value 0 in HTTP `POST /_api/cursor`: The HTTP REST API `POST /_api/cursor` does not accept a `batchSize` parameter value of `0` any longer. A batch size of 0 never made much sense, but previous versions of ArangoDB did not check for this value. Now creating a cursor using a `batchSize` value 0 will result in an HTTP 400 error response * REST Server: fix memory leaks when failing to add jobs * 'EDGES' AQL Function The AQL function `EDGES` got a new fifth option parameter. Right now only one option is available: 'includeVertices'. This is a boolean parameter that allows to modify the result of the `EDGES` function. Default is 'includeVertices: false' which does not have any effect. 'includeVertices: true' modifies the result, such that {vertex: , edge: } is returned. * INCOMPATIBLE CHANGE: The result format of the AQL function `NEIGHBORS` has been changed. Before it has returned an array of objects containing 'vertex' and 'edge'. Now it will only contain the vertex directly. Also an additional option 'includeData' has been added. This is used to define if only the 'vertex._id' value should be returned (false, default), or if the vertex should be looked up in the collection and the complete JSON should be returned (true). Using only the id values can lead to significantly improved performance if this is the only information required. In order to get the old result format prior to ArangoDB 2.6, please use the function EDGES instead. Edges allows for a new option 'includeVertices' which, set to true, returns exactly the format of NEIGHBORS. Example: NEIGHBORS(, , , , ) This can now be achieved by: EDGES(, , , , {includeVertices: true}) If you are nesting several NEIGHBORS steps you can speed up their performance in the following way: Old Example: FOR va IN NEIGHBORS(Users, relations, 'Users/123', 'outbound') FOR vc IN NEIGHBORS(Products, relations, va.vertex._id, 'outbound') RETURN vc This can now be achieved by: FOR va IN NEIGHBORS(Users, relations, 'Users/123', 'outbound') FOR vc IN NEIGHBORS(Products, relations, va, 'outbound', null, {includeData: true}) RETURN vc ^^^^ ^^^^^^^^^^^^^^^^^^^ Use intermediate directly include Data for final * INCOMPATIBLE CHANGE: The AQL function `GRAPH_NEIGHBORS` now provides an additional option `includeData`. This option allows controlling whether the function should return the complete vertices or just their IDs. Returning only the IDs instead of the full vertices can lead to improved performance . If provided, `includeData` is set to `true`, all vertices in the result will be returned with all their attributes. The default value of `includeData` is `false`. This makes the default function results incompatible with previous versions of ArangoDB. To get the old result style in ArangoDB 2.6, please set the options as follows in calls to `GRAPH_NEIGHBORS`: GRAPH_NEIGHBORS(, , { includeData: true }) * INCOMPATIBLE CHANGE: The AQL function `GRAPH_COMMON_NEIGHBORS` now provides an additional option `includeData`. This option allows controlling whether the function should return the complete vertices or just their IDs. Returning only the IDs instead of the full vertices can lead to improved performance . If provided, `includeData` is set to `true`, all vertices in the result will be returned with all their attributes. The default value of `includeData` is `false`. This makes the default function results incompatible with previous versions of ArangoDB. To get the old result style in ArangoDB 2.6, please set the options as follows in calls to `GRAPH_COMMON_NEIGHBORS`: GRAPH_COMMON_NEIGHBORS(, , , { includeData: true }, { includeData: true }) * INCOMPATIBLE CHANGE: The AQL function `GRAPH_SHORTEST_PATH` now provides an additional option `includeData`. This option allows controlling whether the function should return the complete vertices and edges or just their IDs. Returning only the IDs instead of full vertices and edges can lead to improved performance . If provided, `includeData` is set to `true`, all vertices and edges in the result will be returned with all their attributes. There is also an optional parameter `includePath` of type object. It has two optional sub-attributes `vertices` and `edges`, both of type boolean. Both can be set individually and the result will include all vertices on the path if `includePath.vertices == true` and all edges if `includePath.edges == true` respectively. The default value of `includeData` is `false`, and paths are now excluded by default. This makes the default function results incompatible with previous versions of ArangoDB. To get the old result style in ArangoDB 2.6, please set the options as follows in calls to `GRAPH_SHORTEST_PATH`: GRAPH_SHORTEST_PATH(, , , { includeData: true, includePath: { edges: true, vertices: true } }) The attributes `startVertex` and `vertex` that were present in the results of `GRAPH_SHORTEST_PATH` in previous versions of ArangoDB will not be produced in 2.6. To calculate these attributes in 2.6, please extract the first and last elements from the `vertices` result attribute. * INCOMPATIBLE CHANGE: The AQL function `GRAPH_DISTANCE_TO` will now return only the id the destination vertex in the `vertex` attribute, and not the full vertex data with all vertex attributes. * INCOMPATIBLE CHANGE: All graph measurements functions in JavaScript module `general-graph` that calculated a single figure previously returned an array containing just the figure. Now these functions will return the figure directly and not put it inside an array. The affected functions are: * `graph._absoluteEccentricity` * `graph._eccentricity` * `graph._absoluteCloseness` * `graph._closeness` * `graph._absoluteBetweenness` * `graph._betweenness` * `graph._radius` * `graph._diameter` * Create the `_graphs` collection in new databases with `waitForSync` attribute set to `false` The previous `waitForSync` value was `true`, so default the behavior when creating and dropping graphs via the HTTP REST API changes as follows if the new settings are in effect: * `POST /_api/graph` by default returns `HTTP 202` instead of `HTTP 201` * `DELETE /_api/graph/graph-name` by default returns `HTTP 202` instead of `HTTP 201` If the `_graphs` collection still has its `waitForSync` value set to `true`, then the HTTP status code will not change. * Upgraded ICU to version 54; this increases performance in many places. based on https://code.google.com/p/chromium/issues/detail?id=428145 * added support for HTTP push aka chunked encoding * issue #1051: add info whether server is running in service or user mode? This will add a "mode" attribute to the result of the result of HTTP GET `/_api/version?details=true` "mode" can have the following values: - `standalone`: server was started manually (e.g. on command-line) - `service`: service is running as Windows service, in daemon mode or under the supervisor * improve system error messages in Windows port * increased default value of `--server.request-timeout` from 300 to 1200 seconds for client tools (arangosh, arangoimp, arangodump, arangorestore) * increased default value of `--server.connect-timeout` from 3 to 5 seconds for client tools (arangosh, arangoimp, arangodump, arangorestore) * added startup option `--server.foxx-queues-poll-interval` This startup option controls the frequency with which the Foxx queues manager is checking the queue (or queues) for jobs to be executed. The default value is `1` second. Lowering this value will result in the queue manager waking up and checking the queues more frequently, which may increase CPU usage of the server. When not using Foxx queues, this value can be raised to save some CPU time. * added startup option `--server.foxx-queues` This startup option controls whether the Foxx queue manager will check queue and job entries. Disabling this option can reduce server load but will prevent jobs added to Foxx queues from being processed at all. The default value is `true`, enabling the Foxx queues feature. * make Foxx queues really database-specific. Foxx queues were and are stored in a database-specific collection `_queues`. However, a global cache variable for the queues led to the queue names being treated database-independently, which was wrong. Since 2.6, Foxx queues names are truly database-specific, so the same queue name can be used in two different databases for two different queues. Until then, it is advisable to think of queues as already being database-specific, and using the database name as a queue name prefix to be avoid name conflicts, e.g.: var queueName = "myQueue"; var Foxx = require("org/arangodb/foxx"); Foxx.queues.create(db._name() + ":" + queueName); * added support for Foxx queue job types defined as app scripts. The old job types introduced in 2.4 are still supported but are known to cause issues in 2.5 and later when the server is restarted or the job types are not defined in every thread. The new job types avoid this issue by storing an explicit mount path and script name rather than an assuming the job type is defined globally. It is strongly recommended to convert your job types to the new script-based system. * renamed Foxx sessions option "sessionStorageApp" to "sessionStorage". The option now also accepts session storages directly. * Added the following JavaScript methods for file access: * fs.copyFile() to copy single files * fs.copyRecursive() to copy directory trees * fs.chmod() to set the file permissions (non-Windows only) * Added process.env for accessing the process environment from JavaScript code * Cluster: kickstarter shutdown routines will more precisely follow the shutdown of its nodes. * Cluster: don't delete agency connection objects that are currently in use. * Cluster: improve passing along of HTTP errors * fixed issue #1247: debian init script problems * multi-threaded index creation on collection load When a collection contains more than one secondary index, they can be built in memory in parallel when the collection is loaded. How many threads are used for parallel index creation is determined by the new configuration parameter `--database.index-threads`. If this is set to 0, indexes are built by the opening thread only and sequentially. This is equivalent to the behavior in 2.5 and before. * speed up building up primary index when loading collections * added `count` attribute to `parameters.json` files of collections. This attribute indicates the number of live documents in the collection on unload. It is read when the collection is (re)loaded to determine the initial size for the collection's primary index * removed remainders of MRuby integration, removed arangoirb * simplified `controllers` property in Foxx manifests. You can now specify a filename directly if you only want to use a single file mounted at the base URL of your Foxx app. * simplified `exports` property in Foxx manifests. You can now specify a filename directly if you only want to export variables from a single file in your Foxx app. * added support for node.js-style exports in Foxx exports. Your Foxx exports file can now export arbitrary values using the `module.exports` property instead of adding properties to the `exports` object. * added `scripts` property to Foxx manifests. You should now specify the `setup` and `teardown` files as properties of the `scripts` object in your manifests and can define custom, app-specific scripts that can be executed from the web interface or the CLI. * added `tests` property to Foxx manifests. You can now define test cases using the `mocha` framework which can then be executed inside ArangoDB. * updated `joi` package to 6.0.8. * added `extendible` package. * added Foxx model lifecycle events to repositories. See #1257. * speed up resizing of edge index. * allow to split an edge index into buckets which are resized individually. This is controlled by the `indexBuckets` attribute in the `properties` of the collection. * fix a cluster deadlock bug in larger clusters by marking a thread waiting for a lock on a DBserver as blocked v2.5.7 (2015-08-02) ------------------- * V8: Upgrade to version 4.1.0.27 - this is intended to be the stable V8 version. v2.5.6 (2015-07-21) ------------------- * alter Windows build infrastructure so we can properly store pdb files. * potentially fixed issue #1313: Wrong metric calculation at dashboard Escape whitespace in process name when scanning /proc/pid/stats This fixes statistics values read from that file * Fixed variable naming in AQL `COLLECT INTO` results in case the COLLECT is placed in a subquery which itself is followed by other constructs that require variables v2.5.5 (2015-05-29) ------------------- * fixed vulnerability in JWT implementation. * fixed format string for reading /proc/pid/stat * take into account barriers used in different V8 contexts v2.5.4 (2015-05-14) ------------------- * added startup option `--log.performance`: specifying this option at startup will log performance-related info messages, mainly timings via the regular logging mechanisms * cluster fixes * fix for recursive copy under Windows v2.5.3 (2015-04-29) ------------------- * Fix fs.move to work across filesystem borders; Fixes Foxx app installation problems; issue #1292. * Fix Foxx app install when installed on a different drive on Windows * issue #1322: strange AQL result * issue #1318: Inconsistent db._create() syntax * issue #1315: queries to a collection fail with an empty response if the collection contains specific JSON data * issue #1300: Make arangodump not fail if target directory exists but is empty * allow specifying higher values than SOMAXCONN for `--server.backlog-size` Previously, arangod would not start when a `--server.backlog-size` value was specified that was higher than the platform's SOMAXCONN header value. Now, arangod will use the user-provided value for `--server.backlog-size` and pass it to the listen system call even if the value is higher than SOMAXCONN. If the user-provided value is higher than SOMAXCONN, arangod will log a warning on startup. * Fixed a cluster deadlock bug. Mark a thread that is in a RemoteBlock as blocked to allow for additional dispatcher threads to be started. * Fix locking in cluster by using another ReadWriteLock class for collections. * Add a second DispatcherQueue for AQL in the cluster. This fixes a cluster-AQL thread explosion bug. v2.5.2 (2015-04-11) ------------------- * modules stored in _modules are automatically flushed when changed * added missing query-id parameter in documentation of HTTP DELETE `/_api/query` endpoint * added iterator for edge index in AQL queries this change may lead to less edges being read when used together with a LIMIT clause * make graph viewer in web interface issue less expensive queries for determining a random vertex from the graph, and for determining vertex attributes * issue #1285: syntax error, unexpected $undefined near '@_to RETURN obj this allows AQL bind parameter names to also start with underscores * moved /_api/query to C++ * issue #1289: Foxx models created from database documents expose an internal method * added `Foxx.Repository#exists` * parallelize initialization of V8 context in multiple threads * fixed a possible crash when the debug-level was TRACE * cluster: do not initialize statistics collection on each coordinator, this fixes a race condition at startup * cluster: fix a startup race w.r.t. the _configuration collection * search for db:// JavaScript modules only after all local files have been considered, this speeds up the require command in a cluster considerably * general cluster speedup in certain areas v2.5.1 (2015-03-19) ------------------- * fixed bug that caused undefined behavior when an AQL query was killed inside a calculation block * fixed memleaks in AQL query cleanup in case out-of-memory errors are thrown * by default, Debian and RedHat packages are built with debug symbols * added option `--database.ignore-logfile-errors` This option controls how collection datafiles with a CRC mismatch are treated. If set to `false`, CRC mismatch errors in collection datafiles will lead to a collection not being loaded at all. If a collection needs to be loaded during WAL recovery, the WAL recovery will also abort (if not forced with `--wal.ignore-recovery-errors true`). Setting this flag to `false` protects users from unintentionally using a collection with corrupted datafiles, from which only a subset of the original data can be recovered. If set to `true`, CRC mismatch errors in collection datafiles will lead to the datafile being partially loaded. All data up to until the mismatch will be loaded. This will enable users to continue with collection datafiles that are corrupted, but will result in only a partial load of the data. The WAL recovery will still abort when encountering a collection with a corrupted datafile, at least if `--wal.ignore-recovery-errors` is not set to `true`. The default value is *true*, so for collections with corrupted datafiles there might be partial data loads once the WAL recovery has finished. If the WAL recovery will need to load a collection with a corrupted datafile, it will still stop when using the default values. * INCOMPATIBLE CHANGE: make the arangod server refuse to start if during startup it finds a non-readable `parameter.json` file for a database or a collection. Stopping the startup process in this case requires manual intervention (fixing the unreadable files), but prevents follow-up errors due to ignored databases or collections from happening. * datafiles and `parameter.json` files written by arangod are now created with read and write privileges for the arangod process user, and with read and write privileges for the arangod process group. Previously, these files were created with user read and write permissions only. * INCOMPATIBLE CHANGE: abort WAL recovery if one of the collection's datafiles cannot be opened * INCOMPATIBLE CHANGE: never try to raise the privileges after dropping them, this can lead to a race condition while running the recovery If you require to run ArangoDB on a port lower than 1024, you must run ArangoDB as root. * fixed inefficiencies in `remove` methods of general-graph module * added option `--database.slow-query-threshold` for controlling the default AQL slow query threshold value on server start * add system error strings for Windows on many places * rework service startup so we announce 'RUNNING' only when we're finished starting. * use the Windows eventlog for FATAL and ERROR - log messages * fix service handling in NSIS Windows installer, specify human readable name * add the ICU_DATA environment variable to the fatal error messages * fixed issue #1265: arangod crashed with SIGSEGV * fixed issue #1241: Wildcards in examples v2.5.0 (2015-03-09) ------------------- * installer fixes for Windows * fix for downloading Foxx * fixed issue #1258: http pipelining not working? v2.5.0-beta4 (2015-03-05) ------------------------- * fixed issue #1247: debian init script problems v2.5.0-beta3 (2015-02-27) ------------------------- * fix Windows install path calculation in arango * fix Windows logging of long strings * fix possible undefinedness of const strings in Windows v2.5.0-beta2 (2015-02-23) ------------------------- * fixed issue #1256: agency binary not found #1256 * fixed issue #1230: API: document/col-name/_key and cursor return different floats * front-end: dashboard tries not to (re)load statistics if user has no access * V8: Upgrade to version 3.31.74.1 * etcd: Upgrade to version 2.0 - This requires go 1.3 to compile at least. * refuse to startup if ICU wasn't initialized, this will i.e. prevent errors from being printed, and libraries from being loaded. * front-end: unwanted removal of index table header after creating new index * fixed issue #1248: chrome: applications filtering not working * fixed issue #1198: queries remain in aql editor (front-end) if you navigate through different tabs * Simplify usage of Foxx Thanks to our user feedback we learned that Foxx is a powerful, yet rather complicated concept. With this release we tried to make it less complicated while keeping all its strength. That includes a rewrite of the documentation as well as some code changes as listed below: * Moved Foxx applications to a different folder. The naming convention now is: /_db///APP Before it was: /databases//: This caused some trouble as apps where cached based on name and version and updates did not apply. Hence the path on filesystem and the app's access URL had no relation to one another. Now the path on filesystem is identical to the URL (except for slashes and the appended APP) * Rewrite of Foxx routing The routing of Foxx has been exposed to major internal changes we adjusted because of user feedback. This allows us to set the development mode per mount point without having to change paths and hold apps at separate locations. * Foxx Development mode The development mode used until 2.4 is gone. It has been replaced by a much more mature version. This includes the deprecation of the javascript.dev-app-path parameter, which is useless since 2.5. Instead of having two separate app directories for production and development, apps now reside in one place, which is used for production as well as for development. Apps can still be put into development mode, changing their behavior compared to production mode. Development mode apps are still reread from disk at every request, and still they ship more debug output. This change has also made the startup options `--javascript.frontend-development-mode` and `--javascript.dev-app-path` obsolete. The former option will not have any effect when set, and the latter option is only read and used during the upgrade to 2.5 and does not have any effects later. * Foxx install process Installing Foxx apps has been a two step process: import them into ArangoDB and mount them at a specific mount point. These operations have been joined together. You can install an app at one mount point, that's it. No fetch, mount, unmount, purge cycle anymore. The commands have been simplified to just: * install: get your Foxx app up and running * uninstall: shut it down and erase it from disk * Foxx error output Until 2.4 the errors produced by Foxx were not optimal. Often, the error message was just `unable to parse manifest` and contained only an internal stack trace. In 2.5 we made major improvements there, including a much more fine-grained error output that helps you debug your Foxx apps. The error message printed is now much closer to its source and should help you track it down. Also we added the default handlers for unhandled errors in Foxx apps: * You will get a nice internal error page whenever your Foxx app is called but was not installed due to any error * You will get a proper error message when having an uncaught error appears in any app route In production mode the messages above will NOT contain any information about your Foxx internals and are safe to be exposed to third party users. In development mode the messages above will contain the stacktrace (if available), making it easier for your in-house devs to track down errors in the application. * added `console` object to Foxx apps. All Foxx apps now have a console object implementing the familiar Console API in their global scope, which can be used to log diagnostic messages to the database. * added `org/arangodb/request` module, which provides a simple API for making HTTP requests to external services. * added optimizer rule `propagate-constant-attributes` This rule will look inside `FILTER` conditions for constant value equality comparisons, and insert the constant values in other places in `FILTER`s. For example, the rule will insert `42` instead of `i.value` in the second `FILTER` of the following query: FOR i IN c1 FOR j IN c2 FILTER i.value == 42 FILTER j.value == i.value RETURN 1 * added `filtered` value to AQL query execution statistics This value indicates how many documents were filtered by `FilterNode`s in the AQL query. Note that `IndexRangeNode`s can also filter documents by selecting only the required ranges from the index. The `filtered` value will not include the work done by `IndexRangeNode`s, but only the work performed by `FilterNode`s. * added support for sparse hash and skiplist indexes Hash and skiplist indexes can optionally be made sparse. Sparse indexes exclude documents in which at least one of the index attributes is either not set or has a value of `null`. As such documents are excluded from sparse indexes, they may contain fewer documents than their non-sparse counterparts. This enables faster indexing and can lead to reduced memory usage in case the indexed attribute does occur only in some, but not all documents of the collection. Sparse indexes will also reduce the number of collisions in non-unique hash indexes in case non-existing or optional attributes are indexed. In order to create a sparse index, an object with the attribute `sparse` can be added to the index creation commands: db.collection.ensureHashIndex(attributeName, { sparse: true }); db.collection.ensureHashIndex(attributeName1, attributeName2, { sparse: true }); db.collection.ensureUniqueConstraint(attributeName, { sparse: true }); db.collection.ensureUniqueConstraint(attributeName1, attributeName2, { sparse: true }); db.collection.ensureSkiplist(attributeName, { sparse: true }); db.collection.ensureSkiplist(attributeName1, attributeName2, { sparse: true }); db.collection.ensureUniqueSkiplist(attributeName, { sparse: true }); db.collection.ensureUniqueSkiplist(attributeName1, attributeName2, { sparse: true }); Note that in place of the above specialized index creation commands, it is recommended to use the more general index creation command `ensureIndex`: ```js db.collection.ensureIndex({ type: "hash", sparse: true, unique: true, fields: [ attributeName ] }); db.collection.ensureIndex({ type: "skiplist", sparse: false, unique: false, fields: [ "a", "b" ] }); ``` When not explicitly set, the `sparse` attribute defaults to `false` for new indexes. This causes a change in behavior when creating a unique hash index without specifying the sparse flag: in 2.4, unique hash indexes were implicitly sparse, always excluding `null` values. There was no option to control this behavior, and sparsity was neither supported for non-unique hash indexes nor skiplists in 2.4. This implicit sparsity of unique hash indexes was considered an inconsistency, and therefore the behavior was cleaned up in 2.5. As of 2.5, indexes will only be created sparse if sparsity is explicitly requested. Existing unique hash indexes from 2.4 or before will automatically be migrated so they are still sparse after the upgrade to 2.5. Geo indexes are implicitly sparse, meaning documents without the indexed location attribute or containing invalid location coordinate values will be excluded from the index automatically. This is also a change when compared to pre-2.5 behavior, when documents with missing or invalid coordinate values may have caused errors on insertion when the geo index' `unique` flag was set and its `ignoreNull` flag was not. This was confusing and has been rectified in 2.5. The method `ensureGeoConstraint()` now does the same as `ensureGeoIndex()`. Furthermore, the attributes `constraint`, `unique`, `ignoreNull` and `sparse` flags are now completely ignored when creating geo indexes. The same is true for fulltext indexes. There is no need to specify non-uniqueness or sparsity for geo or fulltext indexes. They will always be non-unique and sparse. As sparse indexes may exclude some documents, they cannot be used for every type of query. Sparse hash indexes cannot be used to find documents for which at least one of the indexed attributes has a value of `null`. For example, the following AQL query cannot use a sparse index, even if one was created on attribute `attr`: FOR doc In collection FILTER doc.attr == null RETURN doc If the lookup value is non-constant, a sparse index may or may not be used, depending on the other types of conditions in the query. If the optimizer can safely determine that the lookup value cannot be `null`, a sparse index may be used. When uncertain, the optimizer will not make use of a sparse index in a query in order to produce correct results. For example, the following queries cannot use a sparse index on `attr` because the optimizer will not know beforehand whether the comparison values for `doc.attr` will include `null`: FOR doc In collection FILTER doc.attr == SOME_FUNCTION(...) RETURN doc FOR other IN otherCollection FOR doc In collection FILTER doc.attr == other.attr RETURN doc Sparse skiplist indexes can be used for sorting if the optimizer can safely detect that the index range does not include `null` for any of the index attributes. * inspection of AQL data-modification queries will now detect if the data-modification part of the query can run in lockstep with the data retrieval part of the query, or if the data retrieval part must be executed before the data modification can start. Executing the two in lockstep allows using much smaller buffers for intermediate results and starts the actual data-modification operations much earlier than if the two phases were executed separately. * Allow dynamic attribute names in AQL object literals This allows using arbitrary expressions to construct attribute names in object literals specified in AQL queries. To disambiguate expressions and other unquoted attribute names, dynamic attribute names need to be enclosed in brackets (`[` and `]`). Example: FOR i IN 1..100 RETURN { [ CONCAT('value-of-', i) ] : i } * make AQL optimizer rule "use-index-for-sort" remove sort also in case a non-sorted index (e.g. a hash index) is used for only equality lookups and all sort attributes are covered by the index. Example that does not require an extra sort (needs hash index on `value`): FOR doc IN collection FILTER doc.value == 1 SORT doc.value RETURN doc Another example that does not require an extra sort (with hash index on `value1`, `value2`): FOR doc IN collection FILTER doc.value1 == 1 && doc.value2 == 2 SORT doc.value1, doc.value2 RETURN doc * make AQL optimizer rule "use-index-for-sort" remove sort also in case the sort criteria excludes the left-most index attributes, but the left-most index attributes are used by the index for equality-only lookups. Example that can use the index for sorting (needs skiplist index on `value1`, `value2`): FOR doc IN collection FILTER doc.value1 == 1 SORT doc.value2 RETURN doc * added selectivity estimates for primary index, edge index, and hash index The selectivity estimates are returned by the `GET /_api/index` REST API method in a sub-attribute `selectivityEstimate` for each index that supports it. This attribute will be omitted for indexes that do not provide selectivity estimates. If provided, the selectivity estimate will be a numeric value between 0 and 1. Selectivity estimates will also be reported in the result of `collection.getIndexes()` for all indexes that support this. If no selectivity estimate can be determined for an index, the attribute `selectivityEstimate` will be omitted here, too. The web interface also shows selectivity estimates for each index that supports this. Currently the following index types can provide selectivity estimates: - primary index - edge index - hash index (unique and non-unique) No selectivity estimates will be provided when running in cluster mode. * fixed issue #1226: arangod log issues * added additional logger if arangod is started in foreground mode on a tty * added AQL optimizer rule "move-calculations-down" * use exclusive native SRWLocks on Windows instead of native mutexes * added AQL functions `MD5`, `SHA1`, and `RANDOM_TOKEN`. * reduced number of string allocations when parsing certain AQL queries parsing numbers (integers or doubles) does not require a string allocation per number anymore * RequestContext#bodyParam now accepts arbitrary joi schemas and rejects invalid (but well-formed) request bodies. * enforce that AQL user functions are wrapped inside JavaScript function () declarations AQL user functions were always expected to be wrapped inside a JavaScript function, but previously this was not enforced when registering a user function. Enforcing the AQL user functions to be contained inside functions prevents functions from doing some unexpected things that may have led to undefined behavior. * Windows service uninstalling: only remove service if it points to the currently running binary, or --force was specified. * Windows (debug only): print stacktraces on crash and run minidump * Windows (cygwin): if you run arangosh in a cygwin shell or via ssh we will detect this and use the appropriate output functions. * Windows: improve process management * fix IPv6 reverse ip lookups - so far we only did IPv4 addresses. * improve join documentation, add outer join example * run jslint for unit tests too, to prevent "memory leaks" by global js objects with native code. * fix error logging for exceptions - we wouldn't log the exception message itself so far. * improve error reporting in the http client (Windows & *nix) * improve error reports in cluster * Standard errors can now contain custom messages. v2.4.7 (XXXX-XX-XX) ------------------- * fixed issue #1282: Geo WITHIN_RECTANGLE for nested lat/lng v2.4.6 (2015-03-18) ------------------- * added option `--database.ignore-logfile-errors` This option controls how collection datafiles with a CRC mismatch are treated. If set to `false`, CRC mismatch errors in collection datafiles will lead to a collection not being loaded at all. If a collection needs to be loaded during WAL recovery, the WAL recovery will also abort (if not forced with `--wal.ignore-recovery-errors true`). Setting this flag to `false` protects users from unintentionally using a collection with corrupted datafiles, from which only a subset of the original data can be recovered. If set to `true`, CRC mismatch errors in collection datafiles will lead to the datafile being partially loaded. All data up to until the mismatch will be loaded. This will enable users to continue with a collection datafiles that are corrupted, but will result in only a partial load of the data. The WAL recovery will still abort when encountering a collection with a corrupted datafile, at least if `--wal.ignore-recovery-errors` is not set to `true`. The default value is *true*, so for collections with corrupted datafiles there might be partial data loads once the WAL recovery has finished. If the WAL recovery will need to load a collection with a corrupted datafile, it will still stop when using the default values. * INCOMPATIBLE CHANGE: make the arangod server refuse to start if during startup it finds a non-readable `parameter.json` file for a database or a collection. Stopping the startup process in this case requires manual intervention (fixing the unreadable files), but prevents follow-up errors due to ignored databases or collections from happening. * datafiles and `parameter.json` files written by arangod are now created with read and write privileges for the arangod process user, and with read and write privileges for the arangod process group. Previously, these files were created with user read and write permissions only. * INCOMPATIBLE CHANGE: abort WAL recovery if one of the collection's datafiles cannot be opened * INCOMPATIBLE CHANGE: never try to raise the privileges after dropping them, this can lead to a race condition while running the recovery If you require to run ArangoDB on a port lower than 1024, you must run ArangoDB as root. * fixed inefficiencies in `remove` methods of general-graph module * added option `--database.slow-query-threshold` for controlling the default AQL slow query threshold value on server start v2.4.5 (2015-03-16) ------------------- * added elapsed time to HTTP request logging output (`--log.requests-file`) * added AQL current and slow query tracking, killing of AQL queries This change enables retrieving the list of currently running AQL queries inside the selected database. AQL queries with an execution time beyond a certain threshold can be moved to a "slow query" facility and retrieved from there. Queries can also be killed by specifying the query id. This change adds the following HTTP REST APIs: - `GET /_api/query/current`: for retrieving the list of currently running queries - `GET /_api/query/slow`: for retrieving the list of slow queries - `DELETE /_api/query/slow`: for clearing the list of slow queries - `GET /_api/query/properties`: for retrieving the properties for query tracking - `PUT /_api/query/properties`: for adjusting the properties for query tracking - `DELETE /_api/query/`: for killing an AQL query The following JavaScript APIs have been added: - require("org/arangodb/aql/queries").current(); - require("org/arangodb/aql/queries").slow(); - require("org/arangodb/aql/queries").clearSlow(); - require("org/arangodb/aql/queries").properties(); - require("org/arangodb/aql/queries").kill(); * fixed issue #1265: arangod crashed with SIGSEGV * fixed issue #1241: Wildcards in examples * fixed comment parsing in Foxx controllers v2.4.4 (2015-02-24) ------------------- * fixed the generation template for foxx apps. It now does not create deprecated functions anymore * add custom visitor functionality for `GRAPH_NEIGHBORS` function, too * increased default value of traversal option *maxIterations* to 100 times of its previous default value v2.4.3 (2015-02-06) ------------------- * fix multi-threading with openssl when running under Windows * fix timeout on socket operations when running under Windows * Fixed an error in Foxx routing which caused some apps that worked in 2.4.1 to fail with status 500: `undefined is not a function` errors in 2.4.2 This error was occurring due to seldom internal rerouting introduced by the malformed application handler. v2.4.2 (2015-01-30) ------------------- * added custom visitor functionality for AQL traversals This allows more complex result processing in traversals triggered by AQL. A few examples are shown in [this article](http://jsteemann.github.io/blog/2015/01/28/using-custom-visitors-in-aql-graph-traversals/). * improved number of results estimated for nodes of type EnumerateListNode and SubqueryNode in AQL explain output * added AQL explain helper to explain arbitrary AQL queries The helper function prints the query execution plan and the indexes to be used in the query. It can be invoked from the ArangoShell or the web interface as follows: require("org/arangodb/aql/explainer").explain(query); * enable use of indexes for certain AQL conditions with non-equality predicates, in case the condition(s) also refer to indexed attributes The following queries will now be able to use indexes: FILTER a.indexed == ... && a.indexed != ... FILTER a.indexed == ... && a.nonIndexed != ... FILTER a.indexed == ... && ! (a.indexed == ...) FILTER a.indexed == ... && ! (a.nonIndexed == ...) FILTER a.indexed == ... && ! (a.indexed != ...) FILTER a.indexed == ... && ! (a.nonIndexed != ...) FILTER (a.indexed == ... && a.nonIndexed == ...) || (a.indexed == ... && a.nonIndexed == ...) FILTER (a.indexed == ... && a.nonIndexed != ...) || (a.indexed == ... && a.nonIndexed != ...) * Fixed spuriously occurring "collection not found" errors when running queries on local collections on a cluster DB server * Fixed upload of Foxx applications to the server for apps exceeding approx. 1 MB zipped. * Malformed Foxx applications will now return a more useful error when any route is requested. In Production a Foxx app mounted on /app will display an html page on /app/* stating a 503 Service temporarily not available. It will not state any information about your Application. Before it was a 404 Not Found without any information and not distinguishable from a correct not found on your route. In Development Mode the html page also contains information about the error occurred. * Unhandled errors thrown in Foxx routes are now handled by the Foxx framework itself. In Production the route will return a status 500 with a body {error: "Error statement"}. In Development the route will return a status 500 with a body {error: "Error statement", stack: "..."} Before, it was status 500 with a plain text stack including ArangoDB internal routing information. * The Applications tab in web interface will now request development apps more often. So if you have a fixed a syntax error in your app it should always be visible after reload. v2.4.1 (2015-01-19) ------------------- * improved WAL recovery output * fixed certain OR optimizations in AQL optimizer * better diagnostics for arangoimp * fixed invalid result of HTTP REST API method `/_admin/foxx/rescan` * fixed possible segmentation fault when passing a Buffer object into a V8 function as a parameter * updated AQB module to 1.8.0. v2.4.0 (2015-01-13) ------------------- * updated AQB module to 1.7.0. * fixed V8 integration-related crashes * make `fs.move(src, dest)` also fail when both `src` and `dest` are existing directories. This ensures the same behavior of the move operation on different platforms. * fixed AQL insert operation for multi-shard collections in cluster * added optional return value for AQL data-modification queries. This allows returning the documents inserted, removed or updated with the query, e.g. FOR doc IN docs REMOVE doc._key IN docs LET removed = OLD RETURN removed FOR doc IN docs INSERT { } IN docs LET inserted = NEW RETURN inserted FOR doc IN docs UPDATE doc._key WITH { } IN docs LET previous = OLD RETURN previous FOR doc IN docs UPDATE doc._key WITH { } IN docs LET updated = NEW RETURN updated The variables `OLD` and `NEW` are automatically available when a `REMOVE`, `INSERT`, `UPDATE` or `REPLACE` statement is immediately followed by a `LET` statement. Note that the `LET` and `RETURN` statements in data-modification queries are not as flexible as the general versions of `LET` and `RETURN`. When returning documents from data-modification operations, only a single variable can be assigned using `LET`, and the assignment can only be either `OLD` or `NEW`, but not an arbitrary expression. The `RETURN` statement also allows using the just-created variable only, and no arbitrary expressions. v2.4.0-beta1 (2014-12-26) -------------------------- * fixed superstates in FoxxGenerator * fixed issue #1065: Aardvark: added creation of documents and edges with _key property * fixed issue #1198: Aardvark: current AQL editor query is now cached * Upgraded V8 version from 3.16.14 to 3.29.59 The built-in version of V8 has been upgraded from 3.16.14 to 3.29.59. This activates several ES6 (also dubbed *Harmony* or *ES.next*) features in ArangoDB, both in the ArangoShell and the ArangoDB server. They can be used for scripting and in server-side actions such as Foxx routes, traversals etc. The following ES6 features are available in ArangoDB 2.4 by default: * iterators * the `of` operator * symbols * predefined collections types (Map, Set etc.) * typed arrays Many other ES6 features are disabled by default, but can be made available by starting arangod or arangosh with the appropriate options: * arrow functions * proxies * generators * String, Array, and Number enhancements * constants * enhanced object and numeric literals To activate all these ES6 features in arangod or arangosh, start it with the following options: arangosh --javascript.v8-options="--harmony --harmony_generators" More details on the available ES6 features can be found in [this blog](https://jsteemann.github.io/blog/2014/12/19/using-es6-features-in-arangodb/). * Added Foxx generator for building Hypermedia APIs A more detailed description is [here](https://www.arangodb.com/2014/12/08/building-hypermedia-apis-foxxgenerator) * New `Applications` tab in web interface: The `applications` tab got a complete redesign. It will now only show applications that are currently running on ArangoDB. For a selected application, a new detailed view has been created. This view provides a better overview of the app: * author * license * version * contributors * download links * API documentation To install a new application, a new dialog is now available. It provides the features already available in the console application `foxx-manager` plus some more: * install an application from GitHub * install an application from a zip file * install an application from ArangoDB's application store * create a new application from scratch: this feature uses a generator to create a Foxx application with pre-defined CRUD methods for a given list of collections. The generated Foxx app can either be downloaded as a zip file or be installed on the server. Starting with a new Foxx app has never been easier. * fixed issue #1102: Aardvark: Layout bug in documents overview The documents overview was entirely destroyed in some situations on Firefox. We replaced the plugin we used there. * fixed issue #1168: Aardvark: pagination buttons jumping * fixed issue #1161: Aardvark: Click on Import JSON imports previously uploaded file * removed configure options `--enable-all-in-one-v8`, `--enable-all-in-one-icu`, and `--enable-all-in-one-libev`. * global internal rename to fix naming incompatibilities with JSON: Internal functions with names containing `array` have been renamed to `object`, internal functions with names containing `list` have been renamed to `array`. The renaming was mainly done in the C++ parts. The documentation has also been adjusted so that the correct JSON type names are used in most places. The change also led to the addition of a few function aliases in AQL: * `TO_LIST` now is an alias of the new `TO_ARRAY` * `IS_LIST` now is an alias of the new `IS_ARRAY` * `IS_DOCUMENT` now is an alias of the new `IS_OBJECT` The changed also renamed the option `mergeArrays` to `mergeObjects` for AQL data-modification query options and HTTP document modification API * AQL: added optimizer rule "remove-filter-covered-by-index" This rule removes FilterNodes and CalculationNodes from an execution plan if the filter is already covered by a previous IndexRangeNode. Removing the CalculationNode and the FilterNode will speed up query execution because the query requires less computation. * AQL: added optimizer rule "remove-sort-rand" This rule removes a `SORT RAND()` expression from a query and moves the random iteration into the appropriate `EnumerateCollectionNode`. This is more efficient than individually enumerating and then sorting randomly. * AQL: range optimizations for IN and OR This change enables usage of indexes for several additional cases. Filters containing the `IN` operator can now make use of indexes, and multiple OR- or AND-combined filter conditions can now also use indexes if the filters are accessing the same indexed attribute. Here are a few examples of queries that can now use indexes but couldn't before: FOR doc IN collection FILTER doc.indexedAttribute == 1 || doc.indexedAttribute > 99 RETURN doc FOR doc IN collection FILTER doc.indexedAttribute IN [ 3, 42 ] || doc.indexedAttribute > 99 RETURN doc FOR doc IN collection FILTER (doc.indexedAttribute > 2 && doc.indexedAttribute < 10) || (doc.indexedAttribute > 23 && doc.indexedAttribute < 42) RETURN doc * fixed issue #500: AQL parentheses issue This change allows passing subqueries as AQL function parameters without using duplicate brackets (e.g. `FUNC(query)` instead of `FUNC((query))` * added optional `COUNT` clause to AQL `COLLECT` This allows more efficient group count calculation queries, e.g. FOR doc IN collection COLLECT age = doc.age WITH COUNT INTO length RETURN { age: age, count: length } A count-only query is also possible: FOR doc IN collection COLLECT WITH COUNT INTO length RETURN length * fixed missing makeDirectory when fetching a Foxx application from a zip file * fixed issue #1134: Change the default endpoint to localhost This change will modify the IP address ArangoDB listens on to 127.0.0.1 by default. This will make new ArangoDB installations unaccessible from clients other than localhost unless changed. This is a security feature. To make ArangoDB accessible from any client, change the server's configuration (`--server.endpoint`) to either `tcp://0.0.0.0:8529` or the server's publicly visible IP address. * deprecated `Repository#modelPrototype`. Use `Repository#model` instead. * IMPORTANT CHANGE: by default, system collections are included in replication and all replication API return values. This will lead to user accounts and credentials data being replicated from master to slave servers. This may overwrite slave-specific database users. If this is undesired, the `_users` collection can be excluded from replication easily by setting the `includeSystem` attribute to `false` in the following commands: * replication.sync({ includeSystem: false }); * replication.applier.properties({ includeSystem: false }); This will exclude all system collections (including `_aqlfunctions`, `_graphs` etc.) from the initial synchronization and the continuous replication. If this is also undesired, it is also possible to specify a list of collections to exclude from the initial synchronization and the continuous replication using the `restrictCollections` attribute, e.g.: replication.applier.properties({ includeSystem: true, restrictType: "exclude", restrictCollections: [ "_users", "_graphs", "foo" ] }); The HTTP API methods for fetching the replication inventory and for dumping collections also support the `includeSystem` control flag via a URL parameter. * removed DEPRECATED replication methods: * `replication.logger.start()` * `replication.logger.stop()` * `replication.logger.properties()` * HTTP PUT `/_api/replication/logger-start` * HTTP PUT `/_api/replication/logger-stop` * HTTP GET `/_api/replication/logger-config` * HTTP PUT `/_api/replication/logger-config` * fixed issue #1174, which was due to locking problems in distributed AQL execution * improved cluster locking for AQL avoiding deadlocks * use DistributeNode for modifying queries with REPLACE and UPDATE, if possible v2.3.6 (2015-XX-XX) ------------------- * fixed AQL subquery optimization that produced wrong result when multiple subqueries directly followed each other and a directly following `LET` statement did refer to any but the first subquery. v2.3.5 (2015-01-16) ------------------- * fixed intermittent 404 errors in Foxx apps after mounting or unmounting apps * fixed issue #1200: Expansion operator results in "Cannot call method 'forEach' of null" * fixed issue #1199: Cannot unlink root node of plan v2.3.4 (2014-12-23) ------------------- * fixed cerberus path for MyArangoDB v2.3.3 (2014-12-17) ------------------- * fixed error handling in instantiation of distributed AQL queries, this also fixes a bug in cluster startup with many servers * issue #1185: parse non-fractional JSON numbers with exponent (e.g. `4e-261`) * issue #1159: allow --server.request-timeout and --server.connect-timeout of 0 v2.3.2 (2014-12-09) ------------------- * fixed issue #1177: Fix bug in the user app's storage * fixed issue #1173: AQL Editor "Save current query" resets user password * fixed missing makeDirectory when fetching a Foxx application from a zip file * put in warning about default changed: fixed issue #1134: Change the default endpoint to localhost * fixed issue #1163: invalid fullCount value returned from AQL * fixed range operator precedence * limit default maximum number of plans created by AQL optimizer to 256 (from 1024) * make AQL optimizer not generate an extra plan if an index can be used, but modify existing plans in place * fixed AQL cursor ttl (time-to-live) issue Any user-specified cursor ttl value was not honored since 2.3.0. * fixed segfault in AQL query hash index setup with unknown shapes * fixed memleaks * added AQL optimizer rule for removing `INTO` from a `COLLECT` statement if not needed * fixed issue #1131 This change provides the `KEEP` clause for `COLLECT ... INTO`. The `KEEP` clause allows controlling which variables will be kept in the variable created by `INTO`. * fixed issue #1147, must protect dispatcher ID for etcd v2.3.1 (2014-11-28) ------------------- * recreate password if missing during upgrade * fixed issue #1126 * fixed non-working subquery index optimizations * do not restrict summary of Foxx applications to 60 characters * fixed display of "required" path parameters in Foxx application documentation * added more optimizations of constants values in AQL FILTER conditions * fixed invalid or-to-in optimization for FILTERs containing comparisons with boolean values * fixed replication of `_graphs` collection * added AQL list functions `PUSH`, `POP`, `UNSHIFT`, `SHIFT`, `REMOVE_VALUES`, `REMOVE_VALUE`, `REMOVE_NTH` and `APPEND` * added AQL functions `CALL` and `APPLY` to dynamically call other functions * fixed AQL optimizer cost estimation for LIMIT node * prevent Foxx queues from permanently writing to the journal even when server is idle * fixed AQL COLLECT statement with INTO clause, which copied more variables than v2.2 and thus lead to too much memory consumption. This deals with #1107. * fixed AQL COLLECT statement, this concerned every COLLECT statement, only the first group had access to the values of the variables before the COLLECT statement. This deals with #1127. * fixed some AQL internals, where sometimes too many items were fetched from upstream in the presence of a LIMIT clause. This should generally improve performance. v2.3.0 (2014-11-18) ------------------- * fixed syslog flags. `--log.syslog` is deprecated and setting it has no effect, `--log.facility` now works as described. Application name has been changed from `triagens` to `arangod`. It can be changed using `--log.application`. The syslog will only contain the actual log message. The datetime prefix is omitted. * fixed deflate in SimpleHttpClient * fixed issue #1104: edgeExamples broken or changed * fixed issue #1103: Error while importing user queries * fixed issue #1100: AQL: HAS() fails on doc[attribute_name] * fixed issue #1098: runtime error when creating graph vertex * hide system applications in **Applications** tab by default Display of system applications can be toggled by using the *system applications* toggle in the UI. * added HTTP REST API for managing tasks (`/_api/tasks`) * allow passing character lists as optional parameter to AQL functions `TRIM`, `LTRIM` and `RTRIM` These functions now support trimming using custom character lists. If no character lists are specified, all whitespace characters will be removed as previously: TRIM(" foobar\t \r\n ") // "foobar" TRIM(";foo;bar;baz, ", "; ") // "foo;bar;baz" * added AQL string functions `LTRIM`, `RTRIM`, `FIND_FIRST`, `FIND_LAST`, `SPLIT`, `SUBSTITUTE` * added AQL functions `ZIP`, `VALUES` and `PERCENTILE` * made AQL functions `CONCAT` and `CONCAT_SEPARATOR` work with list arguments * dynamically create extra dispatcher threads if required * fixed issue #1097: schemas in the API docs no longer show required properties as optional v2.3.0-beta2 (2014-11-08) ------------------------- * front-end: new icons for uploading and downloading JSON documents into a collection * front-end: fixed documents pagination css display error * front-end: fixed flickering of the progress view * front-end: fixed missing event for documents filter function * front-end: jsoneditor: added CMD+Return (Mac) CTRL+Return (Linux/Win) shortkey for saving a document * front-end: added information tooltip for uploading json documents. * front-end: added database management view to the collapsed navigation menu * front-end: added collection truncation feature * fixed issue #1086: arangoimp: Odd errors if arguments are not given properly * performance improvements for AQL queries that use JavaScript-based expressions internally * added AQL geo functions `WITHIN_RECTANGLE` and `IS_IN_POLYGON` * fixed non-working query results download in AQL editor of web interface * removed debug print message in AQL editor query export routine * fixed issue #1075: Aardvark: user name required even if auth is off #1075 The fix for this prefills the username input field with the current user's account name if any and `root` (the default username) otherwise. Additionally, the tooltip text has been slightly adjusted. * fixed issue #1069: Add 'raw' link to swagger ui so that the raw swagger json can easily be retrieved This adds a link to the Swagger API docs to an application's detail view in the **Applications** tab of the web interface. The link produces the Swagger JSON directly. If authentication is turned on, the link requires authentication, too. * documentation updates v2.3.0-beta1 (2014-11-01) ------------------------- * added dedicated `NOT IN` operator for AQL Previously, a `NOT IN` was only achievable by writing a negated `IN` condition: FOR i IN ... FILTER ! (i IN [ 23, 42 ]) ... This can now alternatively be expressed more intuitively as follows: FOR i IN ... FILTER i NOT IN [ 23, 42 ] ... * added alternative logical operator syntax for AQL Previously, the logical operators in AQL could only be written as: - `&&`: logical and - `||`: logical or - `!`: negation ArangoDB 2.3 introduces the alternative variants for these operators: - `AND`: logical and - `OR`: logical or - `NOT`: negation The new syntax is just an alternative to the old syntax, allowing easier migration from SQL. The old syntax is still fully supported and will be. * improved output of `ArangoStatement.parse()` and POST `/_api/query` If an AQL query can be parsed without problems, The return value of `ArangoStatement.parse()` now contains an attribute `ast` with the abstract syntax tree of the query (before optimizations). Though this is an internal representation of the query and is subject to change, it can be used to inspect how ArangoDB interprets a given query. * improved `ArangoStatement.explain()` and POST `/_api/explain` The commands for explaining AQL queries have been improved. * added command-line option `--javascript.v8-contexts` to control the number of V8 contexts created in arangod. Previously, the number of V8 contexts was equal to the number of server threads (as specified by option `--server.threads`). However, it may be sensible to create different amounts of threads and V8 contexts. If the option is not specified, the number of V8 contexts created will be equal to the number of server threads. Thus no change in configuration is required to keep the old behavior. If you are using the default config files or merge them with your local config files, please review if the default number of server threads is okay in your environment. Additionally you should verify that the number of V8 contexts created (as specified in option `--javascript.v8-contexts`) is okay. * the number of server.threads specified is now the minimum of threads started. There are situation in which threads are waiting for results of distributed database servers. In this case the number of threads is dynamically increased. * removed index type "bitarray" Bitarray indexes were only half-way documented and integrated in previous versions of ArangoDB so their benefit was limited. The support for bitarray indexes has thus been removed in ArangoDB 2.3. It is not possible to create indexes of type "bitarray" with ArangoDB 2.3. When a collection is opened that contains a bitarray index definition created with a previous version of ArangoDB, ArangoDB will ignore it and log the following warning: index type 'bitarray' is not supported in this version of ArangoDB and is ignored Future versions of ArangoDB may automatically remove such index definitions so the warnings will eventually disappear. * removed internal "_admin/modules/flush" in order to fix requireApp * added basic support for handling binary data in Foxx Requests with binary payload can be processed in Foxx applications by using the new method `res.rawBodyBuffer()`. This will return the unparsed request body as a Buffer object. There is now also the method `req.requestParts()` available in Foxx to retrieve the individual components of a multipart HTTP request. Buffer objects can now be used when setting the response body of any Foxx action. Additionally, `res.send()` has been added as a convenience method for returning strings, JSON objects or buffers from a Foxx action: res.send("

some HTML

"); res.send({ success: true }); res.send(new Buffer("some binary data")); The convenience method `res.sendFile()` can now be used to easily return the contents of a file from a Foxx action: res.sendFile(applicationContext.foxxFilename("image.png")); `fs.write` now accepts not only strings but also Buffer objects as second parameter: fs.write(filename, "some data"); fs.write(filename, new Buffer("some binary data")); `fs.readBuffer` can be used to return the contents of a file in a Buffer object. * improved performance of insertion into non-unique hash indexes significantly in case many duplicate keys are used in the index * issue #1042: set time zone in log output the command-line option `--log.use-local-time` was added to print dates and times in the server-local timezone instead of UTC * command-line options that require a boolean value now validate the value given on the command-line This prevents issues if no value is specified for an option that requires a boolean value. For example, the following command-line would have caused trouble in 2.2, because `--server.endpoint` would have been used as the value for the `--server.disable-authentication` options (which requires a boolean value): arangod --server.disable-authentication --server.endpoint tcp://127.0.0.1:8529 data In 2.3, running this command will fail with an error and requires to be modified to: arangod --server.disable-authentication true --server.endpoint tcp://127.0.0.1:8529 data * improved performance of CSV import in arangoimp * fixed issue #1027: Stack traces are off-by-one * fixed issue #1026: Modules loaded in different files within the same app should refer to the same module * fixed issue #1025: Traversal not as expected in undirected graph * added a _relation function in the general-graph module. This deprecated _directedRelation and _undirectedRelation. ArangoDB does not offer any constraints for undirected edges which caused some confusion of users how undirected relations have to be handled. Relation now only supports directed relations and the user can actively simulate undirected relations. * changed return value of Foxx.applicationContext#collectionName: Previously, the function could return invalid collection names because invalid characters were not replaced in the application name prefix, only in the collection name passed. Now, the function replaces invalid characters also in the application name prefix, which might to slightly different results for application names that contained any characters outside the ranges [a-z], [A-Z] and [0-9]. * prevent XSS in AQL editor and logs view * integrated tutorial into ArangoShell and web interface * added option `--backslash-escape` for arangoimp when running CSV file imports * front-end: added download feature for (filtered) documents * front-end: added download feature for the results of a user query * front-end: added function to move documents to another collection * front-end: added sort-by attribute to the documents filter * front-end: added sorting feature to database, graph management and user management view. * issue #989: front-end: Databases view not refreshing after deleting a database * issue #991: front-end: Database search broken * front-end: added infobox which shows more information about a document (_id, _rev, _key) or an edge (_id, _rev, _key, _from, _to). The from and to attributes are clickable and redirect to their document location. * front-end: added edit-mode for deleting multiple documents at the same time. * front-end: added delete button to the detailed document/edge view. * front-end: added visual feedback for saving documents/edges inside the editor (error/success). * front-end: added auto-focusing for the first input field in a modal. * front-end: added validation for user input in a modal. * front-end: user defined queries are now stored inside the database and are bound to the current user, instead of using the local storage functionality of the browsers. The outcome of this is that user defined queries are now independently usable from any device. Also queries can now be edited through the standard document editor of the front-end through the _users collection. * front-end: added import and export functionality for user defined queries. * front-end: added new keywords and functions to the aql-editor theme * front-end: applied tile-style to the graph view * front-end: now using the new graph api including multi-collection support * front-end: foxx apps are now deletable * front-end: foxx apps are now installable and updateable through github, if github is their origin. * front-end: added foxx app version control. Multiple versions of a single foxx app are now installable and easy to manage and are also arranged in groups. * front-end: the user-set filter of a collection is now stored until the user navigates to another collection. * front-end: fetching and filtering of documents, statistics, and query operations are now handled with asynchronous ajax calls. * front-end: added progress indicator if the front-end is waiting for a server operation. * front-end: fixed wrong count of documents in the documents view of a collection. * front-end: fixed unexpected styling of the manage db view and navigation. * front-end: fixed wrong handling of select fields in a modal view. * front-end: fixed wrong positioning of some tooltips. * automatically call `toJSON` function of JavaScript objects (if present) when serializing them into database documents. This change allows storing JavaScript date objects in the database in a sensible manner. v2.2.7 (2014-11-19) ------------------- * fixed issue #998: Incorrect application URL for non-system Foxx apps * fixed issue #1079: AQL editor: keyword WITH in UPDATE query is not highlighted * fix memory leak in cluster nodes * fixed registration of AQL user-defined functions in Web UI (JS shell) * fixed error display in Web UI for certain errors (now error message is printed instead of 'undefined') * fixed issue #1059: bug in js module console * fixed issue #1056: "fs": zip functions fail with passwords * fixed issue #1063: Docs: measuring unit of --wal.logfile-size? * fixed issue #1062: Docs: typo in 14.2 Example data v2.2.6 (2014-10-20) ------------------- * fixed issue #972: Compilation Issue * fixed issue #743: temporary directories are now unique and one can read off the tool that created them, if empty, they are removed atexit * Highly improved performance of all AQL GRAPH_* functions. * Orphan collections in general graphs can now be found via GRAPH_VERTICES if either "any" or no direction is defined * Fixed documentation for AQL function GRAPH_NEIGHBORS. The option "vertexCollectionRestriction" is meant to filter the target vertices only, and should not filter the path. * Fixed a bug in GRAPH_NEIGHBORS which enforced only empty results under certain conditions v2.2.5 (2014-10-09) ------------------- * fixed issue #961: allow non-JSON values in undocument request bodies * fixed issue 1028: libicu is now statically linked * fixed cached lookups of collections on the server, which may have caused spurious problems after collection rename operations v2.2.4 (2014-10-01) ------------------- * fixed accessing `_from` and `_to` attributes in `collection.byExample` and `collection.firstExample` These internal attributes were not handled properly in the mentioned functions, so searching for them did not always produce documents * fixed issue #1030: arangoimp 2.2.3 crashing, not logging on large Windows CSV file * fixed issue #1025: Traversal not as expected in undirected graph * fixed issue #1020 This requires re-introducing the startup option `--database.force-sync-properties`. This option can again be used to force fsyncs of collection, index and database properties stored as JSON strings on disk in files named `parameter.json`. Syncing these files after a write may be necessary if the underlying storage does not sync file contents by itself in a "sensible" amount of time after a file has been written and closed. The default value is `true` so collection, index and database properties will always be synced to disk immediately. This affects creating, renaming and dropping collections as well as creating and dropping databases and indexes. Each of these operations will perform an additional fsync on the `parameter.json` file if the option is set to `true`. It might be sensible to set this option to `false` for workloads that create and drop a lot of collections (e.g. test runs). Document operations such as creating, updating and dropping documents are not affected by this option. * fixed issue #1016: AQL editor bug * fixed issue #1014: WITHIN function returns wrong distance * fixed AQL shortest path calculation in function `GRAPH_SHORTEST_PATH` to return complete vertex objects instead of just vertex ids * allow changing of attributes of documents stored in server-side JavaScript variables Previously, the following did not work: var doc = db.collection.document(key); doc._key = "abc"; // overwriting internal attributes not supported doc.value = 123; // overwriting existing attributes not supported Now, modifying documents stored in server-side variables (e.g. `doc` in the above case) is supported. Modifying the variables will not update the documents in the database, but will modify the JavaScript object (which can be written back to the database using `db.collection.update` or `db.collection.replace`) * fixed issue #997: arangoimp apparently doesn't support files >2gig on Windows large file support (requires using `_stat64` instead of `stat`) is now supported on Windows v2.2.3 (2014-09-02) ------------------- * added `around` for Foxx controller * added `type` option for HTTP API `GET /_api/document?collection=...` This allows controlling the type of results to be returned. By default, paths to documents will be returned, e.g. [ `/_api/document/test/mykey1`, `/_api/document/test/mykey2`, ... ] To return a list of document ids instead of paths, the `type` URL parameter can be set to `id`: [ `test/mykey1`, `test/mykey2`, ... ] To return a list of document keys only, the `type` URL parameter can be set to `key`: [ `mykey1`, `mykey2`, ... ] * properly capitalize HTTP response header field names in case the `x-arango-async` HTTP header was used in a request. * fixed several documentation issues * speedup for several general-graph functions, AQL functions starting with `GRAPH_` and traversals v2.2.2 (2014-08-08) ------------------- * allow storing non-reserved attribute names starting with an underscore Previous versions of ArangoDB parsed away all attribute names that started with an underscore (e.g. `_test', '_foo', `_bar`) on all levels of a document (root level and sub-attribute levels). While this behavior was documented, it was unintuitive and prevented storing documents inside other documents, e.g.: { "_key" : "foo", "_type" : "mydoc", "references" : [ { "_key" : "something", "_rev" : "...", "value" : 1 }, { "_key" : "something else", "_rev" : "...", "value" : 2 } ] } In the above example, previous versions of ArangoDB removed all attributes and sub-attributes that started with underscores, meaning the embedded documents would lose some of their attributes. 2.2.2 should preserve such attributes, and will also allow storing user-defined attribute names on the top-level even if they start with underscores (such as `_type` in the above example). * fix conversion of JavaScript String, Number and Boolean objects to JSON. Objects created in JavaScript using `new Number(...)`, `new String(...)`, or `new Boolean(...)` were not converted to JSON correctly. * fixed a race condition on task registration (i.e. `require("org/arangodb/tasks").register()`) this race condition led to undefined behavior when a just-created task with no offset and no period was instantly executed and deleted by the task scheduler, before the `register` function returned to the caller. * changed run-tests.sh to execute all suitable tests. * switch to new version of gyp * fixed upgrade button v2.2.1 (2014-07-24) ------------------- * fixed hanging write-ahead log recovery for certain cases that involved dropping databases * fixed issue with --check-version: when creating a new database the check failed * issue #947 Foxx applicationContext missing some properties * fixed issue with --check-version: when creating a new database the check failed * added startup option `--wal.suppress-shape-information` Setting this option to `true` will reduce memory and disk space usage and require less CPU time when modifying documents or edges. It should therefore be turned on for standalone ArangoDB servers. However, for servers that are used as replication masters, setting this option to `true` will effectively disable the usage of the write-ahead log for replication, so it should be set to `false` for any replication master servers. The default value for this option is `false`. * added optional `ttl` attribute to specify result cursor expiration for HTTP API method `POST /_api/cursor` The `ttl` attribute can be used to prevent cursor results from timing out too early. * issue #947: Foxx applicationContext missing some properties * (reported by Christian Neubauer): The problem was that in Google's V8, signed and unsigned chars are not always declared cleanly. so we need to force v8 to compile with forced signed chars which is done by the Flag: -fsigned-char at least it is enough to follow the instructions of compiling arango on raspberry and add "CFLAGS='-fsigned-char'" to the make command of V8 and remove the armv7=0 * Fixed a bug with the replication client. In the case of single document transactions the collection was not write locked. v2.2.0 (2014-07-10) ------------------- * The replication methods `logger.start`, `logger.stop` and `logger.properties` are no-ops in ArangoDB 2.2 as there is no separate replication logger anymore. Data changes are logged into the write-ahead log in ArangoDB 2.2, and not separately by the replication logger. The replication logger object is still there in ArangoDB 2.2 to ensure backwards-compatibility, however, logging cannot be started, stopped or configured anymore. Using any of these methods will do nothing. This also affects the following HTTP API methods: - `PUT /_api/replication/logger-start` - `PUT /_api/replication/logger-stop` - `GET /_api/replication/logger-config` - `PUT /_api/replication/logger-config` Using any of these methods is discouraged from now on as they will be removed in future versions of ArangoDB. * INCOMPATIBLE CHANGE: replication of transactions has changed. Previously, transactions were logged on a master in one big block and shipped to a slave in one block, too. Now transactions will be logged and replicated as separate entries, allowing transactions to be bigger and also ensure replication progress. This change also affects the behavior of the `stop` method of the replication applier. If the replication applier is now stopped manually using the `stop` method and later restarted using the `start` method, any transactions that were unfinished at the point of stopping will be aborted on a slave, even if they later commit on the master. In ArangoDB 2.2, stopping the replication applier manually should be avoided unless the goal is to stop replication permanently or to do a full resync with the master anyway. If the replication applier still must be stopped, it should be made sure that the slave has fetched and applied all pending operations from a master, and that no extra transactions are started on the master before the `stop` command on the slave is executed. Replication of transactions in ArangoDB 2.2 might also lock the involved collections on the slave while a transaction is either committed or aborted on the master and the change has been replicated to the slave. This change in behavior may be important for slave servers that are used for read-scaling. In order to avoid long lasting collection locks on the slave, transactions should be kept small. The `_replication` system collection is not used anymore in ArangoDB 2.2 and its usage is discouraged. * INCOMPATIBLE CHANGE: the figures reported by the `collection.figures` method now only reflect documents and data contained in the journals and datafiles of collections. Documents or deletions contained only in the write-ahead log will not influence collection figures until the write-ahead log garbage collection kicks in. The figures for a collection might therefore underreport the total resource usage of a collection. Additionally, the attributes `lastTick` and `uncollectedLogfileEntries` have been added to the result of the `figures` operation and the HTTP API method `PUT /_api/collection/figures` * added `insert` method as an alias for `save`. Documents can now be inserted into a collection using either method: db.test.save({ foo: "bar" }); db.test.insert({ foo: "bar" }); * added support for data-modification AQL queries * added AQL keywords `INSERT`, `UPDATE`, `REPLACE` and `REMOVE` (and `WITH`) to support data-modification AQL queries. Unquoted usage of these keywords for attribute names in AQL queries will likely fail in ArangoDB 2.2. If any such attribute name needs to be used in a query, it should be enclosed in backticks to indicate the usage of a literal attribute name. For example, the following query will fail in ArangoDB 2.2 with a parse error: FOR i IN foo RETURN i.remove and needs to be rewritten like this: FOR i IN foo RETURN i.`remove` * disallow storing of JavaScript objects that contain JavaScript native objects of type `Date`, `Function`, `RegExp` or `External`, e.g. db.test.save({ foo: /bar/ }); db.test.save({ foo: new Date() }); will now print Error: cannot be converted into JSON shape: could not shape document Previously, objects of these types were silently converted into an empty object (i.e. `{ }`). To store such objects in a collection, explicitly convert them into strings like this: db.test.save({ foo: String(/bar/) }); db.test.save({ foo: String(new Date()) }); * The replication methods `logger.start`, `logger.stop` and `logger.properties` are no-ops in ArangoDB 2.2 as there is no separate replication logger anymore. Data changes are logged into the write-ahead log in ArangoDB 2.2, and not separately by the replication logger. The replication logger object is still there in ArangoDB 2.2 to ensure backwards-compatibility, however, logging cannot be started, stopped or configured anymore. Using any of these methods will do nothing. This also affects the following HTTP API methods: - `PUT /_api/replication/logger-start` - `PUT /_api/replication/logger-stop` - `GET /_api/replication/logger-config` - `PUT /_api/replication/logger-config` Using any of these methods is discouraged from now on as they will be removed in future versions of ArangoDB. * INCOMPATIBLE CHANGE: replication of transactions has changed. Previously, transactions were logged on a master in one big block and shipped to a slave in one block, too. Now transactions will be logged and replicated as separate entries, allowing transactions to be bigger and also ensure replication progress. This change also affects the behavior of the `stop` method of the replication applier. If the replication applier is now stopped manually using the `stop` method and later restarted using the `start` method, any transactions that were unfinished at the point of stopping will be aborted on a slave, even if they later commit on the master. In ArangoDB 2.2, stopping the replication applier manually should be avoided unless the goal is to stop replication permanently or to do a full resync with the master anyway. If the replication applier still must be stopped, it should be made sure that the slave has fetched and applied all pending operations from a master, and that no extra transactions are started on the master before the `stop` command on the slave is executed. Replication of transactions in ArangoDB 2.2 might also lock the involved collections on the slave while a transaction is either committed or aborted on the master and the change has been replicated to the slave. This change in behavior may be important for slave servers that are used for read-scaling. In order to avoid long lasting collection locks on the slave, transactions should be kept small. The `_replication` system collection is not used anymore in ArangoDB 2.2 and its usage is discouraged. * INCOMPATIBLE CHANGE: the figures reported by the `collection.figures` method now only reflect documents and data contained in the journals and datafiles of collections. Documents or deletions contained only in the write-ahead log will not influence collection figures until the write-ahead log garbage collection kicks in. The figures for a collection might therefore underreport the total resource usage of a collection. Additionally, the attributes `lastTick` and `uncollectedLogfileEntries` have been added to the result of the `figures` operation and the HTTP API method `PUT /_api/collection/figures` * added `insert` method as an alias for `save`. Documents can now be inserted into a collection using either method: db.test.save({ foo: "bar" }); db.test.insert({ foo: "bar" }); * added support for data-modification AQL queries * added AQL keywords `INSERT`, `UPDATE`, `REPLACE` and `REMOVE` (and `WITH`) to support data-modification AQL queries. Unquoted usage of these keywords for attribute names in AQL queries will likely fail in ArangoDB 2.2. If any such attribute name needs to be used in a query, it should be enclosed in backticks to indicate the usage of a literal attribute name. For example, the following query will fail in ArangoDB 2.2 with a parse error: FOR i IN foo RETURN i.remove and needs to be rewritten like this: FOR i IN foo RETURN i.`remove` * disallow storing of JavaScript objects that contain JavaScript native objects of type `Date`, `Function`, `RegExp` or `External`, e.g. db.test.save({ foo: /bar/ }); db.test.save({ foo: new Date() }); will now print Error: cannot be converted into JSON shape: could not shape document Previously, objects of these types were silently converted into an empty object (i.e. `{ }`). To store such objects in a collection, explicitly convert them into strings like this: db.test.save({ foo: String(/bar/) }); db.test.save({ foo: String(new Date()) }); * honor startup option `--server.disable-statistics` when deciding whether or not to start periodic statistics collection jobs Previously, the statistics collection jobs were started even if the server was started with the `--server.disable-statistics` flag being set to `true` * removed startup option `--random.no-seed` This option had no effect in previous versions of ArangoDB and was thus removed. * removed startup option `--database.remove-on-drop` This option was used for debugging only. * removed startup option `--database.force-sync-properties` This option is now superfluous as collection properties are now stored in the write-ahead log. * introduced write-ahead log All write operations in an ArangoDB server instance are automatically logged to the server's write-ahead log. The write-ahead log is a set of append-only logfiles, and it is used in case of a crash recovery and for replication. Data from the write-ahead log will eventually be moved into the journals or datafiles of collections, allowing the server to remove older write-ahead log logfiles. Figures of collections will be updated when data are moved from the write-ahead log into the journals or datafiles of collections. Cross-collection transactions in ArangoDB should benefit considerably by this change, as less writes than in previous versions are required to ensure the data of multiple collections are atomically and durably committed. All data-modifying operations inside transactions (insert, update, remove) will write their operations into the write-ahead log directly, making transactions with multiple operations also require less physical memory than in previous versions of ArangoDB, that required all transaction data to fit into RAM. The `_trx` system collection is not used anymore in ArangoDB 2.2 and its usage is discouraged. The data in the write-ahead log can also be used in the replication context. The `_replication` collection that was used in previous versions of ArangoDB to store all changes on the server is not used anymore in ArangoDB 2.2. Instead, slaves can read from a master's write-ahead log to get informed about most recent changes. This removes the need to store data-modifying operations in both the actual place and the `_replication` collection. * removed startup option `--server.disable-replication-logger` This option is superfluous in ArangoDB 2.2. There is no dedicated replication logger in ArangoDB 2.2. There is now always the write-ahead log, and it is also used as the server's replication log. Specifying the startup option `--server.disable-replication-logger` will do nothing in ArangoDB 2.2, but the option should not be used anymore as it might be removed in a future version. * changed behavior of replication logger There is no dedicated replication logger in ArangoDB 2.2 as there is the write-ahead log now. The existing APIs for starting and stopping the replication logger still exist in ArangoDB 2.2 for downwards-compatibility, but calling the start or stop operations are no-ops in ArangoDB 2.2. When querying the replication logger status via the API, the server will always report that the replication logger is running. Configuring the replication logger is a no-op in ArangoDB 2.2, too. Changing the replication logger configuration has no effect. Instead, the write-ahead log configuration can be changed. * removed MRuby integration for arangod ArangoDB had an experimental MRuby integration in some of the publish builds. This wasn't continuously developed, and so it has been removed in ArangoDB 2.2. This change has led to the following startup options being superfluous: - `--ruby.gc-interval` - `--ruby.action-directory` - `--ruby.modules-path` - `--ruby.startup-directory` Specifying these startup options will do nothing in ArangoDB 2.2, but the options should be avoided from now on as they might be removed in future versions. * reclaim index memory when last document in collection is deleted Previously, deleting documents from a collection did not lead to index sizes being reduced. Instead, the already allocated index memory was re-used when a collection was refilled. Now, index memory for primary indexes and hash indexes is reclaimed instantly when the last document from a collection is removed. * inlined and optimized functions in hash indexes * added AQL TRANSLATE function This function can be used to perform lookups from static lists, e.g. LET countryNames = { US: "United States", UK: "United Kingdom", FR: "France" } RETURN TRANSLATE("FR", countryNames) * fixed datafile debugger * fixed check-version for empty directory * moved try/catch block to the top of routing chain * added mountedApp function for foxx-manager * fixed issue #883: arango 2.1 - when starting multi-machine cluster, UI web does not change to cluster overview * fixed dfdb: should not start any other V8 threads * cleanup of version-check, added module org/arangodb/database-version, added --check-version option * fixed issue #881: [2.1.0] Bombarded (every 10 sec or so) with "WARNING format string is corrupt" when in non-system DB Dashboard * specialized primary index implementation to allow faster hash table rebuilding and reduce lookups in datafiles for the actual value of `_key`. * issue #862: added `--overwrite` option to arangoimp * removed number of property lookups for documents during AQL queries that access documents * prevent buffering of long print results in arangosh's and arangod's print command this change will emit buffered intermediate print results and discard the output buffer to quickly deliver print results to the user, and to prevent constructing very large buffers for large results * removed sorting of attribute names for use in a collection's shaper sorting attribute names was done on document insert to keep attributes of a collection in sorted order for faster comparisons. The sort order of attributes was only used in one particular and unlikely case, so it was removed. Collections with many different attribute names should benefit from this change by faster inserts and slightly less memory usage. * fixed a bug in arangodump which got the collection name in _from and _to attributes of edges wrong (all were "_unknown") * fixed a bug in arangorestore which did not recognize wrong _from and _to attributes of edges * improved error detection and reporting in arangorestore v2.1.1 (2014-06-06) ------------------- * fixed dfdb: should not start any other V8 threads * signature for collection functions was modified The basic change was the substitution of the input parameter of the function by an generic options object which can contain multiple option parameter of the function. Following functions were modified remove removeBySample replace replaceBySample update updateBySample Old signature is yet supported but it will be removed in future versions v2.1.0 (2014-05-29) ------------------- * implemented upgrade procedure for clusters * fixed communication issue with agency which prevented reconnect after an agent failure * fixed cluster dashboard in the case that one but not all servers in the cluster are down * fixed a bug with coordinators creating local database objects in the wrong order (_system needs to be done first) * improved cluster dashboard v2.1.0-rc2 (2014-05-25) ----------------------- * fixed issue #864: Inconsistent behavior of AQL REVERSE(list) function v2.1.0-rc1 (XXXX-XX-XX) ----------------------- * added server-side periodic task management functions: - require("org/arangodb/tasks").register(): registers a periodic task - require("org/arangodb/tasks").unregister(): unregisters and removes a periodic task - require("org/arangodb/tasks").get(): retrieves a specific tasks or all existing tasks the previous undocumented function `internal.definePeriodic` is now deprecated and will be removed in a future release. * decrease the size of some seldom used system collections on creation. This will make these collections use less disk space and mapped memory. * added AQL date functions * added AQL FLATTEN() list function * added index memory statistics to `db..figures()` function The `figures` function will now return a sub-document `indexes`, which lists the number of indexes in the `count` sub-attribute, and the total memory usage of the indexes in bytes in the `size` sub-attribute. * added AQL CURRENT_DATABASE() function This function returns the current database's name. * added AQL CURRENT_USER() function This function returns the current user from an AQL query. The current user is the username that was specified in the `Authorization` HTTP header of the request. If authentication is turned off or the query was executed outside a request context, the function will return `null`. * fixed issue #796: Searching with newline chars broken? fixed slightly different handling of backslash escape characters in a few AQL functions. Now handling of escape sequences should be consistent, and searching for newline characters should work the same everywhere * added OpenSSL version check for configure It will report all OpenSSL versions < 1.0.1g as being too old. `configure` will only complain about an outdated OpenSSL version but not stop. * require C++ compiler support (requires g++ 4.8, clang++ 3.4 or Visual Studio 13) * less string copying returning JSONified documents from ArangoDB, e.g. via HTTP GET `/_api/document//` * issue #798: Lower case http headers from arango This change allows returning capitalized HTTP headers, e.g. `Content-Length` instead of `content-length`. The HTTP spec says that headers are case-insensitive, but in fact several clients rely on a specific case in response headers. This change will capitalize HTTP headers if the `X-Arango-Version` request header is sent by the client and contains a value of at least `20100` (for version 2.1). The default value for the compatibility can also be set at server start, using the `--server.default-api-compatibility` option. * simplified usage of `db._createStatement()` Previously, the function could not be called with a query string parameter as follows: db._createStatement(queryString); Calling it as above resulted in an error because the function expected an object as its parameter. From now on, it's possible to call the function with just the query string. * make ArangoDB not send back a `WWW-Authenticate` header to a client in case the client sends the `X-Omit-WWW-Authenticate` HTTP header. This is done to prevent browsers from showing their built-in HTTP authentication dialog for AJAX requests that require authentication. ArangoDB will still return an HTTP 401 (Unauthorized) if the request doesn't contain valid credentials, but it will omit the `WWW-Authenticate` header, allowing clients to bypass the browser's authentication dialog. * added REST API method HTTP GET `/_api/job/job-id` to query the status of an async job without potentially fetching it from the list of done jobs * fixed non-intuitive behavior in jobs API: previously, querying the status of an async job via the API HTTP PUT `/_api/job/job-id` removed a currently executing async job from the list of queryable jobs on the server. Now, when querying the result of an async job that is still executing, the job is kept in the list of queryable jobs so its result can be fetched by a subsequent request. * use a new data structure for the edge index of an edge collection. This improves the performance for the creation of the edge index and in particular speeds up removal of edges in graphs. Note however that this change might change the order in which edges starting at or ending in a vertex are returned. However, this order was never guaranteed anyway and it is not sensible to guarantee any particular order. * provide a size hint to edge and hash indexes when initially filling them this will lead to less re-allocations when populating these indexes this may speed up building indexes when opening an existing collection * don't requeue identical context methods in V8 threads in case a method is already registered * removed arangod command line option `--database.remove-on-compacted` * export the sort attribute for graph traversals to the HTTP interface * add support for arangodump/arangorestore for clusters v2.0.8 (XXXX-XX-XX) ------------------- * fixed too-busy iteration over skiplists Even when a skiplist query was restricted by a limit clause, the skiplist index was queried without the limit. this led to slower-than-necessary execution times. * fixed timeout overflows on 32 bit systems this bug has led to problems when select was called with a high timeout value (2000+ seconds) on 32bit systems that don't have a forgiving select implementation. when the call was made on these systems, select failed so no data would be read or sent over the connection this might have affected some cluster-internal operations. * fixed ETCD issues on 32 bit systems ETCD was non-functional on 32 bit systems at all. The first call to the watch API crashed it. This was because atomic operations worked on data structures that were not properly aligned on 32 bit systems. * fixed issue #848: db.someEdgeCollection.inEdge does not return correct value when called the 2nd time after a .save to the edge collection v2.0.7 (2014-05-05) ------------------- * issue #839: Foxx Manager missing "unfetch" * fixed a race condition at startup this fixes undefined behavior in case the logger was involved directly at startup, before the logger initialization code was called. This should have occurred only for code that was executed before the invocation of main(), e.g. during ctor calls of statically defined objects. v2.0.6 (2014-04-22) ------------------- * fixed issue #835: arangosh doesn't show correct database name v2.0.5 (2014-04-21) ------------------- * Fixed a caching problem in IE JS Shell * added cancelation for async jobs * upgraded to new gyp for V8 * new Windows installer v2.0.4 (2014-04-14) ------------------- * fixed cluster authentication front-end issues for Firefox and IE, there are still problems with Chrome v2.0.3 (2014-04-14) ------------------- * fixed AQL optimizer bug * fixed front-end issues * added password change dialog v2.0.2 (2014-04-06) ------------------- * during cluster startup, do not log (somewhat expected) connection errors with log level error, but with log level info * fixed dashboard modals * fixed connection check for cluster planning front end: firefox does not support async:false * document how to persist a cluster plan in order to relaunch an existing cluster later v2.0.1 (2014-03-31) ------------------- * make ArangoDB not send back a `WWW-Authenticate` header to a client in case the client sends the `X-Omit-WWW-Authenticate` HTTP header. This is done to prevent browsers from showing their built-in HTTP authentication dialog for AJAX requests that require authentication. ArangoDB will still return an HTTP 401 (Unauthorized) if the request doesn't contain valid credentials, but it will omit the `WWW-Authenticate` header, allowing clients to bypass the browser's authentication dialog. * fixed issues in arango-dfdb: the dfdb was not able to unload certain system collections, so these couldn't be inspected with the dfdb sometimes. Additionally, it did not truncate corrupt markers from datafiles under some circumstances * added `changePassword` attribute for users * fixed non-working "save" button in collection edit view of web interface clicking the save button did nothing. one had to press enter in one of the input fields to send modified form data * fixed V8 compile error on MacOS X * prevent `body length: -9223372036854775808` being logged in development mode for some Foxx HTTP responses * fixed several bugs in web interface dashboard * fixed issue #783: coffee script not working in manifest file * fixed issue #783: coffee script not working in manifest file * fixed issue #781: Cant save current query from AQL editor ui * bumped version in `X-Arango-Version` compatibility header sent by arangosh and other client tools from `1.5` to `2.0`. * fixed startup options for arango-dfdb, added details option for arango-dfdb * fixed display of missing error messages and codes in arangosh * when creating a collection via the web interface, the collection type was always "document", regardless of the user's choice v2.0.0 (2014-03-10) ------------------- * first 2.0 release v2.0.0-rc2 (2014-03-07) ----------------------- * fixed cluster authorization v2.0.0-rc1 (2014-02-28) ----------------------- * added sharding :-) * added collection._dbName attribute to query the name of the database from a collection more detailed documentation on the sharding and cluster features can be found in the user manual, section **Sharding** * INCOMPATIBLE CHANGE: using complex values in AQL filter conditions with operators other than equality (e.g. >=, >, <=, <) will disable usage of skiplist indexes for filter evaluation. For example, the following queries will be affected by change: FOR doc IN docs FILTER doc.value < { foo: "bar" } RETURN doc FOR doc IN docs FILTER doc.value >= [ 1, 2, 3 ] RETURN doc The following queries will not be affected by the change: FOR doc IN docs FILTER doc.value == 1 RETURN doc FOR doc IN docs FILTER doc.value == "foo" RETURN doc FOR doc IN docs FILTER doc.value == [ 1, 2, 3 ] RETURN doc FOR doc IN docs FILTER doc.value == { foo: "bar" } RETURN doc * INCOMPATIBLE CHANGE: removed undocumented method `collection.saveOrReplace` this feature was never advertised nor documented nor tested. * INCOMPATIBLE CHANGE: removed undocumented REST API method `/_api/simple/BY-EXAMPLE-HASH` this feature was never advertised nor documented nor tested. * added explicit startup parameter `--server.reuse-address` This flag can be used to control whether sockets should be acquired with the SO_REUSEADDR flag. Regardless of this setting, sockets on Windows are always acquired using the SO_EXCLUSIVEADDRUSE flag. * removed undocumented REST API method GET `/_admin/database-name` * added user validation API at POST `/_api/user/` * slightly improved users management API in `/_api/user`: Previously, when creating a new user via HTTP POST, the username needed to be passed in an attribute `username`. When users were returned via this API, the usernames were returned in an attribute named `user`. This was slightly confusing and was changed in 2.0 as follows: - when adding a user via HTTP POST, the username can be specified in an attribute `user`. If this attribute is not used, the API will look into the attribute `username` as before and use that value. - when users are returned via HTTP GET, the usernames are still returned in an attribute `user`. This change should be fully downwards-compatible with the previous version of the API. * added AQL SLICE function to extract slices from lists * made module loader more node compatible * the startup option `--javascript.package-path` for arangosh is now deprecated and does nothing. Using it will not cause an error, but the option is ignored. * added coffee script support * Several UI improvements. * Exchanged icons in the graphviewer toolbar * always start networking and HTTP listeners when starting the server (even in console mode) * allow vertex and edge filtering with user-defined functions in TRAVERSAL, TRAVERSAL_TREE and SHORTEST_PATH AQL functions: // using user-defined AQL functions for edge and vertex filtering RETURN TRAVERSAL(friends, friendrelations, "friends/john", "outbound", { followEdges: "myfunctions::checkedge", filterVertices: "myfunctions::checkvertex" }) // using the following custom filter functions var aqlfunctions = require("org/arangodb/aql/functions"); aqlfunctions.register("myfunctions::checkedge", function (config, vertex, edge, path) { return (edge.type !== 'dislikes'); // don't follow these edges }, false); aqlfunctions.register("myfunctions::checkvertex", function (config, vertex, path) { if (vertex.isDeleted || ! vertex.isActive) { return [ "prune", "exclude" ]; // exclude these and don't follow them } return [ ]; // include everything else }, false); * fail if invalid `strategy`, `order` or `itemOrder` attribute values are passed to the AQL TRAVERSAL function. Omitting these attributes is not considered an error, but specifying an invalid value for any of these attributes will make an AQL query fail. * issue #751: Create database through API should return HTTP status code 201 By default, the server now returns HTTP 201 (created) when creating a new database successfully. To keep compatibility with older ArangoDB versions, the startup parameter `--server.default-api-compatibility` can be set to a value of `10400` to indicate API compatibility with ArangoDB 1.4. The compatibility can also be enforced by setting the `X-Arango-Version` HTTP header in a client request to this API on a per-request basis. * allow direct access from the `db` object to collections whose names start with an underscore (e.g. db._users). Previously, access to such collections via the `db` object was possible from arangosh, but not from arangod (and thus Foxx and actions). The only way to access such collections from these places was via the `db._collection()` workaround. * allow `\n` (as well as `\r\n`) as line terminator in batch requests sent to `/_api/batch` HTTP API. * use `--data-binary` instead of `--data` parameter in generated cURL examples * issue #703: Also show path of logfile for fm.config() * issue #675: Dropping a collection used in "graph" module breaks the graph * added "static" Graph.drop() method for graphs API * fixed issue #695: arangosh server.password error * use pretty-printing in `--console` mode by default * simplified ArangoDB startup options Some startup options are now superfluous or their usage is simplified. The following options have been changed: * `--javascript.modules-path`: this option has been removed. The modules paths are determined by arangod and arangosh automatically based on the value of `--javascript.startup-directory`. If the option is set on startup, it is ignored so startup will not abort with an error `unrecognized option`. * `--javascript.action-directory`: this option has been removed. The actions directory is determined by arangod automatically based on the value of `--javascript.startup-directory`. If the option is set on startup, it is ignored so startup will not abort with an error `unrecognized option`. * `--javascript.package-path`: this option is still available but it is not required anymore to set the standard package paths (e.g. `js/npm`). arangod will automatically use this standard package path regardless of whether it was specified via the options. It is possible to use this option to add additional package paths to the standard value. Configuration files included with arangod are adjusted accordingly. * layout of the graphs tab adapted to better fit with the other tabs * database selection is moved to the bottom right corner of the web interface * removed priority queue index type this feature was never advertised nor documented nor tested. * display internal attributes in document source view of web interface * removed separate shape collections When upgrading to ArangoDB 2.0, existing collections will be converted to include shapes and attribute markers in the datafiles instead of using separate files for shapes. When a collection is converted, existing shapes from the SHAPES directory will be written to a new datafile in the collection directory, and the SHAPES directory will be removed afterwards. This saves up to 2 MB of memory and disk space for each collection (savings are higher, the less different shapes there are in a collection). Additionally, one less file descriptor per opened collection will be used. When creating a new collection, the amount of sync calls may be reduced. The same may be true for documents with yet-unknown shapes. This may help performance in these cases. * added AQL functions `NTH` and `POSITION` * added signal handler for arangosh to save last command in more cases * added extra prompt placeholders for arangosh: - `%e`: current endpoint - `%u`: current user * added arangosh option `--javascript.gc-interval` to control amount of garbage collection performed by arangosh * fixed issue #651: Allow addEdge() to take vertex ids in the JS library * removed command-line option `--log.format` In previous versions, this option did not have an effect for most log messages, so it got removed. * removed C++ logger implementation Logging inside ArangoDB is now done using the LOG_XXX() macros. The LOGGER_XXX() macros are gone. * added collection status "loading" v1.4.16 (XXXX-XX-XX) -------------------- * fixed too eager datafile deletion this issue could have caused a crash when the compaction had marked datafiles as obsolete and they were removed while "old" temporary query results still pointed to the old datafile positions * fixed issue #826: Replication fails when a collection's configuration changes v1.4.15 (2014-04-19) -------------------- * bugfix for AQL query optimizer the following type of query was too eagerly optimized, leading to errors in code-generation: LET a = (FOR i IN [] RETURN i) LET b = (FOR i IN [] RETURN i) RETURN 1 the problem occurred when both lists in the subqueries were empty. In this case invalid code was generated and the query couldn't be executed. v1.4.14 (2014-04-05) -------------------- * fixed race conditions during shape / attribute insertion A race condition could have led to spurious `cannot find attribute #xx` or `cannot find shape #xx` (where xx is a number) warning messages being logged by the server. This happened when a new attribute was inserted and at the same time was queried by another thread. Also fixed a race condition that may have occurred when a thread tried to access the shapes / attributes hash tables while they were resized. In this cases, the shape / attribute may have been hashed to a wrong slot. * fixed a memory barrier / cpu synchronization problem with libev, affecting Windows with Visual Studio 2013 (probably earlier versions are affected, too) The issue is described in detail here: http://lists.schmorp.de/pipermail/libev/2014q1/002318.html v1.4.13 (2014-03-14) -------------------- * added diagnostic output for Foxx application upload * allow dump & restore from ArangoDB 1.4 with an ArangoDB 2.0 server * allow startup options `temp-path` and `default-language` to be specified from the arangod configuration file and not only from the command line * fixed too eager compaction The compaction will now wait for several seconds before trying to re-compact the same collection. Additionally, some other limits have been introduced for the compaction. v1.4.12 (2014-03-05) -------------------- * fixed display bug in web interface which caused the following problems: - documents were displayed in web interface as being empty - document attributes view displayed many attributes with content "undefined" - document source view displayed many attributes with name "TYPEOF" and value "undefined" - an alert popping up in the browser with message "Datatables warning..." * re-introduced old-style read-write locks to supports Windows versions older than Windows 2008R2 and Windows 7. This should re-enable support for Windows Vista and Windows 2008. v1.4.11 (2014-02-27) -------------------- * added SHORTEST_PATH AQL function this calculates the shortest paths between two vertices, using the Dijkstra algorithm, employing a min-heap By default, ArangoDB does not know the distance between any two vertices and will use a default distance of 1. A custom distance function can be registered as an AQL user function to make the distance calculation use any document attributes or custom logic: RETURN SHORTEST_PATH(cities, motorways, "cities/CGN", "cities/MUC", "outbound", { paths: true, distance: "myfunctions::citydistance" }) // using the following custom distance function var aqlfunctions = require("org/arangodb/aql/functions"); aqlfunctions.register("myfunctions::distance", function (config, vertex1, vertex2, edge) { return Math.sqrt(Math.pow(vertex1.x - vertex2.x) + Math.pow(vertex1.y - vertex2.y)); }, false); * fixed bug in Graph.pathTo function * fixed small memleak in AQL optimizer * fixed access to potentially uninitialized variable when collection had a cap constraint v1.4.10 (2014-02-21) -------------------- * fixed graph constructor to allow graph with some parameter to be used * added node.js "events" and "stream" * updated npm packages * added loading of .json file * Fixed http return code in graph api with waitForSync parameter. * Fixed documentation in graph, simple and index api. * removed 2 tests due to change in ruby library. * issue #756: set access-control-expose-headers on CORS response the following headers are now whitelisted by ArangoDB in CORS responses: - etag - content-encoding - content-length - location - server - x-arango-errors - x-arango-async-id v1.4.9 (2014-02-07) ------------------- * return a document's current etag in response header for HTTP HEAD requests on documents that return an HTTP 412 (precondition failed) error. This allows retrieving the document's current revision easily. * added AQL function `SKIPLIST` to directly access skiplist indexes from AQL This is a shortcut method to use a skiplist index for retrieving specific documents in indexed order. The function capability is rather limited, but it may be used for several cases to speed up queries. The documents are returned in index order if only one condition is used. /* return all documents with mycollection.created > 12345678 */ FOR doc IN SKIPLIST(mycollection, { created: [[ '>', 12345678 ]] }) RETURN doc /* return first document with mycollection.created > 12345678 */ FOR doc IN SKIPLIST(mycollection, { created: [[ '>', 12345678 ]] }, 0, 1) RETURN doc /* return all documents with mycollection.created between 12345678 and 123456790 */ FOR doc IN SKIPLIST(mycollection, { created: [[ '>', 12345678 ], [ '<=', 123456790 ]] }) RETURN doc /* return all documents with mycollection.a equal 1 and .b equal 2 */ FOR doc IN SKIPLIST(mycollection, { a: [[ '==', 1 ]], b: [[ '==', 2 ]] }) RETURN doc The function requires a skiplist index with the exact same attributes to be present on the specified collection. All attributes present in the skiplist index must be specified in the conditions specified for the `SKIPLIST` function. Attribute declaration order is important, too: attributes must be specified in the same order in the condition as they have been declared in the skiplist index. * added command-line option `--server.disable-authentication-unix-sockets` with this option, authentication can be disabled for all requests coming in via UNIX domain sockets, enabling clients located on the same host as the ArangoDB server to connect without authentication. Other connections (e.g. TCP/IP) are not affected by this option. The default value for this option is `false`. Note: this option is only supported on platforms that support Unix domain sockets. * call global arangod instance destructor on shutdown * issue #755: TRAVERSAL does not use strategy, order and itemOrder options these options were not honored when configuring a traversal via the AQL TRAVERSAL function. Now, these options are used if specified. * allow vertex and edge filtering with user-defined functions in TRAVERSAL, TRAVERSAL_TREE and SHORTEST_PATH AQL functions: // using user-defined AQL functions for edge and vertex filtering RETURN TRAVERSAL(friends, friendrelations, "friends/john", "outbound", { followEdges: "myfunctions::checkedge", filterVertices: "myfunctions::checkvertex" }) // using the following custom filter functions var aqlfunctions = require("org/arangodb/aql/functions"); aqlfunctions.register("myfunctions::checkedge", function (config, vertex, edge, path) { return (edge.type !== 'dislikes'); // don't follow these edges }, false); aqlfunctions.register("myfunctions::checkvertex", function (config, vertex, path) { if (vertex.isDeleted || ! vertex.isActive) { return [ "prune", "exclude" ]; // exclude these and don't follow them } return [ ]; // include everything else }, false); * issue #748: add vertex filtering to AQL's TRAVERSAL[_TREE]() function v1.4.8 (2014-01-31) ------------------- * install foxx apps in the web interface * fixed a segfault in the import API v1.4.7 (2014-01-23) ------------------- * issue #744: Add usage example arangoimp from Command line * issue #738: added __dirname, __filename pseudo-globals. Fixes #733. (@by pluma) * mount all Foxx applications in system apps directory on startup v1.4.6 (2014-01-20) ------------------- * issue #736: AQL function to parse collection and key from document handle * added fm.rescan() method for Foxx-Manager * fixed issue #734: foxx cookie and route problem * added method `fm.configJson` for arangosh * include `startupPath` in result of API `/_api/foxx/config` v1.4.5 (2014-01-15) ------------------- * fixed issue #726: Alternate Windows Install Method * fixed issue #716: dpkg -P doesn't remove everything * fixed bugs in description of HTTP API `_api/index` * fixed issue #732: Rest API GET revision number * added missing documentation for several methods in HTTP API `/_api/edge/...` * fixed typos in description of HTTP API `_api/document` * defer evaluation of AQL subqueries and logical operators (lazy evaluation) * Updated font in WebFrontend, it now contains a version that renders properly on Windows * generally allow function return values as call parameters to AQL functions * fixed potential deadlock in global context method execution * added override file "arangod.conf.local" (and co) v1.4.4 (2013-12-24) ------------------- * uid and gid are now set in the scripts, there is no longer a separate config file for arangod when started from a script * foxx-manager is now an alias for arangosh * arango-dfdb is now an alias for arangod, moved from bin to sbin * changed from readline to linenoise for Windows * added --install-service and --uninstall-service for Windows * removed --daemon and --supervisor for Windows * arangosh and arangod now uses the config-file which maps the binary name, i. e. if you rename arangosh to foxx-manager it will use the config file foxx-manager.conf * fixed lock file for Windows * fixed issue #711, #687: foxx-manager throws internal errors * added `--server.ssl-protocol` option for client tools this allows connecting from arangosh, arangoimp, arangoimp etc. to an ArangoDB server that uses a non-default value for `--server.ssl-protocol`. The default value for the SSL protocol is 4 (TLSv1). If the server is configured to use a different protocol, it was not possible to connect to it with the client tools. * added more detailed request statistics This adds the number of async-executed HTTP requests plus the number of HTTP requests per individual HTTP method type. * added `--force` option for arangorestore this option allows continuing a restore operation even if the server reports errors in the middle of the restore operation * better error reporting for arangorestore in case the server returned an HTTP error, arangorestore previously reported this error as `internal error` without any details only. Now server-side errors are reported by arangorestore with the server's error message * include more system collections in dumps produced by arangodump previously some system collections were intentionally excluded from dumps, even if the dump was run with `--include-system-collections`. for example, the collections `_aal`, `_modules`, `_routing`, and `_users` were excluded. This makes sense in a replication context but not always in a dump context. When specifying `--include-system-collections`, arangodump will now include the above- mentioned collections in the dump, too. Some other system collections are still excluded even when the dump is run with `--include-system-collections`, for example `_replication` and `_trx`. * fixed issue #701: ArangoStatement undefined in arangosh * fixed typos in configuration files v1.4.3 (2013-11-25) ------------------- * fixed a segfault in the AQL optimizer, occurring when a constant non-list value was used on the right-hand side of an IN operator that had a collection attribute on the left-hand side * issue #662: Fixed access violation errors (crashes) in the Windows version, occurring under some circumstances when accessing databases with multiple clients in parallel * fixed issue #681: Problem with ArchLinux PKGBUILD configuration v1.4.2 (2013-11-20) ------------------- * fixed issue #669: Tiny documentation update * ported Windows version to use native Windows API SRWLocks (slim read-write locks) and condition variables instead of homemade versions MSDN states the following about the compatibility of SRWLocks and Condition Variables: Minimum supported client: Windows Server 2008 [desktop apps | Windows Store apps] Minimum supported server: Windows Vista [desktop apps | Windows Store apps] * fixed issue #662: ArangoDB on Windows hanging This fixes a deadlock issue that occurred on Windows when documents were written to a collection at the same time when some other thread tried to drop the collection. * fixed file-based logging in Windows the logger complained on startup if the specified log file already existed * fixed startup of server in daemon mode (`--daemon` startup option) * fixed a segfault in the AQL optimizer * issue #671: Method graph.measurement does not exist * changed Windows condition variable implementation to use Windows native condition variables This is an attempt to fix spurious Windows hangs as described in issue #662. * added documentation for JavaScript traversals * added --code-page command-line option for Windows version of arangosh * fixed a problem when creating edges via the web interface. The problem only occurred if a collection was created with type "document collection" via the web interface, and afterwards was dropped and re-created with type "edge collection". If the web interface page was not reloaded, the old collection type (document) was cached, making the subsequent creation of edges into the (seeming-to-be-document) collection fail. The fix is to not cache the collection type in the web interface. Users of an older version of the web interface can reload the collections page if they are affected. * fixed a caching problem in arangosh: if a collection was created using the web interface, and then removed via arangosh, arangosh did not actually drop the collection due to caching. Because the `drop` operation was not carried out, this caused misleading error messages when trying to re-create the collection (e.g. `cannot create collection: duplicate name`). * fixed ALT-introduced characters for arangosh console input on Windows The Windows readline port was not able to handle characters that are built using CTRL or ALT keys. Regular characters entered using the CTRL or ALT keys were silently swallowed and not passed to the terminal input handler. This did not seem to cause problems for the US keyboard layout, but was a severe issue for keyboard layouts that require the ALT (or ALT-GR) key to construct characters. For example, entering the character `{` with a German keyboard layout requires pressing ALT-GR + 9. * fixed issue #665: Hash/skiplist combo madness bit my ass this fixes a problem with missing/non-deterministic rollbacks of inserts in case of a unique constraint violation into a collection with multiple secondary indexes (with at least one of them unique) * fixed issue #664: ArangoDB installer on Windows requires drive c: * partly fixed issue #662: ArangoDB on Windows hanging This fixes dropping databases on Windows. In previous 1.4 versions on Windows, one shape collection file was not unloaded and removed when dropping