NOTICE: Please see https://arkime.com/faq#upgrading-moloch for upgrading info OpenSearch Versions: * Arkime >= 3.0.0 supports 1.x or 2.x Elasticsearch Versions: * Arkime >= 4.0.0 supports ES >= 7.10.0 or 8.x, not 9.x or later * Arkime >= 3.0.0 supports ES >= 7.10.0, not 8.x or later * Moloch >= 2.7.0 supports ES >= 7.4.0, not 8.x or later * Moloch >= 2.2.0 supports ES >= 6.8.0 or >= 7.1.0, not 8.x or later * Moloch >= 2.0.0 supports ES >= 6.7.0 or >= 7.1.0, not 8.x or later * Moloch >= 1.5.0 supports ES >= 5.5.0, 6.x, not 7.x or later * Moloch >= 1.0.0 supports ES >= 5.5.0, 6.x (not prod tested, only for new installs), not 7.x or later * Moloch >= 0.50.0 supports ES >= 5.5.0, not 6.x or later * Moloch >= 0.18.1 supports ES 2.4.x, >= 5.3.1 not 6.x or later Node Versions: * Arkime >= 5.0.2 requires NodeJS >= 18.15.0 or < 21 * Arkime >= 5.0.0 requires NodeJS >= 18.0.0 and < 19 * Arkime >= 4.2.0 requires NodeJS >= 16.0.0 and < 19 * Arkime >= 3.4.0 requires NodeJS >= 16.0.0 and < 17 * Arkime >= 3.0.0 requires NodeJS 14.x * Moloch >= 2.4.0 requires NodeJS 12.x * Moloch >= 2.0.0 requires NodeJS 10.x * Moloch >= 1.6.0 requires NodeJS 8.x, 8.12 or later * Moloch >= 1.0.0 requires NodeJS 8.x * Moloch >= 0.20.0 requires NodeJS 6.x * Moloch >= 0.18.1 requires NodeJS 4.x NOTICE: Restart wiseService before capture when upgrading NOTICE: Cross-cluster Shortcuts require you to not restart all your viewers at once after upgrading NOTICE: Create a parliament config file before upgrading (see https://arkime.com/settings#parliament and https://arkime.com/faq#how_do_i_upgrade_to_arkime_5) 5.0.2 2024/03/xx ## Release - #2667 support Node 20 ## Capture/Viewer - #2694 New DNS parser that captures all the answers, enable with dnsOutputAnswers=true (thanks @mcgillowen) ## Capture - #2674 Fix filelist not working in scheme mode - #2679 Cert.alt can be used in rules - #2699 Disable reader s3 download timeout ## Cont3xt - #2683 lock integration settings - #2240 snap to dates ## Viewer - #2668 fix pcap export with only default time range and no date param in url - #2680 add default user settings to viewer config https://arkime.com/settings#user-setting-defaults - #2681 fix unique of numerical fields - #2701 Make sure pcap reassembly doesn't starve viewer - #2704 Support viewUrl having a path - #2705 Support querying non Arkime indices, enable with queryExtraIndices (thanks @mmguero) 5.0.1 2024/02/20 ## Release - #2631 CyberChef 10.6.0 - #2648 Build for Ubuntu 24.04 - #2655 Support rpm fips installs again - #2558 Node 18.19.1 ## Capture - #2634 add esp packet stats (fixes #1116) - #2638 support readTruncatedPackets on live captures ## db.pl - #2633 noprompt outputs less warnings - #2639 fix init not working with large number of indices ## JA4+ - Fixed memory leak - Fixed JA4H issue with long cookies ## Parliament - #2645 Fixed issues not being detected - #2659 Fixed parliament crashing if userPrefix not set ## Viewer - #2632 fix field labels not expanding fully - #2637 fix session detail grip - #2668 fix pcap export with only default time range and no date param in url ## WISE - #2653 set a threatstream.indicator field 5.0.0 2024/02/06 ## BREAKING - #2297 s3Compression/simpleCompression now defaults to zstd - #2297 s3WriteGzip removed, use s3Compression=gzip for gzip instead of new zstd default - #2297 s3GapPacketPos defaults to TRUE - #2297 enablePacketDedup defaults to TRUE - #2299 #2308 authMode defaults to digest now - #2312 removed old v1 viewer APIs - #2349 parliament password removed, must configure common auth via the UI before upgrading or manually in the config file see [parliament](https://arkime.com/settings#parliament) and [how do I upgrade to 5](https://arkime.com/faq#how_do_i_upgrade_to_arkime_5) - #2402 WISE/tagger must now use http.request.FIELD/http.response.FIELD when referencing header defined with headers-http-request/headers-http-response - #2450 Centos 7 build no longers includes pfring support - #2453 Increase simpleCompressionBlockSize default to 64000 ## Release - #2448 zstd 1.5.5, nghttp2 1.57.0, maxmind 1.7.1, yara 4.2.3 - #2443 Centos 7, Ubuntu 18, Alpine use unofficial builds of node - #2543 node v18.19.0 - #2447 support building on alpine - #2549 use configure prefix more places (thanks @vpiserchia) - #2584 AL2023 & Ubuntu22.04 ARM builds ## All - #2316 programs support same config file formats (ini/json/yaml) and retrieval (file, elasticsearch) - #2419 json/yaml config file formats now allow arrays instead of comma/semi separated - #2299 #2308 authMode setting added - #2299 #2408 #2463 added authMode: basic, form, basic+form, basic+oidc, headerOnly, header+digest (same as header), header+basic - #2387 notifiers for parliament and arkime merged conflicts mitigated by appending "Parliament" to parliament notifiers - #2396 drop privileges is now AFTER http(s) list - #2509 add optional login message for form auth - #2511 new authOIDCScope setting - #2482 new logoutUrl setting - #2571 new scheme pcap reading - #2618 better error message when can't use OpenSearch/Elasticsearch on startup ## Capture - #2295 moloch converted to arkime - #2312 override ips can now set any field - #2312 overrideIpsFiles setting - #2314 packetDropIpsFiles setting - #2390 can have negative cert.validDays/cert.remainingDays (thanks @mcgillowen) - #2390 added cert.remainingSeconds/cert.remainingSeconds (thanks @mcgillowen) - #2390 cert.remainingDays is now based on the firstPacket of session instead of current time (thanks @mcgillowen) - #2409 JA4 support - #2409 JA3/JA4 support for smtp STARTTLS - #2297 always build zstd (except arch) - #2517 new custom-fields-remap feature - #2186 count the number of http methods per session - #2528 new oui.txt location, some names have changes, fixes #2347 - #2539 new tls:has_esni tag if the client hello has esni - #2553 fix rules range matching not working always - #2554 support fieldSet tcpflag rules - #2575 fix startup complaint about aliases, category, and transforms - #2576 support different dlt for pcap-over-ip - #2592 fix sometimes not identifying quic protocol correctly - #2600 add tls:has_ech tag (thanks @renini) - #2614 new kafka-config section - #2622 fix malicious quic packet crashing capture ## Cont3xt - #2121 new bulk UI and support for bulk queries - #2271 lots of keyboard shortcut improvements - #2383 new array syntax for links substitution - #2382 new OpenSearch/Elasticsearch integration (config file only) - #2441 new csv/json file/url/redis integration (config file only) - #2385 new viewRoles in config file per integration to control access - #2407 transfer ownership of resources - #2437 new csv/json data source supports - #2441 new redis data source support - #2507 demoMode added - #2527 skipChildren added - #2532 new wise integration - #2580 add links to integration search page from card - #2565 added punycode decoding ## db.pl - #2588 db.pl won't try and backup indices that don't exist - #2588 db.pl backup cont3xt indices ## ESProxy - #2483 #2484 support field updates/deletes ## Viewer - #2296 removed x-moloch-auth - #2392 files/history/stats now have cluster dropdown for multiviewer - #2402 http.request.FIELD and http.response.FIELD supported - #2404 add editor for resources - #2407 transfer ownership of resources - #2482 added uploadRoles to control who can upload - #2501 add defaultTimeRange setting - #2521 add footerTemplate setting - #2525 add [config setting](https://arkime.com/settings#spiViewCategoryOrder) to set spiview category order - #2523 resize session detail field label/values - #2552 added %URIEncodedText% for URI encoded substitution (thanks @vpiserchia) - #2574 fix longstanding issue with backslash search and SMB - #2601 patch cyberchef xss vuln (https://github.com/gchq/CyberChef/issues/1468) - #2606 zstd sometimes didn't read all packets - #2607 improved session detail display - #2621 session detail link a link now, multi select info column items now ## Parliament - #2377 dashboard-only mode removed, if you want users to just see the dashboard don't assign them the parliamentUser role - #2395 configuration is now stored in opensearch/elasticsearch - #2530 add Users page ## WISE - #2537 new urlScrapePrefix/urlScrapeSuffix used with urlScrapeRedirect - #2537 new jsonl format supported - #2588 don't setup auth if --webconfig isn't used 4.6.0 2023/10/16 - release - curl 8.4.0 - release - fix viewer systemd file - capture - fix zstd hanging capture on full buffer - viewer - corrupt http session decoding might hang viewer - viewer - handle uncompressing pcap errors better - viewer - role check in UI didn't always work - all - handle cookies encoded with bad proxy 4.5.0 2023/09/13 - release - node 16.20.2 - release - added missingok to default logrotate for arkime - capture - dns answers were double parsed - capture - custom-fields honors viewerOnly:true - capture - added dns.https fields - capture - added cert:certificate-authority tag (thanks mcgillowen ) - cont3xt - remove raw view button for link groups on the cont3xt search page - cont3xt - Overview shortcut - cont3xt - fixed overviews not updating on switch - db.pl - don't allow '.' to be used for sync/add path - viewer - fixed ipv6 session display issues when :: in ip - viewer - http display rewritten to not depending on nodejs internals - viewer - gpe display improvements 4.4.0 2023/08/02 - release - cyberchef 10.5.2 - release - update arkime_update_geo.sh to use different manuf location - all - improved json verification - all - better logging when requiredAuthHeader fails - all - better role creation/usage validation - all - don't allow circular role dependencies - all - now need to be an userAdmin and *Admin to update *Admin change settings for another user - all - more auth debugging - all - can now change the password of another *Admin user if you have userAdmin and all the same *Admin - all - hide webEnable, headerAuthEnable checkboxes for roles - all - oidc now uses sameSite: Lax instead of sameSite: Strict for cookies - capture - handle tcp port reuse better - capture - fix kafka memory leak when produce fails - cont3xt - New overview cards - cont3xt - fix startup race condition with db init - cont3xt - new search protocol to prepare for bulk - parliament - fix parliament clean start not letting auth be set up - viewer - gtp decoding - viewer - demo mode improvements, arkimeAdmin can use normally - viewer - fix unique endpoint not enforcing user time limit 4.3.2 2023/06/13 - release - cyberchef 10.4.0 libpcap 1.10.4 - all - config 'prefix' can be at most 50 characters - all - new cookie generation code - capture - handle packets better at epoch time - cont3xt - add twilio country code tidbit - cont3xt - add httpRealm to sample config - cont3xt - help improvements - cont3xt - minor UI improvements - db.pl - set ISM deleteTime for sessions correctly - esproxy - add tests - parliament - fixed occasional missing token error - viewer/wise - Field/Value actions now support all:true to show on every instance - viewer - Fix Src/Dst mouse over for packets/bytes - viewer - Field Actions didn't work in expanded meta - viewer - Fix sending/receiving sessions not working 4.3.1 2023/05/08 - BREAKING - If running mixed versions of Arkime, broken cron queries error might show on OLD version - release - fix ubuntu22 kafka dep - all - passwordSecret log message now has the right [section] - capture - --tags option now works as well as --tag - viewer - new auto cronQueries setting - viewer - change where primary viewer info is stored to not cause constant mapping change - viewer - fixed ipv6 not working, now assumes zero filled with mask (if not provided) - viewer - code refactor into javascript classes 4.3.0 2023/04/27 - BREAKING - Only SuperAdmin can assign *Admin roles now - release - fix kafka library linking - release - al2023 support - release - improve arkime_config_interfaces.sh - release - Configure doesn't offer demo Elasticsearch on Arch - release - reqBodyOnlyUtf8=true in sample config file - all - support colon in OpenSearch/Elasticsearch password - all - fix some prototype pollution - all - improve roles enforcement - all - New authTrustProxy setting - capture - tcpClosingTimeout setting controls delay before saving tcp sessions after close - capture - default dbBulkSize to 1M, min 500K, max 15M and removed from sample config file - capture - s3 writer now writes multiple files based on packetThreads - capture - s3 writer supports zstd, s3Compression setting - capture - s3 writer compression level, s3CompressionBlockSize setting - capture - s3 writer block size, s3CompressionBlockSize setting - capture - s3 writer gap encoding, s3GapPacketPos setting - capture - s3 writer when s3UseECSEnv is true use container env vars to find the id/key/token for s3 auth - capture - improve Gh0st parser (#2225) - capture - new dnp3 & finger classifier - capture - tcphealthcheck adding debugging - capture/viewer - includes setting ignores missing files starting with - - cont3xt - add malicious tidbit from urlscan results - cont3xt - add malicious and brand columns to results table for urlscan - cont3xt - link group UI improvements - cont3xt - add createDate for whois data - db.pl - new --ifneeded option to init/upgrade that will exit if not needed - parliament - fix digest auth - parliament - better auth support - parliament - improve issue page and filters - viewer - display errors when cronQueries isn't configured - viewer - fix first sessions table row obscured sometimes - viewer - disable more apis in demo mode - viewer - allow roles forced expression without user forced expression (#2213) - viewer - s3 now use each file's bucket to determine access style - wise - only send csp headers in initial request for wise page 4.2.0 2023/03/01 - release - node 16.19.1, support node v18 - release - fix arch build issues - release - EL9 build uses sha256 digest - all - OpenSearch/Elasticsearch name cleanup - all - cleanup nodejs dependencies - all - refactor how authentication is done, everything now uses passportjs - all - support oidc authentication method - all - caTrustFile setting should work everywhere - capture - support ERSPAN Type I and vlan for Type II - capture - new kafka plugin for sessions - capture - use malloc instead of GSlice - capture - corrupt DNS alt name memory leak fixed - capture - Added simpleFreeOutputBuffers setting - cont3xt - raw create link groups - cont3xt - two clicks to delete link groups or links - cont3xt - classify domains with multiple dashes correctly - cont3xt - added ability to copy links between link groups - cont3xt - support intl phonenumbers - db.pl - Initial OpenSearch ISM support - db.pl - Better error text for cert verify failure - esproxy - fix converting basic auth to base64 - viewer - fix field actions crash - viewer - can now use expression http.request.FIELD or http.response.FIELD with headers-http-request, headers-http-response defined fields - viewer - support viewing ipv6 DLT_RAW (#1293) - viewer - ESAdmin -> Unflood works on users cluster now also - viewer - support running in s2s auth mode only 4.1.0 2023/01/10 - release - glib 2.72.4 cyberchef 9.55.0 flot 4.2.3 d3 7.7 - db.pl - backup/restore wasn't dealing with templates correctly - db.pl - upgrade failed if there was no moloch_shared user - db.pl - repair now fixes missing history/ecs templates - db.pl - fix users-export/users-import - cont3xt - support missing auth and userTmpl settings - cont3xt - Hide link group when no links match filter - cont3xt - Added landing page - capture - allow wise field dst.ip:port - capture - add VNI field - capture - initial tzsp reader support - capture - y2038 fixes - capture - Integer ops in rules now support a leading min or max which only sets the value if less than or greater than current value - wise - added usersElasticsearchBasicAuth setting and lmdb cache support - wise - add passivetotal value action if at least key is defined - viewer - fix es node stats for different node.roles - viewer/cont3xt - can now search roles - viewer/cont3xt - don't show change password menu item if web auth is enabled for user and disableUserPasswordUI is true 4.0.3 2022/11/28 - release - cyberchef 9.54.0 - release - copy systemd files instead of soft linking - releaes - capture/viewer systemd files now After OpenSearch/Elasticsearch - capture - on short runs, field definitions weren't getting updated - capture - s3 writer sets s3Compress to false with s3WriteGzip true - capture - JA3s value was sometimes incorrect - cont3xt - fixed digest mode fetching settings from config file - db.pl - fixed init not working with OpenSearch sometimes - db.pl - will now count data or data_hot node roles - viewer - fixed showing more than 10 roles 4.0.2 2022/11/01 - release - cyberchef 9.48.0 - all - better console output sanitization - capture/viewer - Add TLS Certificate Organisational Unit field parsing (PR #2038) - capture - use arkime_update_geo.sh in error msg - capture - log error and exit if fields loading fails - release - Stop Configure from destroying systemd files 4.0.1 2022/10/18 - addUser.js - remove WARNING adding first user - addUser.js - --webauthonly now sets header auth flag - all - better console output sanitization - capture - offline pcap allows more outstanding packets based on maxPacketsInQueue - db.pl - Fixed some OpenSearch compatibility - db.pl - Fixed upgrading to 4.x with no _moloch_shared user - viewer - Fix cert notbefore/notafter showing bad dates in sessions table 4.0.0 2022/10/11 - BREAKING - Must be 3.3.0+ to upgrade to 4.x - BREAKING - systemd files auto installed, still need to enable - BREAKING - Move to roles for some permission checking, userAdmin role required to edit users - BREAKING - the version file lives in common directory now - BREAKING - new defaults maxFileSizeG=12, compressES=true - BREAKING - pcap compression is turned on by default, disable with simpleCompression=none - BREAKING - simpleGzipBlockSize renamed simpleCompressionBlockSize - BREAKING - right-click changed to value-actions in config - BREAKING - the userId search in history for admin nolonger adds the surrounding wildcards automatically - BREAKING - views & notifiers are now their own indices - release - cyberchef 9.46.5, node 16.16.0 - release - systemd files are delivered with /opt/arkime path instead of setting at install time - release - CICD tests with OpenSearch - all - Support ES 8 & OpenSearch - all - check for missing users index or no users on startup - all - update code/docs to mention OpenSearch - addUser.js - new --roles option, --admin creates superAdmin user - capture - New ecsEventDataset setting - capture - save sessions not saving packets for across restarts - capture - afpacket rewrite, improve performance & less out of order packets - capture - fix quic crash - capture - make creating fields from config/parsers/wise/tagger use ES bulk call - capture/viewer - new outer fields replace gre fields (PR #1889) - capture/viewer - initial SLL2 support (issue #2002) - capture/viewer - zstd pcap compression - chad - new plugin - cont3xt - new Cont3xt application, see https://arkime.com/cont3xt - cont3xt/viewer - share new user UI - db.pl - fix sync-files/add-missing trying to add non pcap files - db.pl - init/wipe clean up aliases that became indices - db.pl - determine data node using roles array (issue #2006) - db.pl - fix warning: Smartmatch is experimental - db.pl - ilm didn't work if no sessions2 indices - common - fix userAuthIps setting - esproxy - check gzip traffic - esproxy - improved bulk and url sanitization - parliament - added --insecure option - parliament/wise - can be configured to use shared user DB - suricata - support char 127 in json better - viewer - fixed auth fallback to digest from header mode - viewer - field-actions to display configurable menu items on field labels - viewer - share shortcuts with specific users or roles ("arkimeUser" role = old shortcut sharing) - viewer - share views with specific users or roles ("arkimeUser" role = old shortcut sharing) - viewer - share notifiers with specific users or roles (all previous notifiers will be shared with the "arkimeUser" role) - viewer - share queries with specific users or roles - viewer - share hunts with roles - viewer - rework some settings UI, try and make UX similiar when adding things - viewer - no more _moloch_shared user - viewer - configure auto-hiding map/graph on large queries using turnOffGraphDays (default = 30 days) - wise - fix some stats sorting - wise - fix UI saving of INI formatted config - wise - can configure field-actions 3.4.2 2022/03/31 - release - node 16.14.2 - viewer - Packets/s, Sessions/s, Dropped/s didn't have correct total/average - viewer - host = $shortcut should work now - capture - support longer node names (thanks mcgillowen) - capture - host.smb.tokens wasn't defined correctly - wise - alienvault no longer uses key - tagger - support --insecure option - tests - support --insecure with tests 3.4.1 2022/03/16 - release - node 16.14.1 - capture - new snmp parser of a few fields - capture - rules can have numeric ranges - db.pl - stop using history type name - esproxy - added queries/_mapping to GET allow list - viewer - Packets/s, Sessions/s, Dropped/s weren't accurate (thanks mcgillowen) 3.4.0 2022/03/09 - release - node 16.14.0, libpcap 1.10.1 - release - Configure script deals with / in password better - BREAKING - in header auth mode userAuthIps allows only localhost by default - wise - fix issues with redis source - wise - threatstream in sqlite3 mode opens in readonly now - wise - support -o section.var=value command line option - wise - improve json parsing to handle non arrays when expecting an array - wise - didn't always encoding number fields correctly - db.pl - added a repair command that will fix some common issues - viewer - reading packets from S3 failed - viewer - increase speed when searching match fields - viewer - fixed lastUsed when in digest auth - viewer/wise - new userAuthIps setting that has which ips auth requests can come from. header mode - default localhost, other - default all ips - viewer - record which node is cron node and warn if not found - viewer - allow floating point numbers for disk watermarks - capture - switch from deflate to gzip posting to ES, lower min gzip size to 860 - capture - ietf quic improvements - esproxy - can now create a [tee] section that will duplicate all ES calls, but ignore results - tests - Add --elasticsearch option which is actually used correctly 3.3.1 2022/01/26 - viewer - fix displaying large packets or xored packets not always working - capture - refactor curl code based on recommendations - capture - only allow 50 packets per ip4 frag - capture - new modbus parser (thanks mcgillowen) - tests - reduce race conditions 3.3.0 2022/01/19 - BREAKING - non standard pcap files now use the .arkime extension - BREAKING - for wise multiES entries, prefix: now defaults to arkime_ - BREAKING - for wise threatstream source you must create md5 index manually - release - node 14.18.3 - viewer - default to hunt reassembled packets - viewer - add descriptions to hunts - viewer - hide graph/map (speeds up large queries) - viewer - history logs es query/indices - viewer - open up to 50 sessions at a time button - viewer - make sure hunt progress bar shows up - viewer - handle corrupt pcap files better - viewer - handle hunt errors better - viewer - scrubbing won't crash on unsupported files - capture - make sure file/seq es requests have higher priority - capture - support pcap flies with 0 timestamp - capture - use .arkime file extension for non standard pcap files - capture - new _dropBySession rule op - capture - fix infite recursion - thanks albntomat0_1 - capture - improve udp/tcp header length checking - capture - improve error messages for field setting issues - capture - cache when getting pcap data from S3 (thanks pjsg) - capture - New ecsEventProvider setting - wise - switch from node-sqlite3 to better-sqlite3 package - all - support creating gzip files, set simpleGzipBlockSize - all - support creating pcap files with short packet headers, set simpleShortHeader 3.2.1 2021/12/14 - esproxy - handle sessions2 without prefix - capture - new parseHTTPHeaderValueMaxLen replaces hard coded 1024 - viewer - some hunt fixes - viewer - switch from ES bool MUST to FILTER - viewer - increase elasticsearchScrollTimeout default - viewer - new AND arrays with ][ syntax vs OR arrays with [] - viewer - fix --insecure which broke in 3.2.0 - viewer - ES Nodes has new uptime stat - viewer - fix 3.x sending to remote cluster - viewer - disable periodic queries on multiviewer - viewer - history can always toggle open and show api - wise - fixed views that used require: not working - db.pl - sync-files, add-missing, and other fixes since 3.x 3.2.0 2021/12/07 - release - node 14.18.2 - release - remove daily.sh, setup a cron directly now - all - refactor some shared code into common directory - capture - fix memory leak with ip4 frags and packet q overflowing - capture - standardize on config error process exiting - capture - ietf quic improvements - tests - add some auth tests to test suite - viewer - jquery upgrade - viewer - help fields display improvements - viewer - support https urls in wise plugin (issue #1777) - viewer - fix history links with && not working - viewer - userAuthCreateTmpl improvements - viewer - fix cron and database bounding queries - viewer - fix settings page not loading on pre 3.x config - viewer - cyberchef didn't always load the packets 3.1.1 2021/10/13 - release - node 14.18.1 - addUser.js - fix not exiting on complete when certs defined - all - deal with usersElasticsearch being an array better - capture - increase max of pcapWriteSize, tpacketv3NumThreads - capture - decrease max of maxESConns, maxESRequests - viewer - fix vlan display (broke in 3.0.0) - viewer - Issue Query on Page Load "No" option changed to issue query if there is a search expression - viewer - fix position of value actions dropdown in spigraph table - wise - fix error msg about redundant parameters - wise - new mergeQuery setting for splunk 3.1.0 2021/10/04 - all - support float field type - all - support Q-in-Q ether type - all - --insecure has consistent messaging now - all - Finally added elasticsearchBasicAuth/usersElasticsearchBasicAuth can be user:pass or base64(user:pass) - capture - s3 writer uses IMDSv2 (thanks fj604) - capture - dscp src/dst (issue #1626) - capture - refactor how udp tunnels are added, just normal parsers now - capture - initial GENEVE, VXLAN-GPE support - capture - initial IETF QUIC support - capture - fix interfaceOps crash (issue #1763) - release - much improved arkime_config_interfaces.sh (thanks arkaØm) - release - node 14.18.0 - viewer - stop upload menu showing up when empty command - viewer - improve first load time by spliting bundle and lazy load items - viewer - combine multiple calls from UI into one call - viewer - added pcap expire debugging - viewer - uploadCommand can now use INSECURE-ORIGINALNAME to access uploaded filename - viewer - shortcut display improvements - viewer - shortcuts can be in arrays - viewer - shortcut wildcard support (issue #1554) - viewer - didn't decrypt pcap for large packets correctly (issue #1756) - viewer - support wiseURL for wise.js viewer plugin (issue #1758) - viewer - display "0" values - viewer - Test alert includes who requested it - viewer - fixed regex and floats not always working in array expressions - viewer - fix always putting cursor at end of input when selecting typeahead result - viewer - add typeahead results in search expression for values in a list - multies - support elasticsearchApiKey - multies - support scrolling and large queries/pagination - wise - handle empty config file on startup - wise - support ./ with value-actions - wise - support usersElasticsearchAPIKey - esproxy - elasticsearchAPIKey and elasticsearchBasicAuth support 3.0.0 2021/08/18 - BREAKING - Elasticsearch - ES 7.10.0 or later required - BREAKING - if not using a ES prefix, the prefix arkime_ will now be used - BREAKING - multies - multiESNodes requires a name: and prefix: attribute per entry - BREAKING - wise - custom sources will need to be modified - BREAKING - wise - redis urls have a new standard format - BREAKING - wise - for json data keyColumn has been renamed keyPath - BREAKING - wise - now lower case lotermfield and upper case uptermfield fields - BREAKING - capture - override-ips will no longer have leading 0s for ASN - release - curl 7.78.0, node 14.17.5, glib 2.68.3, yara 4.0.2, lua 5.3.6, maxmind 1.4.3, nghttp2 1.44.0 - capture - search for GeoIP files first in /var/lib/GeoIP - capture - fixed possible ABR with time parsing - capture - fixed memory leak with dns bad hostname parsing - capture - new classifier for nfs and several rpc protocols - capture - handle larger oracle connect msgs better - capture - parse small http basic auth headers correctly - capture - rule matches can now be logged - capture - new localPcapIndex setting to enable pcap index on capture node instead of ES - capture - write long open tcp sessions every tcpSaveTimeout even if no syn - capture - fixed possible memory corruption with --flush option - capture - check max packet length in more places - capture - parse proxy-authorization header (PR #1651) - db.pl - new urlinfile://filepath where elasticsearch url is the first line of filepath file - db.pl - fix history mapping issue - viewer - add POST version session query APIs to support long expressions old GET versions still work - viewer - reorganize and standardize APIs - viewer - put npm scripts and common npm packages at top level - viewer - allow viewing of large encrypted pcap files (issue #1555) - viewer - can find nodes for pcap based on --host - viewer - in multiviewer mode can now select which clusters to search (PR #1325) - viewer - fix addTag/removeTag with multiviewer (PR #1556) - viewer - can modify some sessions2 template from esadmin tab - viewer - can modify entire shortcuts now - viewer - spigraph visualization improvements - viewer - Can set watermark settings again - viewer - Start moloch -> arkime cookie changes - viewer - dark theme improvements and new themes - viewer - can now toggle on/off capture start times - viewer - improve toolbar for packet display - viewer - standardize on . ipv6 and : ipv4 port separator - viewer - send shallower queries to ES when possible - viewer - fix cron queries when using autoGenerateId - viewer - improve session settings across sessions - viewer - fix multiple requests when changing views and start/stop times - viewer - fix setting user permissions not applying until page reload - viewer - improve column resizing - viewer - add button to open sessions cron query tagged - viewer - rename cron queries to periodic queries, fix bugs, and add data (created time, creator userId, last run time, enabled/disabled time) - viewer - add create periodic query to action menu dropdown - viewer - notifier links to results - viewer - notifier displays more data (creator userId, created time, last updated time) - viewer - cancel hunts & remove hunt name and id from sessions - viewer - shortcuts sync across all clusters if usersElasticsearch and cronQueries is set - viewer - monitor cert and key files and reload them if they change - viewer - display http request method in history page - viewer - add bad status codes for failing to fetch pcap - viewer - fix uncompress packets by requesting all packets - wise - new config UI, enable with --webconfig - wise - more moloch->arkime changes - wise - for json data can now set arrayPath for start of where to parse - wise - new nlasticsearchfile type - wise - threatstream results now cached - wise - support memcached for cache - wise - new urlinfile://filepath where config is the first line of filepath file - wise - help page - wise - valueactions can be stored in flat file, redis or elasticsearch - all - renamed rightclicks to valueactions - all - many typos fixed - all - support Elasticsearch API Keys (elasticsearchAPIKey setting) - esproxy - first version, see https://arkime.com/esproxy 2.7.1 2020/12/01 - release - glib 2.66.2, curl 7.73.0, nghttp2 1.42.0 - wise - fix UI queries hanging - viewer - fix anonymous user settings not saving - viewer - fix lastUsed time not always saving to ES - capture - new packet dedup feature https://arkime.com/settings#packet-deduplication-settings - capture - close pfring on exit (issue #1538) - capture - fix http2 parsing crash - db - Moloch to Arkime text fixes 2.7.0 2020/11/18 - NOTICE - Requires ES 7.4 or newer - NOTICE - Moloch to Arkime rebranding in UI, everything else still Moloch - all - ES 7 updates, fix most depreciated warnings (mappings/templates still remain) - viewer - fix mpls decoding - viewer - new themes and logo selection - capture - fix http CONNECT response parsing - capture - New pcap-over-ip reader support - db - can import gz files directly now - db - fix issues with version importing 2.4.2 2020/11/10 - NOTICE - db.pl upgrade is required - NOTICE - this is the last version to support ES 6 - release - node 12.19.0 - viewer - support utf8 chars in content-disposition - viewer - add capture process restart to timeline graphs - viewer - add "bookmarks" apply a view's expression to the search input without issuing a query - viewer - fix anonymous users settings not being saved - viewer - share hunts between users - viewer - move all common client bundling, scripts, and npm modules to top level - viewer - display business hours on sessions timeline graph - viewer - fix multi mpls header decoding - viewer - fix viewer crashing when pcap file not available - viewer - new getSessionBySearch setting - viewer - decode vxlan packets better - viewer - add help icon - viewer - added startTime and runningTime capture stats - capture - QUIC version 5x detection - capture - smtp decoding handles clients that break utf8 section incorrectly - capture - fix a json parsing fail would cause next json parse to fail even if good - capture - support 0x6558 Ether Bridging - wise - threatstream improvements when using the sqlite db - db - fix rm-node to delete over 10k items, and bad count display - tests - use our oui/rir files 2.4.1 2020/09/28 - NOTICE - db.pl upgrade is required - NOTICE - the elasticsearch and usersElasticsearch variables must start with http:// or https:// - release - node 12.18.4 - viewer - fixed export pcap from actions menu not working - viewer - capture stats/graph now uses regex instead of wildcard - viewer - support -reindex indices - viewer - log more info when can't open a file - viewer - lastpass boxes removed - viewer - can now edit ILM values from ES Admin tab if ./db.pl ilm has been used previously - viewer - handle hunts with bad regex better - viewer - change capture stats default length to 200 - viewer - fix password change with aes256Encryption turned on - viewer - handle hunts when nodes are down better - wise - UI improvements - wise - theatstream mode sqlite3 no longer copies the db, use sqlite3-copy for old behaviour - parliament - show bits instead of bytes - db - new reindex command - capture - http2 header fields were not always indexed correctly - capture - fix g_hash_table_contains warning - capture - rules can use special ip values ipv4 and ipv6 now - moloch_update_geo.sh - fix possible security issue 2.4.0 2020/08/25 - NOTE - RHEL/Centos 6 is no longer supported, Node 12 required - NOTE - New encoding of packetPos, set gapPacketPos=false for old encoding - NOTE - 2.4.x will be the last versions to support ES 6 - release - node 12.18.2, glib 2.64.5, curl 7.72.0 - release - Ubuntu 20 support - viewer - aes256Encryption now defaults to true - viewer - added a clear cache button to ES Admin tab - viewer - quote expressions with [ or ] in them - viewer - add button to only show data nodes on ES Nodes tab - viewer - files tab can now show the packet pos encoding - viewer - ES Indices tab can now show the avg doc size per index - viewer - ES Nodes tab can now show shards and segments per node - capture - http2 decoding for PRI * h2 sessions - capture - set http2 protocol when alpn is h2 - capture - upgrade h2c http2 decoding - capture - no longer use internal libpcap function - capture - simple writer supports maxFileTimeM (PR #1506) - capture - new packetPos encoding saves 10%-20% overall ES space - capture - remove old disk writer methods, use simple or simple-nodirect now - wise - simple UI - wise - support json file format config files 2.3.2 2020/06/29 - NOTE - 2.3.x will be the last version to support RHEL/Centos 6 - release - node 10.21.0 - capture - minor tcp dns parsing performance improvement - capture - refactor some code to be more type safe - capture - deal with bad utf8/puny in dns and altnames - viewer - warn at starting about missing ./vueapp/dist/index.html - viewer - can now use db: in expressions (PR #1463) - viewer - if esAdminUsers isn't set, ES Admin tab now shows up for admins - viewer - ES Nodes can display molochzone attribute now - viewer - fixed some Users tab issues - viewer - fix percentage sorting on ES Recovery tab - viewer - "right click" actions can how show text in menu with fetch actionType - viewer - "Reverse DNS" menu option on ips 2.3.1 2020/05/27 - all - Lots of changes to support node 12 in the future (thanks rnbwdsh) - viewer - fix bug where the next query after an empty query might hang the UI - viewer - use the same eslint as the UI & parliament, lots of lines changed - viewer - can now modify or delete a view from the popup - viewer - fixed so non admins can cancel their searches again - viewer - fixed columns not always loading with views - viewer - when user creates a view it will auto switch - viewer - does basic ip validation in queries - viewer - fixed users tab header being hidden - capture - fixed out of bounds read in smtp parsing - capture - Lowered the default number of ES retries to 2, added new esMaxRetries setting - wise - upgraded sqlite version and changed from hashtable to Map (thanks rnbwdsh) - db.pl - The info command will now display estimate for how many days can be stored 2.3.0 2020/05/06 - release - CyberChef 9.16.2, node 10.20.1, daq 2.0.7 - viewer - set content-type for cyberchef files - viewer - add support for caTrustFile to addUser and multies - viewer - can now select to show any integer field in graphs, set on the settings page - viewer - graph/header can now be pinned to not scroll off page - viewer - most navbars can be collapsed and hidden - viewer - mouse over in graphs now show total values too - viewer - fixed left/right keys not working in search bar after visiting stats page - viewer - support cancel for multies - viewer - cleaned up some of the Help docs - capture - new parsers arp, bgp, igmp, isis, lldp, ospf, pim, - capture - protocol parsing code has been refactored, can now write parsers of ethernet and other ip protocols - capture - new disableParsers, default of arp.so - capture - new unkEthernet, unkIpProtocol protocols - capture - support QUIC version 46 - capture - new esBulkQuery setting to override the /bulk call - capture - added some more lua examples (thanks Antipokemon) - wise - threatstream fixes to be nicer to the sql database - all - switch most ES apis to typeless format 2.2.3 2020/03/09 - viewer - Experimental treemap view in spigraph - viewer - Hunts now retry talking to failed remote nodes - viewer - Completed Hunts have a repeat button - viewer - Fix some Hunt stat and display issues - viewer - Fixed Hunts/Tags working with ILM - viewer - Fixed notifier issues and issue #1365 - viewer - Increase navbar contrast - viewer - Spiview should be faster loading data - viewer - Debug now prints out config vars like capture - release - Fix lua.so not being included with builds - capture - Fix "-r -" not working - parliament - Ignored issues should remain unless deleted - db - Can set sessions refresh interval 2.2.2 2020/02/18 - release - node 10.19.0 - capture - fix SYN retrans handling - capture - New tcphealthcheck plugin (thanks fj604) - capture - support communityId field in rules/wise - capture - fixed drop stats for long running systems - viewer - Fix decode crash (thanks mammo0) - viewer - experimental pie chart in spigraph - viewer - experimental table view in spigraph - viewer - fix viewer crash when hunting fake sessions (issue #1374) - viewer - fix capture stats sort - viewer - new accessLogFormat, accessLogSuppressPaths settings to better control logging (issue #1375) - viewer - do a better job decoding http 100 continue msgs - decryptPcap.js - can now decrypt Moloch encrypted pcap files to stdout - s3 - Fixes the problem where the s3 token expires during a capture (issue #1370) - s3 - more logging on errors - s3 - new s3Host setting (thank jc2k) - all - debug can be set in config file, used if no command line debug args 2.2.1 2020/01/21 - capture - fix --skip not working with ES 7.x - capture - update TLS ciphers - capture - increase offlineDispatchAfter default to 2500 - capture - cert decode publicAlgorithm and curve - db - optimize-admin doesn't wait for other optimizations to finish - lua - save/pre_save callbacks, can now get most fields - viewer - fix viewer notifiers (issue #1361) 2.2.0 2020/01/13 - NOTE - Elasticsearch 6.7.x is no longer supported - NOTE - MaxMind now requires an account, set up your geoipupdate script https://molo.ch/faq#maxmind - release - node 10.18.1, yara 3.11.0, curl 7.68.0 - release - Configure now installs elasticsearch 7.5.1 - viewer - New aes256Encryption option to upgrade encryption, all viewers must be on 2.2.0 or later before upgrading - viewer - shrink operation deletes old index now - viewer - Querying was sometimes limited to 20000 items (or less) - viewer - Sending sessions between clusters didn't allow pcap to be viewed on receiving side - viewer - Intersection export allows editing of fields - viewer - Panning graph left/right allows selection of how much - viewer - Refresh interval on spigraph page increments time (if it's a date range) - capture - geoLite2ASN and geoLite2Country are now semicolon separated lists. The first one that exists will be used or warning printed. To disable warning set to blank. - capture - disable the 100-Continue feature of curl to reduce bulk errors - capture - smtp parse now maps a few encodings to standards glib understands - s3 - support maxFileTimeM - db - support creating ILM policies and assigning them for sessions2 and history - db - new optimize-admin that only optimizes admin indices - parliament - Click on ES health goes to ES Nodes tab 2.1.2 2019/12/16 - capture - no longer check in configure scripts, use autoreconf (thanks martinpaljak) - capture - new http header raw callback for plugins (thanks pjsg) - lua - Improvements and new sample script (thanks pjsg) - viewer - more cyberchef fixes - viewer - increased timeout on indexing - viewer - show menu/protocol column on sessions page even if there are no visible columns (issue #1337) - parliament - make sure good config before writing (issue #1181) 2.1.1 2019/12/09 - release - cyberchef 9.11.7 - wise - fix view duplication when reload tagger file (#1315) - capture - string fields created with wise/custom-fields are now assumed utf8 - capture - term signal handled better - s3 - handle longer tokens and path vs host access - viewer - support -o option to override config file like capture has - viewer - new "ES Admin" tab, enable with esAdminUsers= in config.ini - viewer - Hopefully fix cyberchef integration on all browsers - viewer - better shrink index support for viewing pcap - viewer - anonymous should work better for more features - viewer - work around ES slow _count API in 7.x - viewer - hunts support views better - db - new warmkind option for specifying what units the warm number is in - db - fix versionNumber printing 2.1.0 2019/11/19 - NOTE: All viewers must be 2.1 or later for them to communicate. If not upgrading at once set s2sSignedAuth=false in default section of config.ini - release - node 10.16.3, curl 7.66.0, glib 2.59.0, libpcap 1.9.1 - release - remove ubuntu 14 builds - release - initial centos8 build - db - new shrink command - db - new --gz option to compress backups - capture - support named pipes better (issue #1169) - capture - New email.smtpHello field - capture - fields created with wise/custom-fields are now set across linked sessions by default now - capture - deal with extra long config values in more places - capture - snf improvements - viewer - default max aggregation size is 10000 to work with ES 7.x OOTB, new maxAggSize setting - viewer - added right-click replacer for %DBFIELD% (thanks tlacuache) - viewer - display JA3s and hassh - viewer - support file expression in more places (issue #1172) - viewer - fix files date display (issue #1164) - viewer - csp header support - viewer - Fix some XSS/rXSS by setting correct content type - viewer - Improve permission checking - viewer - Use RE2 for regex - viewer - shortcuts can be locked from changes - wise - switch to ioredis implementation so clusters/sentinel works - s3 - many fixes (thanks pjsg) - s3 - support compressing pcap when s3WriteGzip is set to true (thanks pjsg) - s3 - can fetch data from metadata service (thanks pjsg) 2.0.1 2019/09/09 - release - cyberchef 9.4.0 - capture - label TLS 1.3 sessions correctly (issue #1137) - capture - New simpleMaxQ setting to control max disk Q - capture - http CONNECT method will now classify payload (issue #1153) - capture - Initial dtls support - viewer - cancel current ES query on new query - viewer - fix connections page timezone - viewer - fix not auto quoting all expressions (issue #1146) - viewer - fix security warnings 2.0.0 2019/08/19 - NOTICE: This versions requires ES 6.7.x (6.8.2+/7.3+ recommended) or later - NOTICE: db.pl upgrade is required, see https://molo.ch/faq#how_do_i_upgrade_to_moloch_2 - release - cyberchef 8.30.0, node 10.16.2, yara 3.10.0 - release - include sample headers parsing and turn them on by default - release - easybutton supports osx - all - Fix some elasticsearch deprecation warnings - all - elasticsearch 7 support - db - backup command now saves meta data so restore can do a rollback (thanks codesniffer) - db - improve optimize to deal with connection closed better - parliament - Can configure multiple of each type of notifier - viewer - Can display pcap retention in Capture Stats tab - viewer - Added uploadFileSizeLimit - viewer - Can interact with users in multiES if usersElasticsearch is set - viewer - Can just delete SPI - viewer - Added shortcuts feature - viewer - add bytes as a graphing choice - viewer - support ip == ipv4 and ip == ipv6 expressions - viewer - pivot dropdown option in spiview (issue #1135) - viewer - optional millisecond display - viewer - Support view parameter for unique/multiunique - viewer - Support ES client auth and insecure better (thanks Scott) - viewer - Lots of stats summing, avg, sorting fixes - capture - Initial ipv6 gtp support - capture - no longer send packet lengths to ES by default (enablePacketLen) - capture - add truncated-pcap tag to sessions where all pcap isn't written - capture - fixed ja3s mishandling of 10/11 extension types (thanks Norwegian Healthcare CERT) - capture - fixed ja3 mishandling of 11 extension types (thanks Norwegian Healthcare CERT) - capture - Added startsWith,contains,endsWith rule expression modifier - capture - honor the caTrustFile directive (thanks Matt) - capture - fix data bytes calculations for icmp/udp (thanks Brian) - capture - initial vxlan support - capture - Myricom/AFPacket improvements (thanks Scott) - capture - updates to classifiers: telnet, mpls - suricata - support timezones and slashes in signatures better - suricata - support huge alert lines - wise - support arrays for json elements 1.8.0 2019/04/03 - NOTICE: This will be the last version to support ES 5 - NOTICE: db.pl upgrade is required - all - support tokens for host.*/http.uri/http.useragent field - viewer - should output csv with commas in fields correctly - viewer - new Show Packets view in session detail - viewer - map show xff countries - capture - libfuzzer fixes - capture - fix core on exit if a pcapDir doesn't exist (issue #1030) - viewer - new elasticsearchTimeout var that is used with ES queries - viewer - can now limit query time frame per user - viewer - node stats now paginates correctly - viewer - support hsts (issue #853) - viewer - support simple range queries field == min-max (no spaces) - viewer - Users page refactor 1.7.1 2019/02/14 - NOTICE: db.pl upgrade is required - viewer - upgrade to d3 v5 for connections page - viewer - typeahead history for spigraph/connections - viewer - stats tasks page has a num item selector now - viewer - welcome message for new users - viewer - save the last time a user used moloch - viewer - two --debug will display why proxying traffic - viewer - connections now uses ipv6.port and ipv4:port - viewer - fix date/time picker timezone and input bugs - wise - support json paths - wise - improve alienvault loading - capture - more tcpflags fields can be matched with rules - capture - print more stats at exit with --debug - capture - fix small bpf memory leak - capture - rules can support most .cnt fields - capture - fix OBR if cert has no serial - capture - libfuzzer support and initial fixes - parliament - add no alert cluster type - parliament - remove selected acknowledged issues - parliament - add help page 1.7.0 2019/01/17 - NOTICE: db.pl upgrade is required - release - node 8.15.0, cyberchef - capture - new cert remainingDays field - capture - new tcp initRTT field - viewer - cron query notifications (issue #489) - viewer - can't use es scroll api with "from" (issue #981) - viewer - Export CSV uses the columns shown - viewer - field history for search expression (issue #595) - viewer - fix date/time picker not using user set timezone (issue #977) - viewer - don’t display undefined or empty field values - viewer - fix timezone parsing in session detail date field values - viewer - show error if using an outdated browser (issue #980) - viewer - export results intersection - viewer - add clickable labels to the info column - viewer - reset the width of the session table columns when switching back to the default - parliament - search and page results on the issues page (issues #982 and #983) - parliament - add a length of time threshold for “low packets” issues (issue #968) - capture - Fixed corrupt file sequence numbers being used when when ES is under heavy load - capture - Fix importing more then 256 files at once not working correctly (issue #984) - all - communityId support for tcp/udp (issue #966) - capture - In live capture clean up sessions even if no packets are being received - db - improve expire efficiency - capture - fix elasticsearch classifier - capture - for offline pcap honor umask when --copy is used (issue #976) - viewer - Fix some rXSS, thanks Esben Sparre Andreasen of Semmle Security Research Team 1.6.2 2018/12/07 - NOTICE: db.pl upgrade is required - suricata - fix crash when signature name > 128 characters long - suricata - fix severity parsing (again) - capture - fix possible crash when exporting invalid utf8 - db - support new --shardsPerNode option - viewer - don't issue search when closing the date/time pickers - viewer - download packets src/dst bytes img - viewer - option to show timezone with every timestamp - viewer - added new user permissions (hideStats, hideFiles, hidePcap, and disablePcapDownload) - parliament - add option to provide link to dashboard in alert notifications - viewer - configure connection node/link popup data - release - build snf plugin with screwdriver - capture - fix tls parser infinite loop - viewer - can customize fields in the info column - viewer - new es recovery tab - viewer - stats page shows when data is being loaded from server 1.6.1 2018/11/06 - NOTICE: db.pl upgrade is required - capture - ja3s support (issue #949) - capture - hassh support (issue #950) - capture - simpleKEKId can be a template - all - Certificate org names can be an array now - wise - reverse dns supports servers setting - all - new written/unwritten stats to see how much Moloch has written or not written to disk - all - don't index packet positions or packet lengths in ES 1.6.0 2018/10/29 - NOTICE: db.pl upgrade is required - release - glib 2.56.2, yara 3.8.1, curl 7.61.1, lua 3.3.5, node 8.12.0 - db - expire checks min lastPacket in each session2 like curator, not just based on name - wise - support any field for ES WISE source - viewer - packet search (hunt) - viewer - admins can see forced expression for users in history - viewer - option to add sessions table column configuration to a view - viewer - files and stats tables can be customized - suricata - parse severity - capture - new _dontCheckYara rule ops - parliament - add --debug option - parliament - add --dasboardOnly flag - capture - set vlan field for afpacket - capture - new setting parseHTTPRequestHeaderAll, which will parse ALL request headers not already parsed into 1 ES field (pr #914) - capture - new setting parseHTTPResponseHeaderAll, which will parse ALL response headers not already parsed into 1 ES field (pr #914) - capture - new setting parseSMTPHeaderAll, which will parse ALL email headers not already parsed into 1 ES field (pr #914) - capture - new setting parseDNSRecordAll, which will parse a full DNS record into multiple new ES fields (pr #914) - viewer - show shortcuts on shift - capture - initial gtp tunnel support (issue #926) - wise - new wiseLogEvery to control how often plugin logs (0 disables) - capture - experimental autoGenerateId config to use ES auto generated ids - viewer - Ability to download files based on hash (pr #927) - viewer - Can resize/save columns (issue #909) - multiviewer - Can save user settings and such (pr #935) - viewer - Can share views with all other users - db.pl - New backup/restore commands, saves everything but sessions 1.5.3 2018/09/10 - release - libpcap 1.9.0 - all - new hourly[2348]/hourly12 rotateIndex - all - deal with talking to multiple wise servers better - all - --insecure option - all - use package-lock.json so all builds use the same packages - viewer - fix some spiview display issues (dns ip, email headers) - viewer - fix upload command tags (issue #888) - viewer - aes-256-ctr: fix issues decoding larger files - viewer - set rejectUnauthorized for ES connections (issue #894) - viewer - fix some payloads not displaying for css reasons - viewer - added zoom buttons to connections - viewer - keyboard shortcuts - capture - require gnu99 supporting compiler - capture - if single tcp data packet left at save time, try and classify it (issue #880) - capture - for live captures prevent out of order stats records - capture - aes-256-ctr: iv is 12B, limit maxFileSizeG to 64G, dek is more random - capture - Added corrupt setting to saveUnknownPackets - capture - new -o option to override config file from command line - capture - trim leading/trailing whitespace from config options - capture - new --nostats option to stop stats from being sent to ES - capture - fix http socket leak on errors - capture - new -F option to specify files that contain a list of pcap files to process - capture - new --reprocess option that won't create a new file entry (but will duplicate SPI data) - capture - add IPPORT_DROPPED count to stats log - capture - fix some possible bad memory reads in oracle/radius/http parsers - capture - fix some possible integer overflow issues - capture - fix tcp sequence number issues - parliament - uses Vue now - parliament - support email alerts - parliament - new edit slider to show edit buttons - parliament - many improvements how issues work 1.5.2 2018/07/26 - capture - new custom-fields section - viewer - new custom-views section - capture - fix for inplace pcap ingestion not displaying pcap in viewer, introduced in 1.5.0 - capture - support QUIC version >= 40 - release - Build Ubuntu 18.04 version - wise - fixed sources that didn't register at startup 1.5.1 2018/07/23 - capture - fix ipv6 sctp hang - viewer - added back many tooltips - viewer - fix crash when node doesn't exist - viewer - fixed some fields not showing up - snf - new config snfFlags - viewer - fix sorting by a column that isn't in all indices - capture - memory leak fix - suricata - keep alerts for suricataExpireMinutes setting 1.5.0 2018/07/16 - BREAKING: wiseCache redis name changes - viewer - sessions, spiview, spigraph, connections, help, users, history pages implemented in Vue instead of AngularJS - viewer - split stats capture graphs/stats into 2 tabs - viewer - stats now has sort option - capture - new maxTcpOutOfOrderPackets setting, default 256 - capture - drophash supports v6 and saving - wise - support talking to wise over https, use wiseURL - capture - basic mqtt parsing - capture - rules reload without restarting - viewer/db - new hide-node/unhide-node commands to hide commands in capture stats - viewer - New queryAllIndices - multiviewer - no longer need to have the same rotateIndex everywhere - capture - initial saveUnknownPackets support - capture - new interfaceOps for ops per interface - capture - new filenameOps for ops per filename (issue #857) - s3 - use 1.0 field names - viewer - fix a case when decoding failed - all - ESP now has first packet capture if trackESP is set - capture - magicMode remove molochmagic and added both mode - wise - initial splunk source - capture - new suricata plugin - wise - [right-click] with no colon now loads directly from wise.ini - capture - decode dns punycode into dns.host, dns.puny has original - capture - yaraEveryPacket (default true) when false only does first packets - capture - yaraFastMode (default true) when false turns off fast mode - viewer - switch to javascript png implementation - all - added some GRE erspan support - viewer - support gt/lt queries for ip - capture/wise - can now configure what fields map to what wise queries and send almost any field (issue #840) - all - added used space stat - all - changed free space stat to use 1000 based units - viewer - removed AngularJS, all pages implemented in Vue - capture - hsrp classify 1.1.1 2018/05/31 - all - fix http.statuscode - capture - fix _dropByDst/_dropBySrc crash - capture - tcpflag are always counted (issue #849) - viewer - fix 10k sessions.json failure 1.1.0 2018/04/30 - all - basic sctp capturing, no decoding (issue #828) - all - initial unencrypted 802.11 Data Frame support (issue #834) - db - new segments option to expire and optimize - release - curl, node, cyberchef - viewer - http.uri and host* allows pasting a URL and doing the right thing (pull #831) - capture - New logHTTPConnections setting (pull #749) - capture - new wiseExcludeDomains setting, used before sending to wiseService (issue #340) defaults to ".in-addr.arpa;.ip6.arpa" - wise - full ipv6 support - capture - basic sccp classify - all - initial frame relay support (issue #838) - all - initial 4 over 6 and 6 over 4 support - viewer - support fields=id for sessions.csv (issue #839) - reindex2 - added --pause option - viewer - more stats page fixes 1.0.0 2018/04/05 - db - always update stats/dstats indices for now - parliament - fix es drop error msg - tests - make server everything has started before running tests 1.0.0-rc2 2018/03/29 - viewer - Change default spiDataMaxIndices to 4 everywhere - viewer - work around for ES 6 issue https://github.com/elastic/elasticsearch/issues/27740 - capture - fixed netflow plugin - tests - initial parliament tests 1.0.0-rc1 2018/03/20 - viewer - minor stats page fixes - release - new top level package.json/node_modules to make package smaller 1.0.0-beta3 2018/03/15 - viewer - stats page implemented in Vue instead of Angular - capture - some code clean and better thread safe counters - viewer - convert field names in saved column sets from pre 1.0 1.0.0-beta2 2018/03/08 - capture - decode some dhcp - capture - tag a tls session with cert:self-signed - capture - reload geo, rir, yara and oui files without restarting (issue #692) - capture - remove yara 1.x support - viewer - cron jobs now use the timestamp time and not last packet time when choosing sessions to look at this means delay is shorter, although when upgrading to 1.0 some sessions will be not looked at. 1.0.0-beta1 2018/02/20 - capture - calculate sha256 too (set supportSha256 tru) - wise - support sha256 lookups - capture - fix disable fields - capture - src/dst ip/port can be used to trigger rules now - capture - ip fields in rules can now be CIDR - capture - simple writer now flushes after 10 seconds of no writing there still can be pagesize bytes unwritten (issue #777) 1.0.0-alpha2 2018/01/31 - Read alpha1 below - release - correct geo files - capture - set default geo file path 1.0.0-alpha1 2018/01/26 - NOTICE: Supported ES Versions: 5.6.x, 6.x (for new installs) - NOTICE: hasheader for email/http for old sessions will not be migrated - all - rename all field names - all - no more analyzed ES fields, everything is a keyword field - all - full ipv6 support - all - tags index removed, tags/hasheader stored as first class fields - all - new reindex2 script to move from pre 1.0 to 1.0 - capture - http uri field no longer starts with // (issue #732) - capture - use maxminddb instead of geoip now (issue #771) - all - Country codes are now 2 letters instead of 3 letters - release - node 8.9.4 0.50.1 2018/03/29 - NOTICE: Supported ES Versions: >= 5.5.0, 6.x is NOT supported - release - upgrade curl, yara, glib - viewer - sessions.csv handle multiple fields parameters - parliament - better dismissing - all - new hourly6 rotateIndex, for every 6 hours - parliament - first version of alerts - parliament - rename server.js to parliament.js - wise - trim spaces after splitting config values - capture - better pop3 detection - capture - correctly lowercase user - release - added --install to easybutton (issue #812) - capture - basic memcache classify - viewer - disable TLS 1.0 - viewer - ES scrolling should be much faster 0.50.0 2018/01/24 - NOTICE: Supported ES Versions: >= 5.5.0, 6.x is NOT supported - viewer - Fixed CSV export of fields within objects (issue #790) - capture - Retry http requests on connect failures - capture - better SLL pcap support (issue #791) - capture - icmp code and type were swapped - viewer - shards tab can now exclude/include nodes/ips to move shards around - capture - removed ES check for http.compression since on default in ES 5 - capture - clean up sockets better - db.pl - now default to a max of 24 shards - parliament - initial display of detected issues - parliament - move all calls behind /parliament to make easier to reverse proxy - viewer - users forceExpression handles special characters correctly again 0.20.2 2018/01/04 - NOTICE: Supported ES Versions: 2.4.x, > 5.3.1, 6.x is NOT supported - viewer - upgrade elasticsearch-js - capture - new --host commandline option to specify how viewers should talk to host - capture - added classify: dhcp, dhcpv6, splunk, isakmp, aruba-papi - capture - improved classify: bitcoin, ntp, ssdp - capture - index first N (default 256) bytes of post bodies, controlled by maxReqBody setting - capture - more stats on packet processing/failures - viewer - cleanup sessionErro - viewer - remove dead code - tests - more viewer tests - viewer - upgrade Cyberchef to 6.8.0 - capture - basic mpls stripping (issue #779) - viewer - start of esshards tab in stats - parliament - first version (can install with Configure --parliament) - wise - can now install with Configure --wise - release - node 6.12.3 0.20.1 2017/11/06 - NOTICE: Supported ES Versions: 2.4.x, 5.1.2, 5.2.x, > 5.3.1, 5.4.x, 5.5.x, 5.6.x - viewer - graph hover now respects timezone setting (issue #757) - capture - decode icmp type/code (issue #751) - viewer - upgrade Cyberchef to 6.4.5 - viewer - es indices tab (#761) - viewer - es tasks tab (#763) - capture - ssh parser crash fixed (introduced in 0.20.0) 0.20.0 2017/10/31 - NOTICE: Supported ES Versions: 2.4.x, 5.1.2, 5.2.x, > 5.3.1, 5.4.x, 5.5.x, 5.6.x - NOTICE: db.pl upgrade is required - capture - added --packetcnt from @jmd1212 - capture - handle monitor directory being empty - capture - basic fbzero parser as quic - capture - pjl, dcerpc detection - release - on ubuntu, plugins weren't loading dependencies correctly - viewer - made number of packets displayed setting accurate - release - move node install from release to easybutton (issue #720) - release - install ES 5 - capture - initial SMTP BDAT support - viewer - initial history support - capture - new advanced setting maxMemPercentage that will abort capture if reached - capture - basic rip, nzsql detection - capture - improved quic, mysql detection - capture - plugins can now replace how SPI data is sent to ES - viewer - fixed right client menu not working (issue #740) - viewer - add/remove tags update tag count (PR #756) - viewer - support sessions resize better - release - use screwdriver to build and test - release - upgrade node version - viewer - fix EXISTS! being reformatted (issue #747) - viewer - Don't allow * by itself in expressions anymore, must use EXISTS! - viewer - removed babel, newer browsers required 0.19.2 2017/08/25 - NOTICE: Supported ES Versions: 2.4.x, 5.1.2, 5.2.x, > 5.3.1, 5.4.x, 5.5.x, 5.6.x - NOTICE: Requires wiseService upgrade if using wise - capture - ja3 support (https://github.com/salesforce/ja3/) (issue #706) - viewer - spi view column sets (issue #713) - viewer - remove jade dependency since using pug now - release - node 6.x - viewer - fix world map button (issue #724) - tests - viewer tests are updated and all passing again - viewer - most session detail labels now have menus (issue #723) - s3 - fixed deadlock (issue #716) - wise - only allow 4096 waiting sessions - viewer - add top 10 countries to map (#564) - viewer - type ahead improvements, disable by default for multiviewer - snf - fix build (issue #719) - viewer - abbreviate large units for data bytes (issue #680) - viewer - add href to nav tab links (issue #651) - viewer - save spiview fields (issue #715) - easybutton - Upgrade yara, glib, curl versions - viewer - session columns are resizable (issue #676) - wise - ja3 support - capture - fixed capture crash when wiseService is restarted - release - build a better NOTICE file for binary releases - wise - don't crash on "null" json values - viewer - decode Basic auth values - viewer - ES scrolling should work better 0.19.1 2017/07/13 - NOTICE: Supported ES Versions: 2.4.x, 5.1.2, 5.2.x, > 5.3.1, 5.4.x - viewer - stats bytes human readable format - capture - fix wise plugin crash - viewer - handle corrupt theme - capture - fix quic parser crash - release - libyaml as a dependency 0.19.0 2017/07/11 - NOTICE: Supported ES Versions: 2.4.x, 5.1.2, 5.2.x, > 5.3.1, 5.4.x - viewer - remove old UI - viewer - CyberChef integration (must always add hex decoding) - viewer - spigraph sort and refreshing improvements - viewer - spiview fixed unique malform url - viewer - stats allow comma separated list - capture - rulesFiles and rules support - capture - fixed daq building - viewer - ES scrolling didn't work (issue #697) - easybutton - Upgrade yara, glib, geoip, curl versions - capture - fixed writer crash with --copy (issue #711) - viewer - lots of stats tab fixes (issue #629,#655) - viewer - setting for query on page load (issue #599) - viewer - don't sort unsortable columns (issue #593) - viewer - clipboard fixes (issue #707) 0.18.3 2017/05/25 - NOTICE: Supported ES Versions: 2.4.x, 5.1.2, 5.2.x, > 5.3.1, 5.4.x - viewer - Other decodings (issues #684, #670) - viewer - Text lineup (issue #598) - wise - track average running time - viewer - session graph uses timezone setting - viewer - fixed cron query creation with forwarding failures - viewer - new serverSecret instead of passwordSecret for S2S - viewer - csv export uses visible columns in session table (issue #601) - viewer - stats page improvements (issue #682) - viewer - do a better job on quoting expressions (issue #694) - viewer - remove duplicate TLS section - viewer - back button works better with new UI - viewer - fixed connections query size (issue #685) - viewer - Real SPA (issue #664) - viewer - Align email header (issue #690) - viewer - Fixed email query generation (issue #689) - viewer - spigraph copy/paste (issue #646) - db - handle _optimize vs _forcemerge 0.18.2 2017/04/16 - NOTICE: Supported ES Versions: 2.4.x, 5.1.2, 5.2.x, > 5.3.1, 5.4.x - capture - New snapLen variable, by default 16384 - release - Configure improvements - release - ethtool startup commands moved to moloch_config_interfaces.sh - viewer - custom themes - viewer - fix clickable session field off-focusing too easily - viewer - can cancel spiview loads (issue #637) - viewer - expose session length in pagination select box (issue #677) - viewer - fix spiview not updating (issue #656) - viewer - type ahead fixes/dismiss (issue #591) - viewer - apply search expression to url (issue #649) - viewer - display array values in sessions correctly (issue #625) - viewer - fix broken node display in stats (issue #672) - viewer - permalink fixes (issue #673) - viewer - spigraph field typeahead fixes (issues #647) - viewer - connections field typeahead fixes (issues #675) - viewer - display when packets are rendering - viewer - make sorting icons consistent - viewer - connections tab links are rendered correctly (issue #641) - viewer - make sure connections popup renders inside view (issue #644) - capture - support ES basic auth (issue #679) - release - Configure supports multiple interfaces - release - only use systemd files if systemctl and /etc/systemd exists 0.18.1 2017/03/28 - NOTICE: Supported ES Versions: 2.4.x, 5.1.2, 5.2.x (5.3.0 is NOT supported) - release - node 4.8.1, lua 5.3.4, curl 7.53.1, glib2 2.52.0 - capture - handle certs with after dates before the before date - capture - actually truncate urls, previous fix didn't work (issue #458, #667) - capture - handle certs with pre epoch times - capture - more magicMode basic detections - db - Handle timeouts and nodeNames better with upgrading from db version 30 to 34 - capture - fix race condition when quitting and processing unsaved sessions 0.18.0 2017/03/21 - NOTICE: Upgrading to 0.18 requires an outage for db.pl upgrade (~5 min) - NOTICE: Must upgrade wiseService BEFORE upgrading moloch-capture. New 0.18.0 wiseService can be used with older moloch-capture. - NOTICE: db.pl upgrade REQUIRES *ALL* moloch-capture to be down - NOTICE: Supported ES Versions: 2.4.x, 5.1.2, 5.2.x (5.3.0 is NOT supported) ES 5 users READ: https://molo.ch/faq#how_do_i_upgrade_to_es_5x - NOTICE: 0.18.0 has a new look and feel. Some features have been added, removed, or temporarily are missing. Feedback using slack or github issues welcome. - wise - md5 lookups now associated with contentType - wise - virustotal support - capture - contentType now sent to wise for md5 lookups - db - all indices except sessions-* are reindex for ES5 support - capture - all readers must use batch interface - multi - fixed tags for new ui /detail and added tests - viewer - files tab uses new ui - viewer - users tab uses new ui - viewer - stats tab uses new ui (/stats.old available temporarily) - viewer - sessions tab uses new ui (/sessions.old available temporarily) - viewer - newUI setting removed - viewer - spiview tab uses new ui (/spiview.old available temporarily) - viewer - connections tab uses new ui (/connections.old available temporarily) - capture - pcapWriteMethod=simple-nodirect uses the simple writer without direct writes. Required for some file systems like zfs - viewer - spigraph tab uses new ui (/spigraph.old available temporarily) - capture - tpacketv3 handles multiple interfaces correctly (issue #658) - easybutton - singlehost and config removed, build remains for now make install & make config should work - capture - 0.17.1 2017/01/30 - NOTICE: ES 2.4 or ES 5.1.2 required (ES 5.x isn't recommended for production yet) ES 5 users READ: https://molo.ch/faq#how_do_i_upgrade_to_es_5x - NOTICE: Can only update directly from Moloch 0.11.1 or later - capture - handle missing syn/ack better - capture - better mongo detection - db - dstats_2 and stats_v1 now use date for time for kibana support, also prepares for ES 5 upgrade - capture - some arm64 fixes (issue #584) - viewer - new settings page - capture - tpacketv3 no longer requires a bpf - capture - pfring uses batch packet api - capture - fix libpcap multiple interface crash (issue #610) - release - Configure improvements 0.17.0 2017/01/05 - NOTICE: npm update required - NOTICE: ES 2.4.x required - NOTICE: Last version that can be updated directly from Moloch 0.11.0 or earlier - viewer - New sessions UI (newUI=true) - all - Initial ES 5.0 support (do not use in production!) - all - Refactor many ES calls to remove depreciated syntax - viewer - Handle multiple query parameters better - WISE - basic elasticsearch source - all - Initial pcap encoding support, variables * simpleEncoding: aes-256-ctr or xor-2048 * simpleKEKId - The kek id to use from the keks section * [keks] - A section with multiple kekid=passwords - capture - Fixed yara 1.7 multithread crash (issue #568) - capture - Handle frag gre with frag ip inside - viewer - New Help UI (based on new ui) - viewer - New bounding select box - viewer - Fix clicking map not applying query expression - db - New sync-files which does both add/rm missing faster - viewer - Add column menus - viewer - Fix reordering columns - viewer - Fix unnecessary whitespace when copying values from session table - viewer - Add IP:Port option to field menus - viewer - Permalink adds "openAll=1" parameter - capture - mysql tls parser - capture - llmnr and mdns - capture - version command line option prints library versions too - release - GeoIP 1.6.9, curl 7.52.1, pcap 1.8.1, glib 2.50.2, node 4.6.2 - capture - Use inotify directly since glib2 monitor doesn't expose close write - capture - support yara 3.5 (issue #521) - capture - new --op option to set any field (issue #412) - capture - added dontSaveSPI=1 support to tagger/wise to not save SPI data (issue #505) - capture - basic kafka, dropbox-lan-sync detection - capture - new scrubspi plugin - viewer - Apply user setting timezone - viewer - Add "EXISTS!" to search expression typeahead results - viewer - split session spi data and pcap loading into two apis 0.16.1 2016/11/28 - NOTICE: db.pl upgrade is required - capture - out-of-order and acked-unseen-segment tags - capture - initial pppoe support (issue #536) - capture - new timestamp field when record written - db - new duplicate firstPacket, lastPacket, ipSrc, ipDst, portSrc, portSrc to make Kibana work better with moloch data. - db - fix info command - viewer - don't crash if we can't decode, still might not decode well though (issue #522) - wise - threatstream sql capture importId and handle stripping hostname for lookups - capture - new pcapReadMethod of tpacketv3 on linux - capture - new option magicMode: libmagic, libmagicnotext, molochmagic, basic, none - capture - The tpacketv3, libpcap* pcapReadMethod now batch packets before adding to packet queues to reduce mutex lock contention - wise - Support dos format tagger files - capture - basic tacacs, flash-policy detection 0.16.0 2016/10/14 - NOTICE: Requires node v4.x (development done with v4.6.0). If upgrading, remove the node_modules and run "npm install" in viewer and wiseService directory - viewer - upgrade many packages - wise - upgrade many packages - easybutton - download prebuilt node 4.6.0 - capture - set accept-encoding for all requests (issue #542) - capture - simple oracle parser - viewer - cron jobs should work much better, unfortunately the delay (~11min) before items are processed is longer (issue #546) - capture - http passer now tags passwd= and pass= as http:password - capture - running on ipv6 improvements (issue #545) - capture - 0.15.1 broke antiSynDrop=false, now works again - viewer - supports multiple --debug on command line to up level - release - supports systemd and lua plugin, slightly more user friendly 0.15.1 2016/10/05 - NOTICE: Last version to support node 0.10 - release - build pfring, daq, snf plugins - wise - threatstream zip method broke in 0.15 (issue #534) - wise - support url type - viewer - read cert/key before dropping permissions (issue #504) - all - tcpflag counting (issue #506) 0.15.0 2016/09/14 - NOTICE: Requires ES 2.1 or later (recommend 2.4) - capture - basic flap detection - db.pl - fixed hourly expiration (issue #501) - capture - detect tcp syslog, udp sip, tcp nsclient - capture - handle syn/ack before or with no syn - capture - support classifiers based on port - capture - detect zabbix - viewer - fixed missing stats data (issue #502) - wise - optionally use external cache (issue #495) - wise - threatstream can now use the opticlink db directly! - capture - also record stats every 10 minutes - viewer - stats can now show 5, 60, 10 minute stats - viewer - make sure userNameHeader is set before trying to use, and warn if viewHost is not localhost - viewer - better ipv6 header decoding - capture - fixed multithread compressES=true crash (issue #524) - capture - Myricom snf reader plugin support - easybutton - ES 2.4.0 0.14.2 2016/07/06 - NOTICE: 0.14.x will be the last version to support ES 1.x and ES 2.0 - viewer - limit autocomplete indexes searched (issue #479) - db.pl - support setting shards and replicas for sessions - capture - simple tds5 parser - capture - simple krb5 parser - capture - fix pfring stats (issue #488) - db.pl - new "add-missing" command, used if ES files table gets messed up (doesn't reindex) - capture - throttle when reading offline pcap before overflowing packet queues - capture - aerospike protocol detection - capture - cassandra protocol detection - viewer - spiview now sorts better when session vs packets is used - capture - minPacketsSaveBPFs which skips saving SPI data for matching sessions, useful for known internal scanning hosts - wise - threatstream includes source now (issue #491) - capture - simple quic parser - capture - pcapDirTemplate for directory naming inside pcapDir supports strftime sequences using local time (issue #496) - capture - more dns parsing - capture - basic ntp detection - capture - basic snmp detection - capture - basic syslog detection - capture - fixed thread waiting so not a busy wait (lower CPU) - capture - basic stun detection 0.14.1 2016/06/02 - NOTICE: glib 2.40 required (should be using 2.47 or higher) - capture - initial lua scripting support - wise - initial bro querying support - capture - debug all config requests with --debug - viewer - fixed XSS (issue #471) - capture - simple ldap parser - capture - simple radius parser - easybutton - ES 2.3.3, run ES as daemon, new config - capture - new db field tags-term with tags in text format - capture - fixed FPE crash in stats (issue #475) - capture - calculate jsonSize better (issue #476) - capture - basic thrift detection - wise - new json format for file/url file types - wise - passivetotal tagging support - wise - new source that proxies to another wise - all - support https elasticsearch 0.14.0 2016/05/08 - NOTICE: libnids is no longer required - NOTICE: You'll want to rerun configure if using the same build directory - NOTICE: db.pl upgrade is required - NOTICE: ipv6 support is experimental, and will change with ES 5.0 - capture - replace libnids with internal tcp - capture - basic threads - capture - memory reporting on linux uses /proc/self/statm now - capture - more stats - capture - basic ipv6 - viewer - Columns can be hidden/moved on stats, sessions pages - capture - pfring and daq support are now plugins (capture/plugins) - capture - can monitor multiple interfaces - capture - udp can parse all packets in session now - dns - now parses all requests/responses - dns - parse multi packet tcp requests - capture - gre support - all - memory/free space percentage (issue #164) - addUser - New option --webauthonly - capture - Added readTruncatedPackets config option - capture - truncate urls at 4096 bytes and mid save sessions with large fields (issue #458) - capture - warn if hostname isn't a FQDN (issue #459) - capture - fix memory leak in ES writing - viewer - Total/Average stats are of filtered data - capture - use getdomainname to try and form FQDN - easybutton - node 0.10.45 0.12.3 2016/03/28 - NOTICE: Only ES 1.7.x and ES 2.x is supported by this version. - NOTICE: Requires running npm update - all - added support for ES 2 - viewer - upgraded to express 4 - viewer - ES load is now a single number to support ES 2 - db.pl - fixed _upgrade call, needed to be POST - tests - http.referer tests - capture - smtp now handles no space for mail from, rcpt to (issue #442) - capture - basic jabber/sip protocol detection - capture - http:password set for case insensitive password= now - capture - rdp "Cookie" if present is stored in user field - viewer - support autocomplete on all fields (experimental) - capture - fix for tagger.so crash on XFF fields introduced in 0.12.2 - easybutton - node 0.10.43, libpcap 1.7.3 - easybutton - disable bluetooth in libpcap (issue #445) 0.12.2 2016/01/15 - NOTICE: Only ES 1.[67].x is supported by this version. - capture - basic redis detection - viewer - connections node distance (issue #428) - easybutton - glib 2.47.4 (issue #423) - easybutton - disable usb support for libpcap (issue #426) - viewer - connections highlighting (issue #431) - capture - basic mongo detection - viewer - added Export Unique IP:Ports - capture - switched memory reporting to even more accurate statm on linux 0.12.1 2015/11/10 - capture - fixed startup glib error - configure - fixed version number - easybutton - ES 1.7.3 0.12.0 2015/11/09 - NOTICE: libcurl (>= 7.21.7) is now required, sorry - NOTICE: db.pl upgrade is required - NOTICE: duplicate items in available both in tags and elsewhere have been removed from tags for future data - capture - http module rewritten to use libcurl, woot! - capture/s3 - S3 traffic now sent over https - viewer - upgraded to DataTables 1.10.7 (issue #379) - viewer - add search to files tab (issue #70) - viewer - handle older indexes without bytes/databytes/packets per src/dst (issue #396) - viewer - upgrade D3 to 3.5.5 and switch to min version - viewer - upgrade jquery to 2.1.4 - db - New rm-node command to delete from ES nodes that are no longer active - capture - cert.hash is sha1 hash for SSL certificate (issue #388) - viewer - New snapto time search that uses current query result for future queries (issue #398) - capture - Removed most tags items - viewer - Delta Time display (issue #398) - capture - pcapWriteMethod defaults to thread-direct now - viewer - freeSpaceG defaults to 5% now - viewer - 404 page and status code logged (issue #397) - easybutton - ES 1.6.2 - easybutton - build curl with --without-librtmp (issue #403) - easybutton - mirror sourceforge downloads for now (issue #406) - db - Increased optimize timeout - db - Require 1.4.x or later - db - Removed "index.codec.bloom.load=false" - db - Added warning for ES below 1.6.2 - wise - use native hashtable, required for threatstream - viewer - fixed addTags to work with session that have no tags - capture - fixed multiple smtp, smb and ssh buffer overflow read/writes (reported by jbremer) - viewer - fixed multiple XSS injections (reported by jbremer) - viewer - fixed crash when viewing large mime messages - capture - irc detection improvement - capture - --quiet flag (issue #427) 0.11.5 2015/06/02 - NOTICE: Only ES 1.[45].x is supported by this version. Restart viewer AFTER upgrading ES versions - NOTICE: If using multies all viewers must be upgraded at once - NOTICE: Requires running npm update - capture - http logs connecting time - capture - fix http crash issue - capture - compiles on OSX again - capture - only classify initial udp traffic, performance increase - tests - new spigraph tests - tests - many new multi tests - viewer - switched from ES facets to ES aggregations - viewer - maps support src vs dst now - multi - fixed hasheader/tag support - easybutton - ES 1.5.2, node 0.10.38 - viewer - Fixed incorrect expression for http.cookie.key - viewer - Added --debug to viewer - viewer - Only keep 10 files on expire instead of 100, delete up to 200 at a time - viewer - Switched hacking viewer - viewer - Decoding is now handled with node streams which will allow chaining and plugin decoders eventually - npm update required - capture - Added cookie value parsing, default off controlled by parseCookieValue (issue #371) - viewer - EXISTS! fixes, tests, and docs (issue #367) - viewer - Reuse Uncompress and Files settings between sessions - viewer - Anonymous users still have a singled saved settings - viewer - Switch actions/views menu to jquery-ContextMenu which looks more like a menu - capture - Handle ASCII formated SMB strings - capture - payload8 wasn't always all 8 bytes - viewer - Initial configuration of displayed columns (issue #257) - capture - reenabled ftp/lmtp classify - capture - vnc classify (issue #382) - capture - fixed tcp_flags for netflow (pr #386) - viewer - spiview shows protocols per category (issue #385) - viewer - spiview supports filename (issue #389) 0.11.4 2015/04/08 - NOTICE: db.pl upgrade is required - NOTICE: Only ES 1.4.x is supported by this version. ES 1.[23] may still work Restart viewer AFTER upgrading ES versions - http connection handling rework, hopefully better - moloch-capture will no longer monitor itself even if it uses the same interface to talk to ES - now save packet lengths to ES - writing pcap to disk is now pluggable - S3 is the first pcap writer plugin, currently experimental https://github.com/aol/moloch/wiki/S3 - multies supports file query - maxFileSizeG can now be a float - Configurable right click actions on various data types https://github.com/aol/moloch/wiki/Settings#rightclick - Upgraded jQuery-contextMenu - WISE - url source supports headers (issue #346) - easybutton installs node 0.10.37, libpcap 1.7.2, and ES 1.4.4 now - connections.csv returns session count (issue #356) - Expose the id/rootId in SPI Meta (issue #355) - WISE - right click loading from files - spiview improvements - collapsible sections, sorting, right click to set load on default (issue #360) - db.pl for ES >= 1.4 now issues _upgrade on optimize/expire - WISE - configurable reversedns stripDomains (issue #365) 0.11.3 2015/02/26 - NOTICE: Only 1.[234].x are supported by this version. Restart viewer AFTER upgrading ES versions - NOTICE: Requires running npm update for ES 1.3 and moment support - NOTICE: Requires running db.pl host:port upgrade - For NEW installs can now set a prefix= config variable and db.pl --prefix option that will prefix all ES tables. This makes it easier for Moloch to share ES with other services OR multiple Moloch clusters can use 1 ES cluster with different prefixes. - New usersElasticsearch= and usersPrefix= config that make it possible for multiple Moloch clusters to share a single users table. - viewer: removal of pre 1.2 ES things - Some cron efficiency improvements - Check more often if files need to be expired - More SMB1 parsing - More TLS ciphers - Major viewer test suite restructure and improvements - Handle searching for ip 255.255.255.255 (issue #301) - Fixed RangeError (issue #299) - CronQuery changes to split up multi day queries - Fixed viewer crashes in pristine state (#304) - Added MultiES and fress install test cases - HTTP Authorization parsing (http.authtype, http.user) - Moved HTTP URI parsing from message complete to headers complete - Better Socks4 support - Updated easybutton versions of glib, es, node, geoip - New data feed framework, WISE - https://github.com/aol/moloch/wiki/WISE - http LOG message has total time now - netflow plugin sends flows for both directions - netflow plugin more time fixes - WISE - threatq support - WISE - reversedns support (issue #217) - WISE - CIDR support (issue #312) - WISE - filtering (issue #314) - WISE - AlienVault support - MultiES fixes with tags search - Start of viewer plugins, set with viewerPlugins - WISE - views now downloaded from wiseService - Requires viewerPlugins=wise.js in ini file - if upgrading (cd plugins ; rm emergingthreats.detail.jade opendns.detail.jade threatq.detail.jade threatstream.detail.jade) - New offlineFilenameRegex setting to control witch files are matched with -R (issue #313) - monitor + recursive should monitor new directories (issue #305) - Fixed addUser.js error with when multiple es nodes are listed in config.ini (issue #322) - WISE - Tagger files can have views defined with #view: - New cert.notbefore, cert.notafter, cert.validfor fields (issue #329) - New starttime, stoptime, view fields (issue #307) - New tls.sessionid.dst, tls.sessionid.src, tls.sessionid fields (issue #326) - Use ELS doc_values for some fields to reduce ES memory - Added cert.cnt back - Handle bad ip.protocol strings better (issue #330) - Added dontSaveBPFs config - Switched capture memory reporting to more accurate getrusage - Added capture cpu reporting to stats (requires db.pl upgrade) 0.11.2 2014/10/16 - NOTICE: ES 1.1.x, 1.2.x, 1.3.x are supported by this version. ES 0.90.12 still works but will no longer be supported. Restart viewer AFTER upgrading ES versions - NOTICE: Requires running db.pl upgrade - NOTICE: Requires running npm update for ES 1.3 support - New experimental "Cron Queries" feature * ONE and ONLY one viewer should have "cronQueries=true" * New [moloch-clusters] config section to send sessions from one cluster to another - Doubled the number of sockets from viewer to ES, now 20 - Regex and wildcard support for file expression - Regex is stricter about back slashing (issue #281) - Cache user lookups for 5 seconds - dontSaveTags config can now have a : for each tag which specifies the total packets to save. (issue #278) - Allow multiple -r and -R options - Fixed update vs upgrade message (issue #287) - Fixed expression errors not displayed on connections tab (issue #288) - Added vlan and mac.src, mac.dst, mac indexing/expressions (issue #284) - Can disable/enable fields from being indexed with './db.pl field disable ' - Directory monitoring support (issue #191) * --monitor (-m) to enable * --recursive required to monitor recursively - --delete removes files after processing, requires --copy - --skip (-s) skips files that have already been processed - Tagger now loads items from ES faster - Tagger now supports setting almost any field using match;FIELD=value;FIELD2=value2 It is now possible to have a different tag per match - Tagger now supports matching email and uri paths - Sort session sections - New http.cookie.key expression - Handle larger SSL/TLS certificates - New fields can be defined in tagger input files - New tls.version and tls.cipher fields 0.11.1 2014/08/07 - NOTICE: ES 0.90.12+, 1.1.x, 1.2.0 are supported by this version. ES 1.0 is NOT supported. This is the LAST version to support 0.90.x Restart viewer AFTER upgrading ES versions - NOTICE: When upgrading your runes.sh for 1.x add a -d to the command, ES no longer runs in background by default - Parsers can register for session save events (issue #248) - Fix compressES check with ES 1.x (issue #255) - Show error for ip queries with regex or wildcard (issue #252) - added session.segments and session.length (issue #254) - support elasticsearch=http:// or https:// format (issue #249) - Only libmagic the first 50 bytes - users tab can now sort various tabs - Turn of bloom filter for previous indexes if using db.pl expire - Set threadpool search queue size to unlimited - stats page works again with dynamic scripts disabled - New db.pl rm-missing command (issue #242) - Upgrade qtip2 to 2.2.0 - Mouse over view names shows expression (issue #220) - Display SPI Data even if node is unavailabe (issue #219) - Netflow plugin timestamp fixes (issue #241) - Comma separated list of elasticsearch hosts (issue #176) - New includes directive (issue #144) - Initial bigendian support in viewer (issue #259) - List queries can now have wildcard and regex items. example: http.uri = [term, w*ldcard, /.*regex/] - freeSpaceG now supports a percentage - Show up to 25 items of each SPI data field with a ... to show more (issue #262) - If a http header went across two packets the leading piece would be chopped 0.11.0 2014/05/08 - BREAKING: elasticsearch 0.90.7 or newer required, recommend 0.90.12+, 1.x not supported yet - BREAKING: node 0.10.20 or newer required, 0.11+ not supported yet - BREAKING: Many of the older expression that were kept for backwards compatibility no longer work - BREAKING: All plugins need to be updated and rebuilt - BREAKING: Glib 2.30 or newer is now required, short term workaround is adding "#define G_VALUE_INIT { 0, { { 0 } } }" to moloch.h, but please upgrade - BREAKING: switched to official elasticsearch javascript client, npm update required (issue #222) - Major internal fields refactoring - Fields are now 'easy' to create, only need to change 2 places - db.pl upgrade should be needed less often - Plugins/Parsers can have their own sessionDetail UI - New protocols, dns.status, dns.query.type, dns.query.class fields - Fixed bug with http parser not capturing last query value - http connecting is now mostly async for faster startup (issue #225) - tagger loading is now mostly async for faster startup - titleTemplate config option (issue #229) - output buffers are now mmaped so they are more likely to be returned to OS - free output buffers are now cached, controlled by maxFreeOutputBuffers - More untagging, new fields http.method, http.statuscode, http.bodymagic - More untagging, new fields email.bodymagic - Start of viewer regression testing - Fix reverse http header parsing - simple mysql parser - Fix smtp subject empty encoded sections - Increase ES query timeout to 5 minutes - simple postgresql parser - More same src/dst ip fixes - easybutton installs node 0.10.28 & ES 0.90.13 now 0.10.1 2014/03/30 - Status code not being set when . after mime data - db.pl has simple mv/rm commands now - Fixed all pagination (issue #192) - multies tag fix (issue #205) - New email.hasheader - New packets.(src|dst), bytes.(src|dst), databytes.(src|dst) (issue #206) - New payload8.(src|dst), payload.(hex|utf8), payload.(src|dst).(hex|utf8) (issue #209) - pcapDir can now be a semicolon separated list of directories, currently just round robin is supported - UI: Fix Search/Actions showing up on second line on page load - capture now does memlock and max schedule priority on startup (issue #199) - when yara is disabled don't retain extra data - parse email user names - antiSynDrop config option - remove schedule priority change for now - Changing memlock failure message to WARNING - new pcapWriteMethod advanced setting, supports direct, thread, thread-direct now - Change ES updates to support "script.disable_dynamic: false" - DNS parsing improvements - Deal with windows-1252 subject encoding better (issue #213) - Tagger supports md5s - Increased default pcap size to 8096 - Added viewHost and multiESHost - Both Yara 1.x and 2.x now supported (issue #201) - DNS status support (issue #218) 0.10.0 2013/12/31 - IMPORTANT: all parsers have been broken out into individual shared libraries. It still isn't possible to easily add new db fields yet, coming soon. - parsersDir and pluginsDir can now be a list of directories - jade 1.0 support (issue #194) - webBasePath fix (issue #193) - reverse socks support - memory reduction - fixed plugin and header sections when together not working - fixed memory leak with GErrors - support traffic to/from same ip better - more capture tests 0.9.3 2013/12/17 - db.pl only open/closes indexes for pre version 12 - Custom date was broken for urls with no date param - Non standard date param added to menu - Http file parsing improvements - ES health loaded on page load (issue #172) - Session detail check boxes work multiple times again - core fix with empty tagging plugin information - multiple connections.csv files (issue #163) - fixed view editing - unique.txt tags fixed - plugins can add fields - start of capture regression tests - SNI support (issue #157) - lots of socks decoding improvements - fixed socks memory leak - smtp status code tagging (issue #180) - added missing DNS qtypes - tcp DNS support (issue #173) - DNS MX support - easybutton builds libpcap 1.5.1 - proxy content type correctly - fixed viewer exit (issue #183) - added unique email filenames - src/dst raw view (issue #178) - SMTP subject encoded parsing - SMTP received header parsing (issue #175) - Basic IMAP tagging (issue #186) - Basic RDP tagging (issue #187) - Better bad passwordSecret error message (issue #190) - Upgrade d3 package - smtp file finger printing (#174) - include smtp user-agent header 0.9.2 2013/11/14 - BREAKING: nodejs 0.8 is no longer supported - Upgrade d3 and cubism - Fixed searches so numbers don't have to be quoted - Fixed export hitting max number of stack frames - Connections tab new UI - Connections tab allows any field for src/dst - More user settings - Fixed unique.txt to deal with multi value fields - viewer.js now uses forever-agent package to help multi machine communication. (npm install required) - easybutton installs node 0.10.20 now - fixed race condition with tag lookup rate limiting - expression ip.dst == ip:port wasn't working - more max stack fixes - users tab improvements (issue #152) - New views concept (issue #146), created in settings tab - settings tab improvements - Ability to search for http.uri.path, http.uri.key,http.uri.value for uri path, query string key, and query string value (parseQSValue must be set to true) - --dryrun doesn't use ES for anything now - New session hash algorithm - Token checking function now shared - Fixed broken upload - Change 'npm install' to 'npm update' everywhere - New maxFileTimeM for time rotation (issue #158) - Increased SMB decode buffer size - Fixed SMB decode infinite loop - Fixed expire bug with multi nodes on same machine and different traffic rates - Added connections.csv (issue #163) - Added unlock button to connections - small resolution UI improvements (issue #159) - sessionDetail cleanup - Permalinks are faster (issue #162) - Missing rir data would cause session detail to not open - Reassembled IP frames > ~5k would cause session detail to not open - Fixed right click issues (issue #169) - New payload8.src, payload8.dst that saves the first 8 bytes of sessions in hex - New socks.user field (issue #167) - Tagger supports CIDR and 1 level hostname lookups (issue #160) - DHT tagging (issue #154) - stylus > 0.39 fix - javascript loop length "improvements" - switch from forever-agent to keep-alive-agent, npm update required - caTrustFile config option (issue #161, pull #171 - start of some javascript cleanup - BREAKING: Upgrade to jquery 2.x, no more IE <= 8 support - remove connect-timeout package requirement - increase 2 minute http timeout to 10 minutes - increase max session queried to 2 million 0.9.1 2013/10/03 - Make sure at least one stats record is written per run - Display IRC channel in sessions view - Fix right click on sessions view info column - Fixed post increment issue in js0n code (issue #128) - Fixed broken hourly rotateIndex in viewer (issue #130) - Fixed broken settings page for other user (issue #126) - Basic SMB tagging - Basic ES query throttling - Added missing ssh.ver from spigraph - EXPERIMENTAL: Multi cluster search (issue #97) - Fixed CSV not equal search queries with range fields (issue #132) - BREAKING: To specify install dir with ./easybutton-build.sh use --dir for example: ./easybutton-build.sh --dir /nids/moloch - Can build with PFRING now, easybutton-build.sh has --pfring or easybutton-single.sh asks - Basic smb parsing, disable with parseSMB=false - Basic socks4 and socks5 decoding - rir lookups, configure with rirFile=ipv4-address-space.csv https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv - Netflowish CSV exporting from UI - clean up db.pl some, rename rotate command to expire - With custom date queries can now select bounded by - New user setting for sessions sort order - Fixed encoding issues - New plugin pre save callback - Fixed entirePcap not setting correct Content-Type - New right click pivot option in spiview 0.9.0 2013/08/26 - 32bit fix for lpd/fpd - easybutton now uses nodejs 0.10, 0.8 is still supported for now - Work around for tcp seq number wrapping causing viewer exit - dns parsing core fix - switch to nonblocking pcap saves - more debugging info on proxy failure - Fixed bug when setting viewUrl - Limit number of libnids errors (issue #115) - Display possible reasons for libnids IP Header error - Another domainless hostname fix (issue #116) - Exports should be between 2x-5x faster - Added actions menu for search/sessions - Scrub and Delete actions, user must have remove right enabled (issue #119, issue #89) - Add/Remove(remove right required) tags actions - Hourly rotation (issue #122) - unique.txt fixes (issue #123) - Actions can be done on linked sessions (issue #120) - SPI Graph auto refresh (issue #111) - Better error handling for SPI data display (issue #109) - List queries using [] syntax (issue #106) - user prefs with timezone display (issue #95) - Basic IRC searches - Disk Queue stats display 0.8.7 2013/07/12 - Use recent versions of express which REQUIRES "npm install" in viewer directory - Use recent version of jade which requires extra spaces, use "git diff -w" to - Now index Host headers with and without port - pcapng exporting with meta data - Basic upload feature, doesn't support transfers of meta data yet - addUser.js has better help and error reporting - ES optimizations to use bool instead of and/or, also use regexp filter instead of regexp query - Changed ES stats shown to hopefully more useful ones - Fixed viewer exit on empty data gzip decode 0.8.6 2013/06/20 - Deal with non data ES nodes - Viewer prints error if it can't find pcapDir setting - New setting dbFlushTimeout that controls how often we flush to ES - New setting compressES that turns on compression to ES, requires http.compression: true in elasticsearch yml file - libnids was overreporting traffic, switch to libpcap stats, bytes/sec and total bytes/sec in stats will be lower - Fixed recent jade warnings - Fixed openned export - minor ui improvements 0.8.5 2013/06/14 - NOTICE: Requires at least 0.90.1 ES - New export dialog that asks for filename and number selection - spigraph shows health, decodes tags/ips, has sort by name - spigraph/spiview show total counts - upgrade to jvectormap 1.2.2 which fixes spigraph issues - deal with 113 (SLL) pcap type - header search and header cnt search didn't always work - ignore case of trailing .pcap when processing a directory - fixed bad bug with exporting large files corrupting pcap - HTTP file decoding works better - On exit ignore http queue limits 0.8.4 2013/05/28 - NOTICE: Last version to support 0.20 ES - NOTICE: Changed some expressions, old versions are supported for now email.ct* => email.content-type* email.mv* => email.content-type* email.id* => email.message-id* email.ua* => email.x-mailer* header* => http.hasheader* ua* => http.user-agent* http.ua* => http.user-agent* - valgrind fixes and memory reduction - New SPI Graph tab which lets you graph an expression per field - Now possible to chose which http request, response and smtp headers to index using headers-http-request, headers-http-response, headers-email sections - Session Graph now shows the full queried range instead of data available range - Fixed db.pl wipe error - Added density to db.pl info - Added override-ips config section that allows overriding of country, tag, asn for ips and cidr ips - Clean up add users UI a little, and clear fields on successful add 0.8.3 2013/05/09 - full text for uri is now available - regex searches using == /REGEXHERE/ regex can be slow so be careful - regex and wildcard searches full text instead of tokenized - fixed bug with uri.cnt not be recorded - filenumber generation rewritten, can now deal with multiple instances running and other edge cases - http body content is md5, although the encoded and non encoded version will get different md5s - detect when npm install needs to be run - quoted strings and regex strings detect better - new centerTime=time&timeWindow=minutes option to do +- views - show tags names in unique views - remember view setting for future session views - DNS qclass and qtype tags - Upgrade yara and glib version 0.8.2 2013/04/29 - Install ES 0.90 - fixed dropped packet stats - netflow plugin (issue #27) - memory capture improvements - record capture memory in stats - record filesize for offline pcap - remove port from http host header (issue #63) - db.pl prints more info by default, multiple -v even more information, and new info command - fixed viewer crashes if pcap can't be read (issue #67) - minor css cleanup - Display CERT info in session view 0.8.1 2013/04/19 - Should support nodejs 0.10.3, but still use 0.8.23 for now - Support RAW link type pcap files - renamed decode.js to pcap.js - Setting spiDataMaxIndices to -1 allows all for spiview - Log userId for requests - fixed uri.cnt - don't exit moloch-capture until all file creates finish 0.8.0 2013/04/17 - New SPI View tab, REQUIRES elasticsearch 0.90 RC2 or later - config spiDataMaxIndices controls how many indices to run against since spiview feature can cause elastic search to blowup memory. - display date as year/mon/day - Lots of UI cleanup, slightly less ugly as before hopefully - 32 bit builds should work - Fixed bug where status codes/http methods weren't always recorded - New SMTP plugin callbacks, more to come - offline capture reading should work better with old libpcap versions - DB now stores full and tokenized version of user agents, ASNs, and cert info - verify the config file has a defaults section - display elastic search health for admin users on pages - display elastic search stats on stats page - display ip protocol friendly name - display simple png view of raw session data and attachments on mouseover, requires "npm install" in viewer directory - new much more accurate world map [thanks Dave] - fixed user name XSS issue [thanks z0mbiehunt3r] - fixed many viewer exits - timestamp display option in sessionDetail - graph now uses seconds if less than 30 minutes and hours if more then 5 days. This makes display faster - Refactored how capture stores spi data in memory - Refactored hash table code - Added host.dns, host.http, host.email