5.0.10 | 2023-09-12 13:22:16 -0700 * Load strings utils from extend-email btest (Tim Wojtulewicz, Corelight) 5.0.9-19 | 2023-09-12 11:26:54 -0700 * Add extract_limit_includes_missing option for file extraction (Johanna Amann, Corelight) Setting this option to false does not count missing bytes in files towards the extraction limits, and allows to extract data up to the desired limit, even when partial files are written. When missing bytes are encountered, files are now written as sparse files. Using this option requires the underlying storage and utilities to support sparse files. * File extraction: use fseek (Johanna Amann, Corelight) In the past, we allocated a buffer with zeroes and wrote that with fwrite. Now, instead we just fseek to the correct offset. This changes the way in which the file extract limit is counted a bit; skipped bytes do no longer count against the file size limit. * ftp: Do not log non-pending commands (Arne Welzel, Corelight) OSS Fuzz generated a CWD request and reply followed by very many EPRT requests. This caused Zeek to re-log the CWD request and invoke `build_url_ftp()` over and over again resulting in long processing times. Avoid this scenario by not logging commands that aren't pending anymore. * http/smtp: Fix wrong character class usage (Arne Welzel, Corelight) A call to `extract_filename_from_content_disposition()` is only efficient if the string is guaranteed to contain the pattern that is removed by `sub()`. Due to missing brackets around the `[:blank:]` character class, an overly long string (756kb) ending in "Type:dtanameaa=" matched the wrong pattern causing `sub()` to exhibit quadratic runtime. Besides that, we may have potentially extracted wrong information from a crafted header value. * Make sure that all sessions/connections are done before deleting plugins (Tim Wojtulewicz, Corelight) 5.0.9-5 | 2023-08-08 14:58:31 -0700 * socks/dpd: Add newer auth methods (Arne Welzel, Corelight) The IANA has allocated a few more authentication methods, minimally extend the signature to catch those. (cherry-picked from commit c389d9804bf06154ba7683d46c5ba93b6590c2cc) * socks/dpd: Fix socks5_server side signature (Arne Welzel, Corelight) The server replies with \x05 and identifier for the chosen method. Not quite sure what the previous signature meant capture. See also: https://datatracker.ietf.org/doc/html/rfc1928#section-3 (cherry-picked from commit c389d9804bf06154ba7683d46c5ba93b6590c2cc) * Add trigger_mgr to iosource_mgr later during startup (Tim Wojtulewicz, Corelight) This fixes a potential crash due to trigger_mgr getting shutdown earlier than dns_mgr, and dns_mgr then trying to use it after it's been deleted. This change forces the order of initialization/destruction in iosource_mgr to cause dns_mgr to be deleted first. (cherry-picked from commit e9ff3899121fabc4d7081d0a6c451f3a59840044) 5.0.9 | 2023-05-19 08:28:40 -0700 * files: Warn once for missing get_file_handle() (Arne Welzel, Corelight) Repeating the message for every new call to get_file_handle() is not very useful. It's pretty much an analyzer configuration issue so logging it once should be enough. * MIME: Re-use cur_entity_id for EndOfFile() (Arne Welzel, Corelight) If DataIn() was called and a cur_entity_id (file_id) has been produced previously, re-use it for calls to EndOfFile(). This avoids a costly event_mgr.Drain() when we already have that information. It should be safer, too, as `get_file_handle()` in script may generate a different ID and thereby de-synchronizing. * zeek-fuzzer-setup: Configure fake DNS (Arne Welzel, Corelight) I'm not sure if we somehow set this for oss-fuzz through the environment, but didn't find anything obvious. Running oss-fuzz reproducers locally can triggers lookups to malware.hash.cymru.com and potentially other domains due to loading local.zeek. * SupportAnalyzer: Stop delivering to disabled parent analyzer (Arne Welzel, Corelight) When the parent of a support analyzer has been disabled, short-circuit delivering stream or packet data to it. The specific scenario this avoids is the Content-Line analyzer continuing to feed data lines into an disabled SMTP analyzer in turn creating more events. This is primarily useful for our fuzzing setup where data chunks up to 1MB are generated and fed into the analyzer pipeline. In the real-world, chunk sizes are usually bounded to packet size. Certain TCP reassembly constellations may trigger these scenarios, however. Closes #168 * Add length checking to ToRawPktHdrVal for truncated packets (Tim Wojtulewicz, Corelight) * ftp: No unbounded directory command re-use (Arne Welzel, Corelight) OSS-Fuzz generated traffic containing a CWD command with a single very large path argument (427kb) starting with ".___/` \x00\x00...", This is followed by a large number of ftp replies with code 250. The directory logic in ftp_reply() would match every incoming reply with the one pending CWD command, triggering path buildup ending with something 120MB in size. Protect from re-using a directory command by setting a flag in the CmdArg record when it was consumed for the path traversal logic. This doesn't prevent unbounded path build-up generally, but does prevent the amplification of a single large command with very many small ftp_replies. Re-using a pending path command seems like a bug as well. 5.0.8-10 | 2023-05-17 16:57:57 -0700 * Update CHANGES, VERSION, and NEWS (Tim Wojtulewicz, Corelight) * Add cstdint include to WeirdState.h for GCC 13 support (Tim Wojtulewicz, Corelight) * Update CAF for GCC 13 fixes (Tim Wojtulewicz, Corelight) * Drop Spicy stderr from tests. (Benjamin Bannier, Corelight) The stream usually contains non-fatal compiler diagnostics which are not relevant for the behavior under test. * Bump Spicy to latest release. (Benjamin Bannier, Corelight) 5.0.8-7 | 2023-05-04 10:13:20 -0700 * Add cstdint include to WeirdState.h for GCC 13 support (Tim Wojtulewicz, Corelight) * Update CAF for GCC 13 fixes (Tim Wojtulewicz, Corelight) * Bump the OpenSUSE Tumbleweed CI image to get gcc 13 (Tim Wojtulewicz, Corelight) 5.0.8-3 | 2023-05-02 17:02:57 -0700 * Drop Spicy stderr from tests. (Benjamin Bannier, Corelight) The stream usually contains non-fatal compiler diagnostics which are not relevant for the behavior under test. * Bump Spicy to latest release. (Benjamin Bannier, Corelight) 5.0.8 | 2023-04-11 11:50:33 -0700 * Update docs submodule [nomail] [skip ci] (Tim Wojtulewicz) * Update CHANGES, VERSION, and NEWS for v5.0.8 (Tim Wojtulewicz, Corelight) * smtp: Validate mail transaction and disable SMTP analyzer if excessive (Arne Welzel, Corelight) An invalid mail transaction is determined as * RCPT TO command without a preceding MAIL FROM * a DATA command without a preceding RCPT TO and logged as a weird. The testing pcap for invalid mail transactions was produced with a Python script against a local exim4 configured to accept more errors and unknown commands than 3 by default: # exim4.conf.template smtp_max_synprot_errors = 100 smtp_max_unknown_commands = 100 See also: https://www.rfc-editor.org/rfc/rfc5321#section-3.3 * generic-analyzer-fuzzer: Detect disable_analyzer() from scripts (Arne Welzel, Corelight) Test if the analyzer is removed from the TCPSessionAdapter during event processing. If we don't do this, we continue feeding the analyzer even if scripts decided to disable the analyzer. The analyzer instance isn't flagged as disabled itself, so we need to look at the parent's children. * RDP: Instantiate SSL analyzer instead of PIA (Tim Wojtulewicz, Corelight) * RDP: add some enforcement to required values based on MS-RDPBCGR docs (Tim Wojtulewicz, Corelight) * file_analysis/File: Report overflowing chunks as weird and discard/truncate (Arne Welzel, Corelight) This is one level above the Reassembler where we still have information about the file and source. A weird entry may looks as follows: 1679759398.237353 ... file_offset_overflow FXPLGt4SeMmlMKahJc: offset=fffffffffffffff7 len=10 F zeek HTTP * Reassem: Reject blocks overflowing 64bit upper (Arne Welzel, Corelight) The reassembler logic isn't wrap around safe, so just truncate or reject such blocks. For files specifically, a byte offset in the 2**64 bytes represents 16EiB which is the maximum size supported by BTRFS or NTFS (and probably nothing we'd ever see in practice). * zeek-setup: Load scrips before running unit tests (Arne Welzel, Corelight) It is currently not possible to call a->Conn()->GetVal() or construct a zeek/file_analysis/File object from within doctests, as these quickly reference the unpopulated zeek::id namespace to construct Val objects of various types, making it hard write basic tests without completely re-organizing. Move running of the unit tests after parsing the scripts, so it is possible for some basic exercising of File objects within tests. * ftp/main: Special case for intermediate reply lines (Arne Welzel, Corelight) The medium.trace in the private external test suite contains one session/server that violates the multi-line reply protocol and happened to work out fairly well regardless due to how we looked up the pending commands unconditionally before. Continue to match up reply lines that "look like they contain status codes" even if cont_resp = T. This still improves runtime for the OSS-Fuzz generated test case and keeps the external baselines valid. The affected session can be extracted as follows: zcat Traces/medium.trace.gz | tcpdump -r - 'port 1491 and port 21' We could push this into the analyzer, too, minimally the RFC says: > If an intermediary line begins with a 3-digit number, the Server > must pad the front to avoid confusion. * ftp/main: Skip get_pending_command() for intermediate reply lines (Arne Welzel, Corelight) Intermediate lines of multiline replies usually do not contain valid status codes (even if servers may opt to include them). Their content may be anything and likely unrelated to the original command. There's little reason for us trying to match them with a corresponding command. OSS-Fuzz generated a large command reply with very many intermediate lines which caused long processing times due to matching every line with all currently pending commands. This is a DoS vector against Zeek. The new ipv6-multiline-reply.trace and ipv6-retr-samba.trace files have been extracted from the external ipv6.trace. 5.0.7-23 | 2023-04-08 11:30:01 -0700 * Fix a long-standing bug in the math around continue_processing (Tim Wojtulewicz, Corelight) * Add is_processing_suspended BIF (Tim Wojtulewicz, Corelight) 5.0.7-19 | 2023-04-03 12:19:12 -0700 * Allow offline packet sources to register FDs. (Jan Grashoefer, Corelight) (cherry picked from commit c5a9eb920c6596adbec2f3c54facb630b7e9fe34) * GH-2039: GH-2842: PktSrc: Avoid calling ExtractNextPacketInternal() in GetNextTimeout() (Arne Welzel, Corelight) This reworks 2aec7640dd60c3ea3cffd82461588567e406db34 (zeek/zeek#2039) to avoid calling ExtractNextPacketInternal() within GetNextTimeout() for the non-pseudo-realtime case. Also relates to zeek/zeek#2842. The intention of the referenced change was to avoid a 0.00002 timeout when a non-selectable packet source has more packets queued. This was implemented by checking for a new packet within GetNextTimeout(). The proposed change switches to an predictive approach: Use the result of the previous ExtractNextPacket() call (stored as had_packet) as an indication whether more packets are to be expected. Calling ExtractNextPacketInternal() within GetNextTimeout() may cause surprising behavior as some packet source may block [1] or spent a significant amount of time (e.g. applying BPF filters [2]) within ExtractNextPacket(). The result of GetNextTimeout() should be available immediately as guidance for the main-loop and the actual work should happen within the ->Process() method. This change also attempts to separate the pseudo-realtime logic from the non-pseudo-realtime in an attempt show pseudo-realtime as special. [1] https://github.com/hosom/bro-napatech/blob/00c4d657e034927301d5f8d9bc03eca81e619699/src/Napatech.cc#L116 [2] https://github.com/sethhall/bro-myricom/blob/58b25c8ebac6d184ff8ff0c27f8da2b603694dab/src/Myricom.cc#L250 (cherry picked from commit c5a9eb920c6596adbec2f3c54facb630b7e9fe34) * Consider cap len when forwarding into packet analysis. (Jan Grashoefer, Corelight) When forwarding into packet analysis from TCP or UDP, the protocol's length fields were trusted. This might be dangerous in case of truncated packets. (cherry picked from commit 69b6443ddbb45cf86fbf3df9d1418b73b0c40fc9) * Fix forwarding of tunnelled packets. (Jan Grashoefer, Corelight) This fixes a bug for AYIYA, Geneve and VXLAN forwarding encapsulated content only if it's longer than their header. A new weird is introduced to indicate empty tunnels. (cherry picked from commit 5718046b967e22ff375c55402e9a877163565719) * Call AnalyzerConfirmation earlier in VXLAN/Geneve analysis (Tim Wojtulewicz, Corelight) (cherry picked from commit cea7fc49076ad5250243d449a5ddc26ccf0d06cb) * Add validation of session to start of AYIYA/VXLAN/Geneve analysis (Tim Wojtulewicz, Corelight) This mimics how the Teredo analyzer is already doing it, including sending a weird if the session is invalid and bailing out if the protocol was already violated. (cherry picked from commit cea7fc49076ad5250243d449a5ddc26ccf0d06cb) * GH-2684: Stop violating VXLAN for forwarding failures (Tim Wojtulewicz, Corelight) (cherry picked from commit 8d815feb34d2116965c3f5ed7a2cd15c61dcc394) * GH-2851: HTTP: Make Content-Range parsing more robust (Arne Welzel, Corelight) This was exposed by OSS-Fuzz after the HTTP/0.9 changes in zeek/zeek#2851: We do not check the result of parsing the from and last bytes of a Content-Range header and would reference uninitialized values on the stack if these were not valid. This doesn't seem as bad as it sounds outside of yielding non-sensible values: If the result was negative, we weird/bailed. If the result was positive, we already had to treat it with suspicion anyway and the SetPlainDelivery() logic accounts for that. (cherry picked from commit c29b98b224a00fe4c37177ac334567d6cdd3da28) * GH-2851: HTTP: Reset reply_message for HTTP/0.9 (Arne Welzel, Corelight) OSS-Fuzz tickled an assert when sending a HTTP response before a HTTP/0.9 request. Avoid this by resetting reply_message upon seeing a HTTP/0.9 request. PCAP was generated artificially: Server sending a reply providing a Content-Length. Because HTTP/0.9 processing would remove the ContentLine support analyzer, more data was delivered to the HTTP_Message than expected, triggering an assert. (cherry picked from commit e62e57a7cfac0ffe8178b222a76c652233a3778e) * Add weird for unknown HTTP/0.9 request method (Tim Wojtulewicz, Corelight) (cherry picked from commit b14cc413d886692df6d0f2a27ce4af657d9783a3) * Special case HTTP 0.9 early on (Tim Wojtulewicz, Corelight) Mostly, treat HTTP0.9 completely separate. Because we're doing raw delivery of a body directly, fake enough (connection_close=1, and finish headers manually) so that the MIME infrastructure thinks it is seeing a body. This deals better with the body due to accounting for the first line. Also it avoids the content line analyzer to strip CRLF/LF and the analyzer then adding CRLF unconditionally by fully bypassing the content line analyzer. Concretely, the vlan-mpls test case contains a HTTP response with LF only, but the previous implementation would use CRLF, accounting for two many bytes. Same for the http.no-version test which would previously report a body length of 280 and now is at 323 (which agrees with wireshark). Further, the mime_type detection for the http-09 test case works because it's now seeing the full body. Drawback: We don't extract headers when a server actually replies with a HTTP/1.1 message, but grrr, something needs to give I guess. (cherry picked from commit b14cc413d886692df6d0f2a27ce4af657d9783a3) * Remove a couple unnecessary break statements (Tim Wojtulewicz, Corelight) (cherry picked from commit b14cc413d886692df6d0f2a27ce4af657d9783a3) * Add test for new handling of unknown RDP keyboards (Tim Wojtulewicz, Corelight) (cherry picked from commit a82f743bcceb0187a54791173483dbcd3489142b) * Log raw keyboard value on best guess (jeff-bb) (cherry picked from commit 6cfb45d24f3a7b2eb178dd7b0f058d47d05bb3e2) * Avoid excessive fmt calls, return default behavior on unknown (jeff-bb) Using "in" to query the language const. This also handles the case of not having a best guess and continue using the existing behavior. Given keyboard_layout = 1033 (0x0409), "keyboard-English - United States" keyboard_layout = 66569 (0x00010409), "keyboard-English - United States (Best Guess)" keyboard_layout = 12345 (0x3039), "keyboard-12345" (cherry picked from commit 6cfb45d24f3a7b2eb178dd7b0f058d47d05bb3e2) * "Best Guess" unknown keyboard / language variants (jeff-bb) If the lookup table does not have an entry, it will just log as the raw decimal language/keyboard code. With this change, if we do not have an entry in the lookup table, we'll look at the low order / 4 least significant bits to see if we have a match. The high order / 4 most significant bits are flags/modifiers to the base language/keyboard code. We'll append that it is a "Best Guess" (cherry picked from commit 6cfb45d24f3a7b2eb178dd7b0f058d47d05bb3e2) * Update RDP Keyboard Languages (jeff-bb) Using additional sources to add more languages / locales. Changed sort order to match other sections. (cherry picked from commit 6e59793da6bf7a8c5d83eab845bf58de0089ff44) * GH-2801: Fix re-instantiation of analyzers that were just removed. (Robin Sommer, Corelight) (cherry picked from commit a41e282a7d159f42f67f40778a6b16169316e630) 5.0.7-7 | 2023-04-03 10:22:33 -0700 * Update zeekctl submodule [nomail] [skip ci] (Tim Wojtulewicz, Corelight) * Default known stores to false to avoid Broker stores (Tim Wojtulewicz, Corelight) (cherry picked from e47980143cfa40b8634655d573d059eaafd8336a) 5.0.7-5 | 2023-03-16 10:05:15 -0700 * Bump spicy-plugin to v1.3.26. (Arne Welzel, Corelight) 5.0.7-2 | 2023-03-02 13:49:10 -0700 * Bump spicy-plugin to v1.3.24. (Benjamin Bannier, Corelight) 5.0.7 | 2023-02-21 11:13:22 -0700 * ftp: Limit user, password, arg and reply_msg column sizes in log (Arne Welzel, Corelight) The user and password fields are replicated to each of the ftp.log entries. Using a very large username (100s of KBs) allows to bloat the log without actually sending much traffic. Further, limit the arg and reply_msg columns to large, but not unbounded values. Closes #144. * Allow async DNS responses with zero TTLs to be considered valid after expriation (Tim Wojtulewicz, Corelight) This fixes a potential memory leak when getting responses for asnyc DNS requests where the TTL value on the response is zero. We were immediately considering the request as expired and never removing it from the map of requests. This lead to the DNS_Mgr eventually stopping processing of async requests. * Add max_size argument for find_all/find_all_ordered BIFs (Tim Wojtulewicz, Corelight) 5.0.6-9 | 2023-02-17 14:40:31 -0700 * Drop Debian 11 ARM build from Cirrus (Tim Wojtulewicz, Corelight) This was added as part of the cherry-pick for fa6e01d99660e6d73d5930e245505c2414b760c8 but the build is failing for other reasons on Cirrus. We haven't been doing ARM builds for this release yet, so it's easier to just remove it for now. * Move get_relative_seq and get_segment_len to public static methods (Tim Wojtulewicz, Corelight) * HashKey: Do not call Describe() unconditionally in DEBUG mode (Arne Welzel, Corelight) An unnecessary overhead of the Hash() method was uncovered for DEBUG builds due to computing a description of every HashKey() even when the DBG_HASHKEY stream is not enabled. Squelch it. * btest/decode_base_errors: Avoid binary output in baseline (Arne Welzel, Corelight) * Base64: report byte as positive integer (Arne Welzel, Corelight) * netbios_decode: use unsigned char for result (Arne Welzel, Corelight) * analyzer/http: Do not assume char is signed (Arne Welzel, Corelight) 5.0.6-4 | 2023-02-14 07:43:01 -0700 * Bump spicy-plugin to v1.3.23. (Benjamin Bannier, Corelight) * CI: Call 'brew install' for macOS to ensure we're using openssl@3 on all hosts (Tim Wojtulewicz, Corelight) * Add LTS tag to docker image builder workflow (Tim Wojtulewicz, Corelight) 5.0.6 | 2023-02-01 07:51:36 -0700 * Update CHANGES, VERSION, and NEWS for 5.0.6 release (Tim Wojtulewicz, Corelight) * ci/alpine: Use musl-fts-dev rather than fts-dev (Arne Welzel, Corelight) With the release of alpine 3.17 (2022-11-22), the fts-dev package was renamed to musl-fts-dev. Change it accordingly. This is currently breaking CI in a fork of zeek/zeek. Suspect some caching prevents it from breaking here currently. Alpine 3.16 is still supported till 2024-05-23, but we're essentially not doing CI testing for it anymore. Appears to be in line with the comment in .cirrus.yml * Restore/rename field in SMB2::Fscontrol record type (Tim Wojtulewicz, Corelight) b41a4bf06dc2f3a4218cfa0a95b536ec9d253917 removed a field from this record because it had a duplicate name as another field. The field does need to exist, but it needs the correct name. * testing/http: http-11-request-then-cruft (Arne Welzel, Corelight) A client sends a "proper" HTTP/1.1 request and afterwards a few T /\n\n sequences. The latter ones aren't logged. * testing/http: Add pcap extracted from m5-long external test-suite (Arne Welzel, Corelight) This tests that the HTTP version is now updated if it changes in the course of a connection. * analyzers/http: Update request_version on subsequent SetVersion() calls (Arne Welzel, Corelight) The #124 PR introduced special treatment when HTTP version 0.9 was set. With #127, a reproducer that set HTTP/1.0 in the first request was created and subsequent requests wouldn't reset to HTTP version 0.9. This is subtle, but doesn't seem like things fall apart. Improves runtime from 20 seconds to 2 seconds for the given reproducer. Fixes #127. * testing/ftp: Add tests and pcaps with invalid reply lines (Arne Welzel, Corelight) These have been created artificially. The tests show that for an invalid reply line without a numeric code, with a numeric code < 100 or a numeric code not followed by a space we now raise an analyzer violation and disable the analyzer. * ftp: Harden reply handing a bit and don't raise bad replies to script-land (Arne Welzel, Corelight) This improves runtime of the oss-fuzz generated traffic in #125. Specifically, that reproducers included a 064- reply code that was interpreted as needing to be continued. Also, return after AnalyzerViolations() for server replies rather than propagating bad replies them to script-land. This trusts server's to generally behave according to specification. * ftp: ignore invalid commands (Arne Welzel, Corelight) Do not propagate wrong FTP commands to script land. * Update broker submodule for sim_clock::advance_time performance fix (Dominik Charousset, Corelight) * smb1: Prevent accessing uninitialized referenced_tree (Arne Welzel, Corelight) The added pcap was created from an OSS Fuzz test case and is borderline valid SMB traffic, but it triggered a scripting error. (cherry picked from commit bb094e30bfe7b1f7f3acf8d879b12719a429e1fc) * analyzer/files: handle non-analyzer names in describe_file() (Arne Welzel, Corelight) When a fa_file object is created through the use of Input::add_analysis(), the fa_file's source is likely not valid representation of an analyzer's tag and a Files::describe() should not error and instead return an empty description. (cherry picked from commit a349679a4c77d2808bdf8b04f1ea8552edc9714d) * frameworks/notice: Handle fa_file with no or more than a single connection better (Arne Welzel, Corelight) * When a file is transferred over multiple connection, have create_file_info() just pick the first one instead of none. * Do not unconditionally assume cid and cuid as set on a Notice::FileInfo object. (cherry picked from commit a349679a4c77d2808bdf8b04f1ea8552edc9714d) * intel/seen/file-names: Use file_over_new_connection() (Arne Welzel, Corelight) (cherry picked from commit a004bdf5d950e7ef4d9c2ce7112576a76fee438a) * input: Add location info for invalid enums (Arne Welzel, Corelight) (cherry picked from commit a7695bc784bc0bfd108dbe2c72706eabc6b8dfae) 5.0.5-2 | 2023-01-23 13:07:41 -0700 * Bump spicy to v1.5.3. (Benjamin Bannier, Corelight) 5.0.5 | 2023-01-09 10:02:52 -0700 * Update CHANGES, VERSION, and NEWS for 5.0.5 release (Tim Wojtulewicz, Corelight) * CI: drop macOS Big Sur, add macOS Ventura (Tim Wojtulewicz, Corelight) (cherry picked from commit 555f705e2db263757c7c6ac580afcaf20c299065) * Switch macOS Monterey build to new m1-based VM (Tim Wojtulewicz, Corelight) (cherry picked from commit a927c4a998f2cefb136d9d1e712f6e441b3e6720) * Update broker submodule to version 2.3.6 [nomail] (Tim Wojtulewicz, Corelight) 5.0.4 | 2022-11-22 09:05:55 -0700 * Update CHANGES, VERSION, and NEWS for 5.0.4 release (Tim Wojtulewicz, Corelight) * http: Heuristic around rejecting malformed HTTP/0.9 traffic (Arne Welzel, Corelight) oss-fuzz generated "HTTP traffic" containing 250k+ sequences of "T\r\r" which Zeek then logged as individual HTTP requests. Add a heuristic to bail on such request lines. It's a bit specific to the test case, but should work. * ftp: Introduce FTP::max_command_length (Arne Welzel, Corelight) oss-fuzz produced FTP traffic with a ~550KB long FTP command. Cap FTP command length at 100 bytes, log a weird if a command is larger than that and move on to the next. Likely it's not actual FTP traffic, but raising an analyzer violation would allow clients an easy way to disable the analyzer by sending an overly long command. The added test PCAP was generated using a fake Python socket server/client. * Add additional length checking to IPv6::ToVal (Tim Wojtulewicz, Corelight) * Update broker submodule [nomail] (Tim Wojtulewicz, Corelight) 5.0.3 | 2022-11-08 11:31:06 -0700 * {http,smtp}/entities: Align header regexes with extract_filename_from_content_disposition() (Arne Welzel, Corelight) * analyzer/HTTP: Call TCP_ApplicationAnalyzer::Done() after RequestMade()/ReplyMade() (Arne Welzel, Corelight) * test: Add btest verifying max_analyzer_violations functionality (Arne Welzel, Corelight) The pcap has been generated roughly based on the example found on wikipedia with some added garbled response after the STAT command from the client. * analyzer: Cap analyzer violations at 1000 per analyzer instance (Arne Welzel, Corelight) * Report an error if a key is too big for a DictEntry (Tim Wojtulewicz, Corelight) * Use larger size variable for DictEntry key_size to avoid overflows with big keys (Tim Wojtulewicz, Corelight) * ftp: Limit pending commands to FTP::max_pending_commands (default 20) (Arne Welzel, Corelight) * Update baselines after AnalyzePacket changes (Arne Welzel, Corelight) Mainly, for bad headers we do not (can not) put addresses into the log anymore. Some were rather obviously bogus previously already. * protocol/ip: Only attach IP_Hdr to Packet if valid (Arne Welzel, Corelight) 5.0.2-15 | 2022-11-07 10:22:53 -0700 * Bump spicy-1.5.2 and spicy-plugin-1.3.22. (Benjamin Bannier, Corelight) * Update DNS type strings to match correct mappings (Tim Wojtulewicz, Corelight) * Backport Dictionary performance fixes from master (Tim Wojtulewicz, Corelight) This is the changes from https://github.com/zeek/zeek/pull/2478. 5.0.2-7 | 2022-11-02 07:48:53 -0700 * Fix a couple of failing tests from recent cherry-picks (Tim Wojtulewicz, Corelight) * Add toplevel build dir to `PATH` for dev helper scripts. (Benjamin Bannier, Corelight) * IPBasedAnalyzer/TCPSessionAdapter: Fix TCP reassembly decision for known port analyzers (Arne Welzel, Corelight) * packet_analysis: Do not raise analyzer_confirmation per-packet for tunnels (Arne Welzel, Corelight) * fix for deprecated when's where the inner frame is larger than the outer frame (Vern Paxson, Corelight) 5.0.2 | 2022-09-19 11:56:36 -0700 * Release 5.0.2. * Stop signal-masking upon running unit tests (Christian Kreibich, Corelight) (cherry picked from commit 48486b4156e2a7b689566a6a9a3e4d65c2700430) * Pause signal-masking during script parsing (Christian Kreibich, Corelight) (cherry picked from commit 48486b4156e2a7b689566a6a9a3e4d65c2700430) * Add btests to verify Zeek's handling of SIGTERM and reading stdin (Christian Kreibich, Corelight) (cherry picked from commit 48486b4156e2a7b689566a6a9a3e4d65c2700430) * Add procps/procps-ng to several CI Docker images (Christian Kreibich, Corelight) (cherry picked from commit 48486b4156e2a7b689566a6a9a3e4d65c2700430) * CI: remove FreeBSD 13.0, update FreeBSD 12 to 12.3 (Tim Wojtulewicz, Corelight) (cherry picked from commit 5b0d20b5651b1fea3982f9e1d4a6f7df4359b5af) * Add better length check to BuildOptionsVal() (Tim Wojtulewicz, Corelight) * irc: Check parts.size() before removing the nick (Arne Welzel, Corelight) ...also, don't remove the nick twice. * Add some additional checks for caplen in ICMP analyzer (Tim Wojtulewicz, Corelight) * smb: Unify transaction_data_to_val() usage (Arne Welzel, Corelight) the data/string handling in smb1-com-transaction.pcac was improved with c75519ca8869448efc488598090f96d63352312f, re-use the added functionality also for smb1-com-transaction-secondary.pac to avoid buffer overflows. * Check IP payload length before casting to transport structure (Tim Wojtulewicz, Corelight) 5.0.1 | 2022-08-26 10:25:19 -0700 * Release 5.0.1. * Fix potential overflow in modbus analyzer's bytestring_to_coils (Tim Wojtulewicz, Corelight) * Reset packet cap_len before returning from IP::AnalyzePacket (Tim Wojtulewicz, Corelight) * Swap DNS EDNS field order to match script-land type (Tim Wojtulewicz, Corelight) * Add some sanity checking to BadARPEvent method (Tim Wojtulewicz, Corelight) * Check for valid ip_hdr length before trying to make a Val out of it (Tim Wojtulewicz, Corelight) * Update broker submodule to 2.3.2 release tag (Tim Wojtulewicz, Corelight) 5.0.0-18 | 2022-08-26 09:17:29 -0700 * Revert to regular peering fn for ssl_auth_failure (Dominik Charousset, Corelight) With the fixed Broker version, the `ssl_auth_failure` btest can use the regular `peer` function again. (cherry picked from commit 2d7bfb4ca787d6b270bd11ab57effec1f95219f2) * Avoid race conditions in broker.ssl_auth_failure btest (Tim Wojtulewicz, Corelight) (cherry picked from commit c88a362d1b401823dbada62b9c2304124fcb0ca6) * Avoid race conditions in broker.ssl-mismatch test (Dominik Charousset, Corelight) (cherry picked from commit d3ad3c618a62c6b415a47c54fdd679f26c9ed34f) 5.0.0-14 | 2022-08-24 10:34:55 -0700 * Truncate Docker image artifact at end of Docker CI workflow (Christian Kreibich, Corelight) 5.0.0-13 | 2022-08-24 10:25:03 -0700 * Bump Spicy dependencies to spicy-1.5.1 and spicy-plugin-1.3.17. (Benjamin Bannier, Corelight) 5.0.0-11 | 2022-08-24 10:22:33 -0700 * Use correct variable to pass existing Spicy root to spicy-plugin. (Benjamin Bannier, Corelight) spicy-plugin can find Spicy in paths given by `SPICY_ROOT_DIR` while `./configure` instead set `SPICY_ROOT`. With this patch we now set the correct variable. We also adjust variations of the previous variable name with different capitalization which caused us to not properly configure spicy-plugin (which triggers finding Spicy in its given prefix). Closes #2363. 5.0.0-8 | 2022-08-23 16:48:21 -0700 * Remove reporter warning for bad IP protocols (Tim Wojtulewicz, Corelight) It turns out that this can be *very* spammy on networks where we're receiving lots of these packets, and can fill up the reporter log very quickly. Weirds are already reported in all of these cases anyways, so it doesn't make sense to log a reporter warning too. (cherry picked from commit 40b1452905c0eed4d96300ce1aaf87a08166e396) * Remove use of fallible `get_conn_transport_proto` in `analyzer_violation`. (Benjamin Bannier, Corelight) When setting up the DPD info we previously would get the `transport_proto` for the connection with `get_conn_transport_proto`. This function takes a `conn_id` and would fail fatally if the connection for the given ID was unknown. It seems it was possible to run into such scenarios when the `analyzer_violation` event was processed after the connection had been cleaned up. We now get the `transport_proto` directly from the ports in the `connection` passed into `analyzer_violation` via `get_port_transport_proto` which cannot fail. (cherry picked from commit b27c114d6871aa1d52699532b6035dc704195dfa) * Avoid calling safe_realloc unnecessarily from ODesc::Grow() (Tim Wojtulewicz, Corelight) (cherry picked from commit 0728676fd6a135d6b6b0dab1c67d2f3d41d67577) * Fix default build with `--disable-cpp-tests`. (Benjamin Bannier, Corelight) For a build configured with `--disable-cpp-tests` we globally set `-DDOCTEST_CONFIG_DISABLE`. Spicy does not expect this define to be used and its unit tests fail to build with it. This patch adds a Zeek-side workaround for that. We now do not build the Spicy unit test targets as part of `ALL` anymore. Closes #2238. (cherry picked from commit f3313df74326d678bbe57b40fc2cf44f8ad456ff) * GH-2229: Fix some typos in weak-keys.zeek (Tim Wojtulewicz, Corelight) (cherry picked from commit ee88aa3acbe541f679b202687f00f0aa242e77a3) 5.0.0-2 | 2022-08-23 11:59:23 -0700 * Add missing opensuse-leap-15.4 files for CI (Tim Wojtulewicz, Corelight) 5.0.0 | 2022-07-05 12:34:24 -0700 * Management framework: bump zeek-client and cluster testsuite (Christian Kreibich, Corelight) (cherry picked from commit b96d8afa52d8aeafed7dcd3978ff437b57193cf8) * Bump Spicy and spicy-plugin to their latest release. (Benjamin Bannier, Corelight) (cherry picked from commit 94c974744084d4eb164730161495f92cf39d104b) * Bump broker submodule to pick up two fixes (Tim Wojtulewicz, Corelight) - https://github.com/zeek/broker/pull/244 - https://github.com/zeek/broker/pull/245 (cherry picked from commit a6dd3af831471d88d164244c5dcdc0e4e931e768) * News entry about Bloom Filter changes in 5.0. (Johanna Amann, Corelight) (cherry picked from commit ce410b7ed521311f1b237966e1c9511e9dadbccf) 5.0.0-rc2 | 2022-06-23 13:20:24 -0700 * Bump Spicy to the latest development version. (Benjamin Bannier, Corelight) (cherry picked from commit e60c305ac1d1b711ee38809a0d0cca76084cfc59) * Management framework: support for cluster node restarts (Christian Kreibich, Corelight) - bump external cluster testsuite - bump zeek-client - edit pass over docstrings - node restart support - more consistent Supervisor interaction in the agent - log the controller's startup deployment attempt - bugfix for a get_id_value corner case - minor timeout bugfix - make "result" argument plural in multi-result response events (cherry picked from commit 3287b8b793e71e28bfcdd9fe4f6712d4a2707d5e) * Management framework: separate config staging and deployment (Christian Kreibich, Corelight) - bump external cluster testsuite - bump zeek-client - rename set_configuration events to stage_configuration - trigger deployment upon when instances are ready - more resilient node shutdown upon deployment - re-trigger deployment upon controller launch - move most deployment handling to internal function - distinguish internally and externally requested deployments - track instances by their Broker IDs - tweak Supervisor event logging - make helper function a local - rename "log_level" to "level" - add "finish" callback to requests - add a helper for rendering result vectors to a string - agents now skip re-deployment of current config - suppress notify_agent_hello upon Supervisor peering - introduce state machine for configs and persist them - introduce deployment API in controller - rename agent "set_configuration" to "deploy" - consistency fixes to the Result record (cherry picked from commit 54f2f28047ad8910c0122018148fc34c57c8cb13) * Provide zeek-client by default (Christian Kreibich, Corelight) (cherry picked from commit d0578cf34597642e6b67c77ddce7d51d18511d5d) * Management framework: config validation (Christian Kreibich, Corelight) - bump external cluster testsuite - bump zeek-client - add config validation - improvements to port auto-enumeration - control output-to-console in Supervisor - handle no-instances corner case in set-config correctly (cherry picked from commit 4deacefa4c753b5f45e8504161e7b8561fe01a7c) * Management framework: bump external cluster testsuite (Christian Kreibich, Corelight) (cherry picked from commit e7f60d41280d91ea6c7d18059188b38dcc2f618a) * Management framework: make agents support zeek-archiver invocations (Christian Kreibich, Corelight) (cherry picked from commit e7f60d41280d91ea6c7d18059188b38dcc2f618a) * Management framework: fix module naming typo (Christian Kreibich, Corelight) (cherry picked from commit e7f60d41280d91ea6c7d18059188b38dcc2f618a) * Management framework: bump external cluster testsuite (Christian Kreibich, Corelight) (cherry picked from commit 216aaa5774b24a96aa44fca0e90c3c455f7e9e8f) * Management framework: switch default network visibilities (Christian Kreibich, Corelight) (cherry picked from commit 216aaa5774b24a96aa44fca0e90c3c455f7e9e8f) * Management framework: port auto-assignment (Christian Kreibich, Corelight) - support auto-assignment of ports in cluster nodes - also use send_set_configuration_response_error elsewhere - minor log formatting tweak, for consistency - bump zeek-client to pull in relaxed port handling - bump external cluster testsuite (cherry picked from commit 763b0c8d10d064f8a127dbc24c9eccbfd9d15dea) * CI: Add Fedora 36, remove Fedora 34 (Johanna Amann, Corelight) (cherry picked from commit 94e1d5786efc4ea480670ec70062ddc71ed6f2f5) * Update broker submodule to fix put_unique issues from 5.0RC1 (Tim Wojtulewicz, Corelight) (cherry picked from commit 5b85cd6dfdfcee0d0953ce9bda15fb40cf139f44) 5.0.0-rc1.7 | 2022-06-17 09:15:37 -0700 * Bump Spicy and spicy-plugin to latest dev snapshot. (Benjamin Bannier, Corelight) This bump brings in fixes for the following issues: - zeek/spicy#1204 - zeek/spicy#1208 - zeek/spicy#1210 - zeek/spicy-plugin#112 - zeek/spicy-plugin#117 (cherry picked from commit d15ba0889088fc889a6bc841ddf71c32e94d93e1) * Bump Spicy to the latest development version. (Benjamin Bannier, Corelight) (cherry picked from commit d796b3dd576877d48f5e225ce4248dead9fafde0) 5.0.0-rc1.5 | 2022-06-15 12:39:13 -0700 * CI: Add Ubuntu 22.04 (LTS Release, EOL 04/2027) (Johanna Amann, Corelight) * Remove duplicate entry about websockets from 5.0 NEWS (Tim Wojtulewicz, Corelight) (cherry picked from commit 5668a6a6de9fc583e7e0f5ecae94c14d853dc626) 5.0.0-rc1 | 2022-06-03 12:09:34 -0700 * Keep make dist from deleting all paths containing 'build' [skip ci] (Tim Wojtulewicz) * Remove duplicate entry about websockets from 5.0 NEWS (Tim Wojtulewicz, Corelight) * NEWS updates for 5.0 (Vern Paxson, Corelight) * ZAM test suite updates due to recent changes (Vern Paxson, Corelight) 5.0.0-dev.575 | 2022-06-03 10:55:47 -0700 * Management framework: bump external cluster testsuite (Christian Kreibich, Corelight) * Management framework: bump zeek-client to bring back to master (Christian Kreibich, Corelight) 5.0.0-dev.572 | 2022-06-03 10:31:51 -0700 * Fix hang in DNS analyzer with malformed packet (Tim Wojtulewicz, Corelight) (cherry picked from commit dfc9f77b68f6d02d059ade322b9c132e32c26872) 5.0.0-dev.571 | 2022-06-03 09:50:45 -0700 * Additional &is_used tags in the Netcontrol and Sumstats frameworks (Christian Kreibich, Corelight) When running a cluster, these functions only get called in select node types and could trigger no-caller warnings on stderr. 5.0.0-dev.569 | 2022-06-03 09:50:01 -0700 * Management framework: bump zeek-client to pull in rendering tweaks (Christian Kreibich, Corelight) * Management framework: bump external cluster testsuite (Christian Kreibich, Corelight) * Mark lookup_asn() BIF as deprecated in v6.1 (Phil Rzewski) * Define geo_autonomous_system record type (Phil Rzewski) * Add lookup_autonomous_system() BIF that returns AS number and org (Phil Rzewski) 5.0.0-dev.559 | 2022-06-02 16:58:58 -0700 * Mark lookup_asn() BIF as deprecated in v6.1 (Phil Rzewski) * Define geo_autonomous_system record type (Phil Rzewski) * Add lookup_autonomous_system() BIF that returns AS number and org (Phil Rzewski) 5.0.0-dev.553 | 2022-06-02 13:16:44 -0700 * Make broker.web-socket-events test require the Python websockets package (Christian Kreibich, Corelight) 5.0.0-dev.551 | 2022-06-02 12:19:32 -0700 * Expand scripts.base.frameworks.intel.cluster-transparency test (Christian Kreibich, Corelight) * Fix races in scripts.base.frameworks.intel.cluster-transparency-with-proxy test (Christian Kreibich, Corelight) * Add Intel::send_store_on_node_up boolean to control min_data_store delivery (Christian Kreibich, Corelight) * Fix a -Wstringop-overflow memset size warning with GCC 12 (Christian Kreibich, Corelight) 5.0.0-dev.545 | 2022-06-02 12:00:53 +0200 * Add WebSocket support for exchanging events with external clients. (Robin Sommer, Corelight) This exposes Broker's new WebSocket support in Zeek. To enable it, call `Broker::listen_websocket()`. Zeek will then start listening on port 9997 for incoming WebSocket connections. See the Broker documentation for a description of the message format expected over these WebSocket connections. 5.0.0-dev.540 | 2022-06-01 11:08:42 -0700 * GH-2136: Clean up DNS_Mgr before shutting down (Tim Wojtulewicz, Corelight) 5.0.0-dev.538 | 2022-06-01 09:20:21 -0700 * GH-2101: Update cmake submodule to fix c-ares include path problem (Tim Wojtulewicz, Corelight) 5.0.0-dev.535 | 2022-05-31 12:58:32 -0700 * Management framework updates (Christian Kreibich, Corelight) - bump external cluster testsuite - bump zeek-client to pull in set-config rendering - enable stdout/stderr reporting - Supervisor extensions for stdout/stderr handling - disambiguate redef field names in agent and controller - move to ResultVec in agent's set_configuration response - tune request timeout granularity and interval - verify node starts when deploying a configuration - a bit of debug-level logging for troubleshooting 5.0.0-dev.525 | 2022-05-31 12:53:01 -0700 * Add Supervisor::node_status notification event (Christian Kreibich, Corelight) 5.0.0-dev.523 | 2022-05-31 11:13:47 -0700 * Update spicy-plugin with change that checks for zeek version (Tim Wojtulewicz, Corelight) * deprecation messages for unused base script functions (Vern Paxson, Corelight) * clearer messages for warning about unused functions (Vern Paxson, Corelight) * annotate orphan base script components with &deprecated (Vern Paxson, Corelight) * annotate base scripts with &is_used as needed (Vern Paxson, Corelight) * --no-usage-warnings flag to suppress analysis (Vern Paxson, Corelight) * support for associating &is_used attributes with functions (Vern Paxson, Corelight) * classes for evaluating function/hook/event usage (Vern Paxson, Corelight) * broader support for AST traversal, including Attr and Attributes objects (Vern Paxson, Corelight) * include attributes in descriptions of sets and tables (Vern Paxson, Corelight) 5.0.0-dev.508 | 2022-05-27 14:33:47 -0700 * Update zeek-aux submodule to fix a compiler warning (Tim Wojtulewicz, Corelight) 5.0.0-dev.505 | 2022-05-26 16:08:42 -0700 * Management framework updates (Christian Kreibich, Corelight) - bump zeek-client to pull in instance serialization fixes - bump external cluster testsuite - update agent-checkin test to reflect recent changes - place each Zeek process in its own working dir - set defaults for log rotation and persistent state - add spool and state directory config settings - establish stdout/stderr files also for cluster nodes - default to having agents check in with the (local) controller - move role variable from logging into framework-wide config - distinguish supervisor/supervisee when loading agent/controller - simplify agent and controller stdout/stderr files - prefix the management logs with "management-" - comment and layouting tweaks, no functional change - rename env var that labels agents/controllers - increase robustness of agent/controller naming * Add some missing NEWS entries (Tim Wojtulewicz, Corelight) 5.0.0-dev.488 | 2022-05-26 08:23:42 -0700 * GH-2054: Allow nulls as separators for join_string_vec (Tim Wojtulewicz, Corelight) 5.0.0-dev.485 | 2022-05-25 12:09:42 -0700 * Propagate BPF_Program error message to script land (Tim Wojtulewicz, Corelight) * Allow pcap pktsrc to use other BPF_Program::Compile method (Tim Wojtulewicz, Corelight) 5.0.0-dev.482 | 2022-05-25 09:40:18 -0700 * Bump Spicy and spicy-plugin. (Benjamin Bannier, Corelight) * Clean up test `spicy.spicyz-jit`. (Benjamin Bannier, Corelight) This test was created from a copy of `spicy.spicyz-aot`, but updated incompletly. This patch removes the last ahead-of-time compilation bits. 5.0.0-dev.479 | 2022-05-25 09:38:20 -0700 * GH-2125: Fix compiler warnings in ScriptProfile (Tim Wojtulewicz, Corelight) 5.0.0-dev.477 | 2022-05-25 09:37:58 -0700 * Use ZEEK_DICT_DEBUG more in Dict.cc instead of DEBUG (Tim Wojtulewicz) 5.0.0-dev.475 | 2022-05-25 09:37:19 -0700 * Move fuzzer setup in CMake below builtin plugin setup (Tim Wojtulewicz, Corelight) * Remove flex/bison flags from CI build (Tim Wojtulewicz, Corelight) 5.0.0-dev.470 | 2022-05-24 11:55:37 -0700 * Ensure presence of Zeek-related directories in toplevel CMakeLists.txt (Christian Kreibich, Corelight) * Add base/misc/installation.zeek, with Zeek installation directories (Christian Kreibich, Corelight) * Add scripts.base.misc.installation btest (Christian Kreibich, Corelight) 5.0.0-dev.466 | 2022-05-23 17:47:08 -0700 * Add DOCKERFILE_VERSION variable to all CI dockerfiles (Tim Wojtulewicz, Corelight) 5.0.0-dev.462 | 2022-05-19 11:45:38 -0700 * speed up ZAM compilation by capping function size when inlining (Vern Paxson, Corelight) 5.0.0-dev.460 | 2022-05-19 11:24:50 -0700 * Zeekify the scripts.base.utils.paths test (Christian Kreibich, Corelight) * Expand build_path() function to handle empty dir arguments gracefully (Christian Kreibich, Corelight) 5.0.0-dev.457 | 2022-05-19 10:51:45 -0700 * Rework .cirrus.yml a bit to commonize some parts (Tim Wojtulewicz) * Enable 'greedy' mode for other CI builds (Tim Wojtulewicz, Corelight) * Enable ccache for Cirrus builds (Tim Wojtulewicz, Corelight) * Fix SSL tests on Centos Stream 9 (Johanna Amann, Corelight) SHA1 signatures are only allowed in the legacy policy. I have no idea where this is documented - I found it in a patch for the OpenSSL source package. Also adding tar as a specific dependency, since we have a test that requires it, and at least for me it was not present in a minimal installation. Fixes #2115 5.0.0-dev.451 | 2022-05-19 09:13:13 -0700 * Disable Spicy in Zeek Coverity builds (Tim Wojtulewicz, Corelight) 5.0.0-dev.449 | 2022-05-18 11:11:58 -0700 * Always generate helper scripts for Spicy. (Benjamin Bannier, Corelight) We previously only generated these files if configuring a build which included Spicy. This meant that e.g., `zeek-path-dev.*` scripts would emit errors for non-existing paths if Spicy was not included. We now always generate the helper scripts so we can reference them regardless of build configuration. If Spicy is not included the files will still be sourced, but have no actual effect. 5.0.0-dev.447 | 2022-05-18 11:07:30 -0700 * Partially revert OpenSSL hashing changes (Johanna Amann, Corelight) This commit partially reverts the changes that we made in 6217851d6db3859b2add34773be5a6b3ecba49f0. It turns out that reverting to the legacy OpenSSL API means that Zeek can no longer be run in FIPS mode for old (but still used) versions of OpenSSL. This commit thus uses a bunch of #ifdefs to support both implementations simultaneously. 5.0.0-dev.445 | 2022-05-18 11:07:02 -0700 * GH-1672: Keep a reference to a table being expired, just in case (Tim Wojtulewicz, Corelight) 5.0.0-dev.442 | 2022-05-17 09:15:27 -0700 * Remove unnecessary uses of util::fmt() (Tim Wojtulewicz, Corelight) 5.0.0-dev.440 | 2022-05-17 08:50:15 -0700 * Add NEWS item for new requirements due to default-enabled Spicy support. (Benjamin Bannier, Corelight) * Include headers in binary dir before other headers. (Benjamin Bannier, Corelight) If e.g., optional dependencies are installed in the same prefix as a previous Zeek installation, the build would have previously found `zeek-config.h` from the external prefix instead of from the current build. With this patch we now include the current binary dir before any previously configured include directories (here in particular: optional dependencies). With that the correct `zeek-config.h` is found. * Increase memory in CI. (Benjamin Bannier, Corelight) Spicy requires substantial amounts of RAM to compile. In this patch we bump the amount of RAM available in CI so a build including Spicy can be performed. * Add tests for bundled Spicy infrastructure. (Benjamin Bannier, Corelight) * Include spicy in build. (Benjamin Bannier, Corelight) * Add `bare_mode` bif. (Benjamin Bannier, Corelight) * Change test so included plugins can load notice framework. (Benjamin Bannier, Corelight) * Do not log function arguments in test. (Benjamin Bannier, Corelight) This was excercising hooks to validate that they were called. For that it uses pretty verbose logging. Since version numbers are not stable and hard to canonify this script already unloads the version module (there is e.g., code in `base/misc/version` which splits the version strings and works on single components which might appear like pretty ordinary floating point numbers in the log). This test however worked under the assumption that nothing in bare mode loads `base/misc/version` which is hard to guarantee, especially considering that plugins can be embedded and might load that module themself. With this patch we now make the logging slightly less verbose so that function call arguments are not logged anymore. 5.0.0-dev.431 | 2022-05-16 17:00:24 -0700 * Store location during Ascii input parsing for use in error messages (Tim Wojtulewicz, Corelight) * Include file information in input reader error messages (Tim Wojtulewicz, Corelight) 5.0.0-dev.427 | 2022-05-16 16:59:31 -0700 * NEWS items for script profiling, "footprint" BiFs, and Trigger method deprecations (Vern Paxson, Corelight) * make deprecation advice a bit more helpful (Vern Paxson, Corelight) * add deprecation for Trigger Cache/Lookup interfaces (Vern Paxson, Corelight) * helper scripts for -O C++ maintenance (Vern Paxson, Corelight) * "-O use-C++" now fails if no C++ bodies wound up being applicable (Vern Paxson, Corelight) * update environemnt variable names to match command-line arguments (Vern Paxson, Corelight) * final changes for -O C++ feature completeness, mainly "when" statement support (Vern Paxson, Corelight) * Change "when" statements that don't require closures to use new implementation. (Vern Paxson, Corelight) Provide hooks for script optimization access to "when" statements. Regularize treatment of naming and timeouts for Triggers. * switch cached Trigger values to be opaque-and-generic, rather than assuming CallExpr's (Vern Paxson, Corelight) * tracking of when statements/expressions occur in a "when" context (Vern Paxson, Corelight) * fix bug that sometimes associated the wrong location with global statements (Vern Paxson, Corelight) * fix incomplete AST traversal of "when" statements (Vern Paxson, Corelight) * add some missing "when" closures (Vern Paxson, Corelight) * regularize test suite names to avoid custom/outdated suffices (Vern Paxson, Corelight) 5.0.0-dev.410 | 2022-05-16 13:14:31 -0700 * Call pip using python -m flag during FreeBSD CI prep (Tim Wojtulewicz, Corelight) 5.0.0-dev.408 | 2022-05-16 09:10:53 -0700 * Store hash analyzer kind StringVals for later reuse (Tim Wojtulewicz, Corelight) * Add Component::GetCanonicalNameVal to avoid extra StringVal allocations (Tim Wojtulewicz, Corelight) * Avoid type unnecessary type conversion in ComponentManager::GetComponentName (Tim Wojtulewicz, Corelight) * Only set connection history if strings differ (Tim Wojtulewicz, Corelight) 5.0.0-dev.403 | 2022-05-16 09:10:15 -0700 * Explicitly define copy and move operators/constructors for ScriptProfileStats (Tim Wojtulewicz, Corelight) Fixes Coverity 1488768 5.0.0-dev.401 | 2022-05-16 09:09:37 -0700 * Add --retry to curl invocations in update-traces (Tim Wojtulewicz, Corelight) 5.0.0-dev.399 | 2022-05-16 09:09:05 -0700 * GH-2103: Disable leak detection for bifs.footprint in another way (Tim Wojtulewicz, Corelight) 5.0.0-dev.396 | 2022-05-13 16:50:53 -0700 * Fix UB during early shutdown on OpenSSL state (Dominik Charousset, Corelight) 5.0.0-dev.393 | 2022-05-13 16:48:56 -0700 * Correcly typify bloom filters in all cases during merge (Johanna Amann) Before, there were edge-cases where the type was not transferred if one of the involved bloom-filters was untypified. Fixes GH-1938 * Add intersect operation for bloom filters (Johanna Amann) Intersecting two bloom filters yields a bloom filter that returns true when an element was contained in both bloom filters. The false positive rate is potentially a bit higher than in the original bloom filters. This operation also works for counting bloom filters, however the counters are discarded and the bloomfilters are converted to basic bloom filters. The reason is that there is no obvious meaning to the counters when two bloom filters are intersected - besides the fact if an element was inserted at all. * Add bloomfilter_decrement bif (Johanna Amann) This bif implements the decrement operation for counting bloom filters. It also clarifies some of the documentation. 5.0.0-dev.385 | 2022-05-12 11:03:25 +0100 * Include certificate information in SSL::Weak_Key notice (Johanna Amann, Corelight) 5.0.0-dev.382 | 2022-05-11 12:56:13 -0700 * tidy up after generating profile (Vern Paxson, Corelight) * fix for coverage reporting for functions that use "when" statements (Vern Paxson, Corelight) * new global_options() BiF to speed up startup, plus a micro-preen (Vern Paxson, Corelight) * hooks for new --profile-scripts option (Vern Paxson, Corelight) * classes for managing script profiles (Vern Paxson, Corelight) * address some holes in script coverage (Vern Paxson, Corelight) * fix for script coverage missing on-exit activity (Vern Paxson, Corelight) * memory management fixes for loggers (Vern Paxson, Corelight) * make curr_CPU_time() broadly available rather than just isolated to ZAM (Vern Paxson, Corelight) * Add exact name of the Top-k algorithm. (Johanna Amann) I needed to figure out which exact algorithm we use for our probabilistic top-k measurements. It turns out that we do not mention this in our source tree at all so far. 5.0.0-dev.368 | 2022-05-09 10:37:44 +0100 * when printing a description of a type, use its name if available (Vern Paxson, Corelight) 5.0.0-dev.363 | 2022-05-06 15:28:34 -0700 * Management framework: consistency tweaks in zeek-client, plus test updates (Christian Kreibich, Corelight) 5.0.0-dev.361 | 2022-05-06 12:47:49 -0700 * Update doctest to v2.4.8 (Tim Wojtulewicz, Corelight) 5.0.0-dev.359 | 2022-05-06 11:32:13 -0700 * to avoid recursion, track all aggregates, not just records (Vern Paxson, Corelight) isolate the internal methods * simpler public calling interface for computing footprint (Vern Paxson, Corelight) * use stack-based set to prevent infinite recursion rather than a static one (Vern Paxson, Corelight) * change value_footprint() to val_footprint() to be more similar to val_size() (Vern Paxson, Corelight) * make including count of container elements non-optional (Vern Paxson, Corelight) * fix for tracking footprints of mutually-recursive records (Vern Paxson, Corelight) * added value_footprint() and global_container_footprints() BiFs (Vern Paxson, Corelight) 5.0.0-dev.347 | 2022-05-05 18:09:44 -0700 * Management framework: add get_configuration_request/response transaction (Christian Kreibich, Corelight) * Update zeek-archiver submodule (Christian Kreibich, Corelight) 5.0.0-dev.344 | 2022-05-04 11:57:50 -0700 * fix for coverage reporting for functions that use "when" statements (Vern Paxson, Corelight) * ZAM maintenance for recent changes and some newly exercised corner cases (Vern Paxson, Corelight) 5.0.0-dev.341 | 2022-05-04 09:23:55 -0700 * Fixes Coverity issues reported from DNS_Mgr merge (Tim Wojtulewicz, Corelight) Reports fixed: 1488295, 1488292, 1488281, 1488279 5.0.0-dev.338 | 2022-05-03 14:40:57 -0700 * fix for sets containing recursive records (Vern Paxson, Corelight) 5.0.0-dev.336 | 2022-05-03 17:09:54 +0100 * Update X.509 and CT trust anchors (Johanna Amann, Corelight) 5.0.0-dev.332 | 2022-04-28 19:52:04 +0000 * Initialize OpenSSL on startup (Dominik Charousset, Corelight) * Avoid double-initialization of OpenSSL (Dominik Charousset, Corelight) * Canonify intel.log in read-file-dist-cluster test (Dominik Charousset, Corelight) * Port Zeek to latest Broker API (Dominik Charousset) 5.0.0-dev.322 | 2022-04-27 21:00:29 +0000 * Disable OpenSSL initialization starting with 1.1.0 (Johanna Amann, Corelight) Starting with OpenSSL 1.1.0, library initialization is no longer required - and might even be harmful. See https://wiki.openssl.org/index.php/Library_Initialization for details. 5.0.0-dev.319 | 2022-04-27 17:42:42 +0000 * Wrap call to doctest's MESSAGE() method in Reporter in try/catch block (Tim Wojtulewicz, Corelight) Also check whether doctest is even enabled before trying to use it. * Pre-initialize c-ares channel object. Fixes Coverity 1488318 (Tim Wojtulewicz, Corelight) * Ask c-ares for the next timeout instead of passing a fixed value (Tim Wojtulewicz, Corelight) * Remove obsolete DNS_Mgr::asyncs_timeouts (Tim Wojtulewicz, Corelight) 5.0.0-dev.314 | 2022-04-27 09:43:23 -0700 * Management framework: consistency fixes around event() vs Broker::publish() (Christian Kreibich, Corelight) 5.0.0-dev.312 | 2022-04-26 09:52:34 -0700 * Rework FindCAres.cmake to not use ExternalProject, fixing OBS builds (Tim Wojtulewicz, Corelight) 5.0.0-dev.309 | 2022-04-22 13:11:12 -0700 * Add DNS fuzzing corpus from c-ares (Tim Wojtulewicz, Corelight) * Set larger UDP buffer to avoid TCP fallback if possible (Tim Wojtulewicz, Corelight) This commit sets the UDP buffer to a larger size, as well as adds an EDNS block to the DNS request passing this size. This allows DNS servers to return larger responses, and in turn allow c-ares to avoid TCP fallback due to requests failing because of the lack of buffer size. * Add new features to IOSource::Manager, used by DNS_Mgr (Tim Wojtulewicz, Corelight) - iosource_mgr can now track write events to file descriptors as well as read events. This adds an argument to both RegisterFd() and UnregisterFd() for setting the mode, defaulting to read. - IOSources can now implement a ProcessFd() method that allows them to handle events to single file descriptors instead of of having to loop through/track sets of them at processing time. * Add out_ptr, use for c-ares interface calls (Tim Wojtulewicz, Corelight) * Store all mappings in a single map instead of split by type (Tim Wojtulewicz, Corelight) This opens up the possibility of storing other request types outside of T_A, T_PTR and T_TXT without requiring redoing the caching. It also fixes the caching code in DNS_Mapping, adding a version number to the start of the cache file so the cache structure can be modified and old caches invalidated more easily. * Add merging to DNS_Mgr::AddResult() to support both ipv4 and ipv6 responses simultaneously (Tim Wojtulewicz, Corelight) * Rework DNS_Mgr API to be more consistent and to support more request types (Tim Wojtulewicz, Corelight) * Replace nb_dns library with C-Ares (Tim Wojtulewicz, Corelight) * Add unit testing for DNS_Mgr and related classes (Tim Wojtulewicz, Corelight) * Update doc gen VM to ubuntu-latest, output cmake version during configure (Tim Wojtulewicz, Corelight) * Use doctest macro to tie Reporter output to test cases (Tim Wojtulewicz, Corelight) * Add const versions of dereference operators for DictEntry (Tim Wojtulewicz, Corelight) * Add DNS fuzzer (Tim Wojtulewicz, Corelight) 5.0.0-dev.288 | 2022-04-22 07:00:56 -0700 * Fix generate-docs action for running on forks. (Benjamin Bannier, Corelight) The generate-docs action previously always required secrets to run so that it could possibly perform a push (if run from a schedule), and to send out an email on failure. Since secrets are unavailable for forks this meant that this action would always fail for PRs from forks. In this patch we use an unauthenticated clone unless running from a schedule. This is fine as for PRs this action would just regenerate the docs to check for errors, but not to actually update them (no push performed). We also change the failure notification step to only execute for scheduled runs. 5.0.0-dev.286 | 2022-04-21 13:34:34 -0700 * Suppress progress dots in zkg's output in Docker package-install check (Christian Kreibich, Corelight) 5.0.0-dev.284 | 2022-04-21 09:17:28 -0700 * Enable vptr undefined behavior check (Tim Wojtulewicz, Corelight) 5.0.0-dev.282 | 2022-04-20 17:17:55 -0700 * Update libkqueue for Coverity and build warning fixes (Tim Wojtulewicz, Corelight) 5.0.0-dev.280 | 2022-04-19 09:42:28 -0700 * Escape special characters in paths before using them as regexes (Tim Wojtulewicz, Corelight) 5.0.0-dev.277 | 2022-04-18 16:38:27 -0700 * Management framework updates (Christian Kreibich, Corelight) - bump external testsuite - allow selecting cluster nodes in get_id_value - minor tweaks to logging component - bump zeek-client to pull in get-id-value command - improve handling of node run states - add get_id_value dispatch - allow dispatching "actions" on cluster nodes. - some renaming to avoid the term "data cluster" - allow agents to communicate with cluster nodes * Avoid whitespace around function type strings in JSON rendering (Christian Kreibich, Corelight) * Disable TSan CI task temporarily while we sort out some intermittent test failures (Tim Wojtulewicz, Corelight) 5.0.0-dev.265 | 2022-04-18 12:45:08 -0700 * state-holding fix: track unique identifiers for Func's in CompHash's, not Func's themselves (Vern Paxson, Corelight) 5.0.0-dev.263 | 2022-04-18 09:22:30 -0700 * Add "Reporter" entry to fix plugin hook_name() vs HookType imbalance (Christian Kreibich, Corelight) The hook_name() list was missing an entry corresponding to HOOK_REPORTER. Co-authored-by: Peter Cullen 5.0.0-dev.259 | 2022-04-14 10:26:29 -0700 * GH-2038: Don't sleep when non-selectable PktSrc has data available (Anthony Coddington) PktSrc::GetNextTimeout always returned a fixed timeout of 20 microseconds for non-selectable packet sources regardless of whether they have packets available. This adds unnecessary delay every FindReadySources poll_interval when packets are available to be read. Instead, for non-selectable packet sources, check whether packets are available and return a timeout of 0 to indicate data is available. This is closer to the behaviour of the old capture loop. This was mitigated somewhat by the fact FindReadySources poll interval defaults to 100 packets, and live sources are added to the ready list regardless of whether they have packets available (unless it is time to force a poll). 5.0.0-dev.257 | 2022-04-14 10:13:28 -0700 * Re-instantiate providing location information to `LoadFile` hooks. (Robin Sommer, Corelight) #1835 subtly changed the semantics of the `LoadFile` plugin hook to no longer have the current script location available for signature files being loaded through `@load-sigs`. This was undocumented behavior, so it's technically not a regression, but since at least one external plugin is depending on it, this change restores the old behavior. 5.0.0-dev.255 | 2022-04-14 10:12:49 -0700 * Fix another crash during dictionary iteration. (Robin Sommer, Corelight) Closes #2017. * Fix assertions in dictionary that can trigger for benign reasons. (Robin Sommer, Corelight) These assertions were checking for a situation that I believe can happen legitimately: a robust iterator pointing to an index that, after some table resizing, happens to be inside the overflow area and hence empty. We'll now move it to the end of the table in the case. * Fix robust iterators when modifying dictionary during iteration. (Robin Sommer, Corelight) When inserting/deleting elements, we now remove their `DictEntries` from any robust iterators' bookkeeping. First, we don't need that information anymore, and second the `DictEntries` contain pointers that may become invalid. I don't know how to write a unit test for this unfortunately because it depends on where exactly things land in the hash table. Btw, memory mgmt for DictEntries is pretty fragile: They contain pointers to both memory they own (`key`) and memory they don't own (`value`). The former type of pointers is shallow-copied on assignment/copy-construction, meaning that there can be multiple instances seemingly owning the same memory. That only works because deletion is manual, and not part of standard destruction. The second type of pointer has a similar problem, except that it's managed externally. It's important to not end up with multiple `DictEntries` pointing to the same value (which is actually what that iterator bookkeeping did). Addresses #2032. 5.0.0-dev.250 | 2022-04-14 09:51:23 -0700 * Split asan/ubsan CI builds, add tsan build (Tim Wojtulewicz, Corelight) 5.0.0-dev.248 | 2022-04-14 08:59:34 -0700 * Disable object-size analysis if optimization set to -O0 (Tim Wojtulewicz, Corelight) 5.0.0-dev.246 | 2022-04-14 10:48:19 +0200 * Allow analyzer violations to explicitly set tag. (Robin Sommer, Corelight) 5.0.0-dev.244 | 2022-04-13 10:52:58 -0700 * Add test to ensure enum_to_int's return values are ordered (Yacin Nadji, Corelight) 5.0.0-dev.242 | 2022-04-13 10:51:21 -0700 * Add unit test for other get_word() version (Tim Wojtulewicz, Corelight) 5.0.0-dev.240 | 2022-04-11 12:46:51 -0700 * Mask our signal handlers' triggering signals around thread creation (Christian Kreibich, Corelight) 5.0.0-dev.238 | 2022-04-11 12:40:02 -0700 * GH-2026: Ensure both protocol and analyzer confirmation and violation events can be called (Tim Wojtulewicz, Corelight) 5.0.0-dev.235 | 2022-04-09 00:08:50 +0000 * Update libkqueue to 2.6.0 release [skip ci] [nomail] (Tim Wojtulewicz) 5.0.0-dev.233 | 2022-04-08 11:30:52 -0700 * Bump submodules to pull in InstallSymlink fix (Christian Kreibich, Corelight) 5.0.0-dev.231 | 2022-04-05 18:04:47 -0700 * fix for ill-formed (complex) &default function (Vern Paxson, Corelight) * type-checking for use of empty table constructors in expressions (Vern Paxson, Corelight) * catch empty constructors used for type inference (Vern Paxson, Corelight) suppress repeated error messages * factoring to make checking of &default attributes externally accessible (Vern Paxson, Corelight) * bug fix for empty table constructors with &default attributes (plus a typo) (Vern Paxson, Corelight) 5.0.0-dev.222 | 2022-04-05 18:04:15 -0700 * reduce interpreter frames for compiled function bodies (Vern Paxson, Corelight) 5.0.0-dev.219 | 2022-04-05 16:07:48 -0700 * Correct origin documentation of the version field in the HTTP log. (Christian Kreibich, Corelight) 5.0.0-dev.217 | 2022-04-04 13:27:32 -0700 * Move new TLS decryption capabilities up to Zeek 5 in NEWS file (Christian Kreibich, Corelight) * Update NEWS to reflect recent updates (Christian Kreibich, Corelight) 5.0.0-dev.214 | 2022-04-04 10:52:41 -0700 * fix & btest for ZAM bug with inlined nested loop (Vern Paxson, Corelight) 5.0.0-dev.212 | 2022-04-04 10:51:20 -0700 * GH-2009: Use auto to fix ZIP analyzer failure on some platforms (Tim Wojtulewicz, Corelight) 5.0.0-dev.210 | 2022-03-28 17:04:51 -0700 * Add cmake-time reporting of bifcl, binpac, and gen-zam used for build (Christian Kreibich, Corelight) * Build Gen-ZAM from a submodule and support use of pre-existing executable (Christian Kreibich, Corelight) 5.0.0-dev.204 | 2022-03-25 15:31:21 -0700 * --event-trace / -E option to generate event trace (Vern Paxson, Corelight) * hooks to support event tracing (Vern Paxson, Corelight) * classes providing event-tracing/dumping functionality (Vern Paxson, Corelight) * provide access to Val internals for event tracing purposes (Vern Paxson, Corelight) * set_network_time() BiF in support of event replaying (Vern Paxson, Corelight) 5.0.0-dev.195 | 2022-03-24 11:01:28 -0700 * switch variable initialization over to being expression-based (Vern Paxson, Corelight) * simplification of Val classes now that they don't have to support initialization (Vern Paxson, Corelight) * rework type inference due to switch from separate initializers to expressions (Vern Paxson, Corelight) * avoid evaluating calls to determine whether an expression value is ignored (Vern Paxson, Corelight) * reworking of expressions to unify =/+=/-= with initialization (Vern Paxson, Corelight) * allow {} expression lists for =/+=/-= RHS (Vern Paxson, Corelight) 5.0.0-dev.177 | 2022-03-23 13:05:51 +0100 * Improve the formatting of the SSL::Info::ssl_history documentation (Johanna Amann, Corelight) 5.0.0-dev.173 | 2022-03-16 15:06:05 -0700 * Fix document generation (Christian Kreibich, Corelight) 5.0.0-dev.169 | 2022-03-10 11:09:37 -0700 * add raw_bytes_to_v6_addr in docs when raw_bytes_to_v4_addr is present (Yacin Nadji, Corelight) * Zero out bytes by default for consistent return value on error (Yacin Nadji, Corelight) * Add tests for raw_bytes_to_v6_addr (Yacin Nadji, Corelight) * Add raw_bytes_to_v6_addr function (Yacin Nadji, Corelight) 5.0.0-dev.164 | 2022-03-08 09:30:37 -0700 * Update 3rdparty submodule for bsd-getopt-long fix (Tim Wojtulewicz) 5.0.0-dev.162 | 2022-03-07 12:36:37 +0100 * Improve error message when receiving unexpected record content via Broker. (Robin Sommer, Corelight) 5.0.0-dev.160 | 2022-03-02 13:48:07 +0000 * restored record constructor checking for missing-but-mandatory fields. This includea a new btest as well as a fix to the base-scrpts. (Vern Paxson, Corelight) 5.0.0-dev.156 | 2022-03-02 08:23:50 +0000 * The is_num(), is_alpha(), and is_alnum() BiFs now return F on empty string. The testcases for these functions, and for is_ascii() were expanded. The documentation of is_ascii() concerning behavior of an empty string was clarified (Christian Kreibich, Corelight) 5.0.0-dev.151 | 2022-03-02 08:09:28 +0000 * SSL: rudimentary decryption for TLS 1.2 (Florian Wilkens, Johanna Amann) With this version, we support rudimentary decryption of TLS 1.2 connections, if the key material of the connection (in our case the pre-master secret) is available. Note that this functionality only works for TLS 1.2 connections using the TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 cipher suite. No other combinations are currently supported. For more information, see the NEWS entry and the TLS Decryption documentation. 5.0.0-dev.121 | 2022-02-24 09:11:03 -0700 * GH-1980: Deprecate and return warning for zeek-config's caf-root option (Tim Wojtulewicz, Corelight) 5.0.0-dev.118 | 2022-02-23 10:51:57 -0700 * GH-1949: Remove unused timer_mgr_inactivity_timeout global (Tim Wojtulewicz, Corelight) 5.0.0-dev.116 | 2022-02-21 18:17:13 -0700 * remove deprecated union and timer types, addressing #1898 (Matthew Luckie) 5.0.0-dev.114 | 2022-02-11 09:30:04 -0800 * Minor modernizations to Github workflows (Christian Kreibich, Corelight) 5.0.0-dev.112 | 2022-02-10 17:56:27 -0800 * Reorg of the cluster controller to new "Management framework" layout (Christian Kreibich, Corelight) * Bump external cluster testsuite to reflect Management framework reorg (Christian Kreibich, Corelight) * Bump zeek-client to reflect Management framework reorg (Christian Kreibich, Corelight) 5.0.0-dev.108 | 2022-02-10 10:35:02 -0700 * Fixing a big pile of Coverity issues (Tim Wojtulewicz, Corelight) 5.0.0-dev.106 | 2022-02-09 15:15:21 -0800 * Expand generate-docs Github workflow to test docs build on PRs (Christian Kreibich, Corelight) 5.0.0-dev.104 | 2022-02-09 13:14:04 -0800 * Updates to the cluster controller scripts to fix the docs build (Christian Kreibich, Corelight) * Bump zeek-client for Broker enum fix/workaround (Christian Kreibich, Corelight) 5.0.0-dev.100 | 2022-02-07 14:18:50 -0800 * Add capture to a Sumstats when-statement to fix deprecation warning (Christian Kreibich, Corelight) 5.0.0-dev.97 | 2022-02-07 16:24:06 +0100 * Update to latest Broker without public CAF dependencies. (Dominik Charousset, Corelight) * Fix GCC builds and string output for Broker errors (Dominik Charousset, Corelight) 5.0.0-dev.94 | 2022-02-07 08:14:47 -0700 * String/StringVal: Replace char*/string constructors with string_view (Tim Wojtulewicz, Corelight) 5.0.0-dev.92 | 2022-02-04 10:33:47 -0700 * fix existing checks for looking to use C++ when it's not available (Vern Paxson, Corelight) 5.0.0-dev.90 | 2022-02-04 10:32:41 -0700 * fixes for ZAM profiling, which didn't get fully integrated originally (Vern Paxson, Corelight) * minor enhancements for ZAM inlining (Vern Paxson, Corelight) 5.0.0-dev.87 | 2022-02-03 13:17:25 -0800 * Expansion of cluster controller functionality (Christian Kreibich, Corelight) - Bump external cluster testsuite - Bump zeek-client for the get-nodes command - Add ClusterController::API::get_nodes_request/response event pair - Support optional listening ports for cluster nodes - Don't auto-publish Supervisor response events in the cluster agent - Make members of the ClusterController::Types::State enum all-caps - Be more conservative with triggering request timeout events - Move redefs of ClusterController::Request::Request to their places of use - Simplify ClusterController::API::set_configuration_request/response 5.0.0-dev.77 | 2022-02-03 11:20:16 +0000 * Match DPD TLS signature on one-sided connections. (Johanna Amann, Corelight) This commit changes DPD matching for TLS connections. A one-sided match is enough to enable DPD now. This commit also removes DPD for SSLv2 connections. SSLv2 connections do basically no longer happen in the wild. SSLv2 is also really finnicky to identify correctly - there is very little data required to match it, and basically all matches today will be false positives. If DPD for SSLv2 is still desired, the optional signature in policy/protocols/ssl/dpd-v2.sig can be loaded. 5.0.0-dev.74 | 2022-02-02 09:46:00 +0100 * GH-1890: Consistently warn about mixing vector and scalar operand depreciaton (Zeke Medley, Corelight) 5.0.0-dev.72 | 2022-02-02 09:36:30 +0100 * Let TCP-based application analyzers operate without any TCP parent analyzer. (Robin Sommer, Corelight) 5.0.0-dev.70 | 2022-01-25 13:52:00 -0700 * bug fix for vector slice assignment (Vern Paxson, Corelight) 5.0.0-dev.67 | 2022-01-25 12:25:48 +0000 * updated Bro->Zeek in comments in the source tree (Vern Paxson, Corelight) 5.0.0-dev.65 | 2022-01-24 13:41:25 -0800 * CI updates (Christian Kreibich, Corelight) - add Ubuntu 21.10 - remove OpenSUSE Leap 15.2 (EOL) - add CentOS Stream 9 5.0.0-dev.61 | 2022-01-17 10:35:15 +0000 * fix for adding a non-managed type to an empty vector (Vern Paxson, Corelight) 5.0.0-dev.58 | 2022-01-14 14:41:15 -0700 * explicitly provide the frame for evaluating a "when" timeout expression (Vern Paxson, Corelight) * tests for new "when" semantics/errors (Vern Paxson, Corelight) * captures for "when" statements (Vern Paxson, Corelight) update Triggers to IntrusivePtr's and simpler AST traversal introduce IDSet type, migrate associated "ID*" types to "const ID*" * logic (other than in profiling) for assignments that yield separate values (Vern Paxson, Corelight) * option for internal use to mark a function type as allowing non-expression returns (Vern Paxson, Corelight) * removed some now-obsolete profiling functionality (Vern Paxson, Corelight) 5.0.0-dev.47 | 2022-01-14 12:22:11 -0700 * Remove Fedora 33 from CI. (Johanna Amann, Corelight) EOL date was 2021‑11‑30 5.0.0-dev.45 | 2022-01-14 12:21:25 -0700 * Have `make dist` cleanup a few more wayward files before tarring (Tim Wojtulewicz, Corelight) 5.0.0-dev.43 | 2022-01-14 15:59:53 +0000 * Make more StringVal member functions const (Johanna Amann, Corelight) 5.0.0-dev.41 | 2022-01-12 10:16:38 -0700 * Fix --with-jemalloc configure option on FreeBSD (Tim Wojtulewicz, Corelight) 5.0.0-dev.39 | 2022-01-11 15:02:37 -0700 * support for compiling type-based switches to C++ (Vern Paxson, Corelight) * make encountering uncompilable "standalone" functions to be a hard error (Vern Paxson, Corelight) * fix: ZAM could misinterpret a "type" switch that starts with a "default" (Vern Paxson, Corelight) * fix: don't treat pseudo-identifiers in type cases as local variables (Vern Paxson, Corelight) * removed development helper scripts now obsolete with --optimize-files= (Vern Paxson, Corelight) 5.0.0-dev.33 | 2022-01-11 14:57:58 -0700 * Bump CI sanitizer task to Ubuntu 20.04 (Tim Wojtulewicz, Corelight) * Fix uninitialized variable warning (Tim Wojtulewicz, Corelight) * Enable null check for UBSan (Tim Wojtulewicz, Corelight) 5.0.0-dev.28 | 2022-01-10 10:30:39 -0700 * fix for avoiding de-ref of nil pointer (Vern Paxson, Corelight) 5.0.0-dev.25 | 2022-01-10 14:42:50 +0100 * GH-1844: Fix host header normalization in intel framework. (Robin Sommer, Corelight) * GH-1844: Switch to recording unmodified HTTP header. (Robin Sommer, Corelight) 5.0.0-dev.22 | 2022-01-07 12:34:25 -0700 * GH-1624: Migrate check_and_promote and a few Expr methods to IntrusivePtr (Tim Wojtulewicz, Corelight) 5.0.0-dev.19 | 2022-01-06 15:06:59 -0700 * bug fix for reporting poorly formed record constructors (Vern Paxson, Corelight) 5.0.0-dev.17 | 2022-01-06 14:04:48 -0700 * Remove CentOS 8 from CI. (Johanna Amann, Corelight) EOL was 2021-12-31 5.0.0-dev.15 | 2022-01-06 13:54:38 -0700 * test suite alternative baseline fixes for recent test renaming (Vern Paxson, Corelight) * addressed a couple of memory leaks in ZAM execution (Vern Paxson, Corelight) 5.0.0-dev.10 | 2022-01-05 09:13:45 -0700 * Make FreeBSD more flexible, don't look for jemalloc unless we were (Craig Leres) given a specific path for it. * Set JEMALLOC_FOUND when on FreeBSD (suggested by Tim Wojtulewicz) (Craig Leres) * FreeBSD: don't look for jemalloc as a package, it's in the base system (Craig Leres) 5.0.0-dev.6 | 2022-01-05 09:12:33 -0700 * Fix return type of double_to_int (Johanna Amann, Corelight) Fixes GH-1919 4.2.0-dev.514 | 2022-01-03 13:56:12 -0700 * deprecation warning on use of out-of-scope local (Vern Paxson, Corelight) 4.2.0-dev.510 | 2022-01-03 13:54:52 -0700 * Switch BitTorrent analyzer to Zeek's regex engine (Avinal Kumar) - Removes dependency on - Replaces regex function with Zeek's standard regex functions - Some replacements are workaround, may be improved later via an appropiate API - Update test baseline to fix what seems to be capturing on a bug in the existing code. Edit pass by Robin Sommer. Note that our test doesn't cover all the code paths, but it does go through the one with the most substantial change. * Adding test for BitTorrent tracker. (Robin Sommer, Corelight) Our test trace is extracted from https://www.cloudshark.org/captures/b9089aac6eee. There actually seems to be a bug in the existing code: the URI passed to bt_tracker_request() includes a partial HTTP version. This commits includes the baseline as the current code produces it, we'll fix that in a subsequent comment. 4.2.0-dev.506 | 2022-01-03 09:33:43 -0800 * Expansion of the emerging cluster controller framework (Christian Kreibich, Corelight) - Controller/agent connectivity is now controlled by pushed configurations - The Request module now supports state timeouts - Use Result records consistently for responses to the client - Track successful config deployment in cluster controller - Add ClusterController::API::notify_agents_ready event - Make all globals start with a "g_" prefix - Add missing debug() log function to log module's API - Add separate utility module for controller and agent - Additional infrastructure for printing types - Bump zeek-client to v0.2.0 - Add Github action job for cluster tests - Tweak Docker image configure invocation to include zeek-client - Zeekygen documentation pass 4.2.0-dev.477 | 2021-12-14 16:53:57 -0700 * fixes for double-delete and reducing '?' operator with constant alternatives (Vern Paxson, Corelight) * correct usage info for -u flag; -uu no longer supported (Vern Paxson, Corelight) 4.2.0-dev.468 | 2021-12-14 11:34:47 -0700 * factoring of generating C++ initializations, no semantic changes (Vern Paxson, Corelight) * restored support for incremental compilation of scripts to C++ (Vern Paxson, Corelight) * fixes for -O gen-standalone-C++ (Vern Paxson, Corelight) * new ZEEK_FILE_ONLY and ZEEK_FUNC_ONLY environment variables for debugging script optimization - replaces ZEEK_ONLY (Vern Paxson, Corelight) * fix for compiling record constructors to C++ (Vern Paxson, Corelight) * fixes for compiling vector operations to C++ (Vern Paxson, Corelight) * fixed for profiling missing some profile elements (Vern Paxson, Corelight) * minor efficiency tweak for ZAM record construction (Vern Paxson, Corelight) 4.2.0-dev.456 | 2021-12-14 09:23:47 -0700 * GH-1860: Add double_to_int() bif (Tim Wojtulewicz, Corelight) 4.2.0-dev.454 | 2021-12-13 09:41:32 -0700 * Check for sets before attempting to check for same Yield types (Tim Wojtulewicz) * Add early bail-outs to same_type() (Tim Wojtulewicz) * Fix types for Analyzer::register_for_port(s) to be the same (Tim Wojtulewicz) * Update cmake submodule across all other submodules (Tim Wojtulewicz, Corelight) 4.2.0-dev.448 | 2021-12-10 15:35:34 -0700 * update btest to no longer use (unsupported) %S formatting, no longer needed (Vern Paxson, Corelight) * replace --optimize-only with --optimize-funcs and --optimize-files (Vern Paxson, Corelight) 4.2.0-dev.444 | 2021-12-10 13:13:13 -0700 * reintroduction of "-O add-C++" option (Vern Paxson, Corelight) 4.2.0-dev.442 | 2021-12-10 13:12:43 -0700 * fixes for vector operations (Vern Paxson, Corelight) * flag globals initialized to opaque values as non-compilable (Vern Paxson, Corelight) * skip type signatures for lambdas (Vern Paxson, Corelight) * fix for translating filenames beginning with numbers to C++ variable names (Vern Paxson, Corelight) 4.2.0-dev.436 | 2021-12-10 13:11:36 -0700 * update script-to-C++ compilation for new record constructor internals (Vern Paxson, Corelight) 4.2.0-dev.434 | 2021-12-10 13:11:10 -0700 * updates to ZAM to track recent changes in script semantics (Vern Paxson, Corelight) 4.2.0-dev.432 | 2021-12-10 09:28:23 -0700 * GH-1741: Print error if calling a non-hook with hook keyword (Tim Wojtulewicz, Corelight) * GH-1740: Report a better error message if table key is not a list (Tim Wojtulewicz, Corelight) 4.2.0-dev.428 | 2021-12-09 14:58:53 -0700 * GH-1125: Support GRE ARUBA headers (Tim Wojtulewicz, Corelight) * Fix ethertype for ARP in Geneve forwarding rules (Tim Wojtulewicz, Corelight) 4.2.0-dev.425 | 2021-12-09 13:45:17 -0800 * Add LogAscii::json_include_unset_fields flag to control unset field rendering (Christian Kreibich, Corelight) 4.2.0-dev.423 | 2021-12-09 19:56:43 +0000 * Improve error message for clash between variable and function name (Johanna Amann, Corelight) Fixes GH-1832 * Restore --disable-zeekctl configure argument (Tim Wojtulewicz, Corelight) * Update plugin.hooks baseline for recent Geneve change (Tim Wojtulewicz, Corelight) 4.2.0-dev.419 | 2021-12-07 09:34:45 -0700 * GH-1764: Update mappings for Geneve analyzer to IP4/IP6/ARP (Tim Wojtulewicz, Corelight) 4.2.0-dev.417 | 2021-12-06 17:00:16 -0800 * Flip C++ unit tests to being enabled by default (Christian Kreibich, Corelight) To disable them, configure with --disable-cpp-tests. * Support for unit tests in plugins (Christian Kreibich, Corelight) 4.2.0-dev.410 | 2021-12-06 11:29:32 -0700 * Remove separate Tag types, note breaking change in NEWS (Tim Wojtulewicz, Corelight) 4.2.0-dev.408 | 2021-12-06 09:15:24 -0700 * GH-1768: Properly cleanup existing log stream when recreated on with the same ID (Tim Wojtulewicz, Corelight) 4.2.0-dev.406 | 2021-12-01 10:32:34 -0700 * Format Python scripts with yapf. (Benjamin Bannier, Corelight) We also add a very basic yapf configuration file. Most of the changes in this patch were performed automatically, but we broke one overly long string into multiple components on `src/make_dbg_constants.py`. * Format shell scripts with shfmt. (Benjamin Bannier, Corelight) All changes in this patch were performed automatically with `shfmt` with configuration flags specified in `.pre-commit-config.yaml`. 4.2.0-dev.403 | 2021-12-01 10:25:32 -0700 * fix btest comment to more accurately describe the test (Vern Paxson, Corelight) * btests for erroneous script conditionals (Vern Paxson, Corelight) * avoid compiling-to-C++ for functions potentially influenced by conditionals (Vern Paxson, Corelight) * track the use of conditionals in functions and files (Vern Paxson, Corelight) * AST profiles track the associated function/body/expression (Vern Paxson, Corelight) 4.2.0-dev.396 | 2021-12-01 09:44:03 -0700 * GH-1873: Deprecate the tag types differently to avoid type clashes (Tim Wojtulewicz, Corelight) 4.2.0-dev.394 | 2021-11-30 11:53:35 -0700 * Fix for the recent patch that allows segment offloaded packets. (Johanna Amann, Corelight) We recently added support for segment offloaded packets. It turns out that this can lead to problems in UDP/ICMP based parsers since I missed correctly also updating the payloadlength there, and using the capture length instead when segment offloading is enabled. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41391 https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41394 https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41395 (Link to details becomes public 30 days after patch release) 4.2.0-dev.393 | 2021-11-29 13:46:59 -0700 * Fix a number of Coverity findings (Tim Wojtulewicz, Corelight) 1466460: Uninitialized field in gtp-analyzer.pac 1462465: Null pointer dereference in CompositeHash::SingleValHash 1462463: Copy/paste error in TCPSessionAdapter::build_syn_packet_val 1462067: Uninitialized fields in Zinst 4.2.0-dev.391 | 2021-11-29 13:44:11 -0700 * suppress unneeded initializations (Vern Paxson, Corelight) 4.2.0-dev.387 | 2021-11-24 13:32:33 -0700 * fixes for constructing and assigning records with fields that are empty vectors (Vern Paxson, Corelight) 4.2.0-dev.385 | 2021-11-23 19:43:48 -0700 * Changes to speed up compilation of Compiled-to-C++ Zeek Scripts (Vern Paxson, Corelight) * removing unused SubNetType class (Vern Paxson, Corelight) 4.2.0-dev.371 | 2021-11-23 19:41:10 -0700 * Add new tunnel packet analyzers, remove old ones (Tim Wojtulewicz, Corelight) * Add PacketAnalyzer::register_for_port(s) functions (Tim Wojtulewicz, Corelight) These allow packet analyzers to register ports as identifiers to forward from parent analyzers, while also adding those ports to the now-global Analyzer::ports table at the same time. * Add analyzer_confirmation and analyzer_violation events (Tim Wojtulewicz, Corelight) * Add utility function for tunnel analyzers to setup encapsulation (Tim Wojtulewicz, Corelight) * Store some additional information in the packet during processing (Tim Wojtulewicz, Corelight) - Session related to the packet - is_orig information if a UDP header was found * Minor fix in UDP to avoid duplicating tunnels (Tim Wojtulewicz, Corelight) * Fix error text in IPTunnel analyzer (Tim Wojtulewicz, Corelight) * Change Packet::ip_hdr to be a shared_ptr so it can be copied into EncapsulatingConn (Tim Wojtulewicz, Corelight) * Add method for packet analyzers to register for protocol detection (Tim Wojtulewicz, Corelight) * Add concept of "parent" tag namespaces (Tim Wojtulewicz, Corelight) This allows us to create an EnumType that groups all of the analyzer tag values into a single type, while still having the existing types that split them up. We can then use this for certain events that benefit from taking all of the tag types at once. * Unify plugin::Component and plugin::TaggedComponent into a single class (Tim Wojtulewicz, Corelight) These two are almost always used in conjunction with each other, and TaggedComponent is never used by itself. Combining them together into a single class will help simplify some of the code around managing the mapping between Tags and Components. * Remove uses of deprecated Tag types (Tim Wojtulewicz, Corelight) * Unify all of the Tag types into one type (Tim Wojtulewicz, Corelight) - Remove tag types for each component type (analyzer, etc) - Add deprecated versions of the old types - Remove unnecessary tag element from templates for TaggedComponent and ComponentManager - Enable TaggedComponent to pass an EnumType when initializing Tag objects - Update some tests that are affected by the tag enum values changing order 4.2.0-dev.350 | 2021-11-23 15:35:06 +0000 * Add testcase for TCP segment offloading (GH-1829). (Johanna Amann, Corelight) 4.2.0-dev.348 | 2021-11-23 13:45:39 +0000 * OpenSSL 3 compatibility (Johanna Amann, Corelight) Zeek is now compatible with OpenSSL 3.0, our test baselines pass cleanly, and we have a CI run for OpenSSL 3.0. This has a certain amount of new code for X.509 certificate parsing. Apart from that, the main chainge is that we use an older, legacy, API for OpaqueVal hashing, since the newer API does not allow us to serialize data anymore. For details see ticket 1379. 4.2.0-dev.340 | 2021-11-23 10:10:13 +0000 * Accept packets that use tcp segment offloading. (Johanna Amann, Corelight) When checksum offloading is enabled, we now forward packets that have 0 header lengths set - and assume that they have TSO enabled. If checksum offloading is not enabled, we drop the packets (GH-1829) * Updates to NEWS to cover recent additions. [nomail] [skip ci] (Christian Kreibich, Corelight) * Update doc and auxil/zeek-aux submodules [nomail] [skip ci] (Christian Kreibich, Corelight) * Update cmake and aux/zeek-aux submodules [nomail] [skip ci] (Christian Kreibich, Corelight) 4.2.0-dev.333 | 2021-11-17 11:57:04 -0800 * Clean up fully after successful Docker btests (Christian Kreibich, Corelight) 4.2.0-dev.331 | 2021-11-15 10:10:52 -0800 * Fix ref-naming typo in the Github Docker workflow (Christian Kreibich, Corelight) 4.2.0-dev.328 | 2021-11-12 13:46:32 -0700 * Update libkqueue submodule (Tim Wojtulewicz, Corelight) 4.2.0-dev.326 | 2021-11-12 09:30:54 -0700 * Added plugin.unprocessed_packet_hook btest (Tim Wojtulewicz, Corelight) * Fix whitespace in help output (Tim Wojtulewicz, Corelight) * Add command-line option to write unprocessed packets to a file (Tim Wojtulewicz, Corelight) This commit also changes the PcapDumper to automatically flush after every called to Dump(). This is because pcap_dump has an internal buffer of some sort that only writes to the file after a set amount of bytes. When using the new option on a low-traffic network, it might be a while before you see any packets written since it has to overcome that buffer limit first. * GH-1620: Add event and plugin hook to track packets not processed (Tim Wojtulewicz, Corelight) 4.2.0-dev.319 | 2021-11-10 10:20:01 -0700 * Install include headers from `src/3rdparty/`. (Benjamin Bannier, Corelight) This is a fixup commit for 72cbc7cd13b7c1bda98658104431c3b530ff68d6 where we move some header files from `src/` to `src/3rdparty/` but missed adding install rules for these header. Since some of these headers are exposed in installed headers they need to be installed as well. 4.2.0-dev.317 | 2021-11-10 11:33:29 +0000 * Add case-insensitive search for find_str and rfind_str (Abdel) 4.2.0-dev.314 | 2021-11-10 11:16:28 +0100 * GH-1757: Add new hook `HookLoadFileExtended` that allows plugins to supply Zeek script and signature code to parse. (Robin Sommer) The new hook works similar to the existing `HookLoadFile` but, additionally, allows the plugin to return a string that contains the code to be used for the file being loaded. If the plugin does so, the content of any actual file on disk will be ignored. This works for both Zeek scripts and signatures. * Fix an issue where signature files supplied on the command line wouldn't pass through the file loading hooks. (Robin Sommer, Corelight) 4.2.0-dev.310 | 2021-11-09 10:29:59 -0700 * Add Github action exercising pre-commit (Benjamin Bannier, Corelight) This patch adds a Github action which exercises pre-commit linters for commits to the `master` branch or for pull requests. We adds this task as a Github action since we expect it to finish quickly; running outside of Cirrus makes it possible provide feedback quickly. * Add pre-commit config. (Benjamin Bannier, Corelight) This patch adds `clang-format` as only linter for now. This replaces the previously used script from `auxil/run-clang-format` which we remove. This requires the Python program `pre-commit` (https://pypi.org/project/pre-commit/). With that one can then run `clang-format` on the whole codebase with $ pre-commit run -a clang-format or on just the staged files # Explicitly selecting linter. $ pre-commit run clang-format # Run all linters (currently just `clang-format`). $ pre-commit `pre-commit` supports managing Git commit hooks so that linters are run on commit. Linters can be installed with $ pre-commit install The documentation at https://pre-commit.com/ covers these topics in addition to more information. * Format code with `clang-format` (Benjamin Bannier, Corelight) This patch formats files not conforming to the C++ formatting with `clang-format`. * Remove stale files `src/DebugCmdInfoConstants.*` (Benjamin Bannier, Corelight) The files generated from `src/DebugCmdInfoConstants.in` are placed in `build/src/` by the build setup, and generated file in `src/` removed here were unused and possibly out-of-date. * Disable formatting for files in `testing/btest/plugins` (Benjamin Bannier, Corelight) Files in that folder were previously not formatted. With this patch we now disable formatting in that folder explicitly by adding a dedicated `clang-format` config which deactivates any formatting changes. * Move 3rdparty source files to `3rdparty/` (Benjamin Bannier, Corelight) This patch moves in-tree 3rdparty source files to `3rdparty/`. With that we can remove special treatment of these files for `run-clang-format`. 4.2.0-dev.303 | 2021-11-09 09:45:57 -0700 * GH-1819: Handle recursive types when describing type in binary mode (Tim Wojtulewicz, Corelight) 4.2.0-dev.301 | 2021-11-09 09:28:18 -0700 * Remove no-op false-teredo test (Tim Wojtulewicz, Corelight) 4.2.0-dev.297 | 2021-11-05 12:49:55 -0700 * Only push CI's Docker images when we're on the main repo (Christian Kreibich, Corelight) * Add macOS Monterey and drop Catalina in CI (Christian Kreibich, Corelight) * Add Fedora 35 to CI (Christian Kreibich, Corelight) 4.2.0-dev.292 | 2021-11-04 14:28:35 -0700 * Fix C++ set intersection code (Yacin Nadji, Corelight) 4.2.0-dev.286 | 2021-11-03 09:36:41 -0700 * GH-693: use pcap_dump_open_append where supported (Tim Wojtulewicz, Corelight) 4.2.0-dev.284 | 2021-11-03 09:35:10 -0700 * GH-1781: Add .git-blame-ignore-revs file (Tim Wojtulewicz, Corelight) 4.2.0-dev.280 | 2021-11-01 09:20:16 -0700 * Fix issue with broken libpcaps that return repeat packets (Tim Wojtulewicz, Corelight) This is apparently a problem with the Myricom version of libpcap, where instead of returning a null or a zero if no packets are available, it returns the previous packet. This causes Zeek to improperly parse the packet and crash. We thought we had fixed this previously with a check for a null packet but that fix was not enough. 4.2.0-dev.277 | 2021-10-21 17:23:46 -0700 * Apply some missing clang-format changes (Tim Wojtulewicz, Corelight) 4.2.0-dev.274 | 2021-10-20 11:13:16 -0700 * Remove trailing whitespace from script files (Tim Wojtulewicz, Corelight) 4.2.0-dev.271 | 2021-10-19 14:54:56 +0200 * Add parsing of DNS SVCB/HTTPS records (FlyingWithJerome) 4.2.0-dev.260 | 2021-10-15 09:45:45 +0100 * logging/writers/ascii: shadow files: Add fsync() before rename(). This prevents potential problems with leftover files after unclean shutdowns. (Arne Welzel, Corelight) * Fix typo in typedef changes that broke tests on 32-bit Debian 9 (Tim Wojtulewicz, Corelight) 4.2.0-dev.255 | 2021-10-12 09:22:37 -0700 * Replace most uses of typedef with using for type aliasing (Tim Wojtulewicz, Corelight) 4.2.0-dev.253 | 2021-10-11 11:38:30 -0700 * Don't initialize private testsuite in CI when not running out of Zeek project (Christian Kreibich, Corelight) * Drop FreeBSD 11.4 (now EOL), add FreeBSD 14 as informational build (Christian Kreibich, Corelight) 4.2.0-dev.249 | 2021-10-04 12:47:42 -0700 * Expand table/set tests to cover new index types (Christian Kreibich, Corelight) * Expression tweaks to get complex types to work as container indices (Christian Kreibich, Corelight) 4.2.0-dev.245 | 2021-10-01 14:19:19 -0700 * Enable customization of the Docker image's configure invocation (Christian Kreibich, Corelight) * Add Python Broker module load test to Docker image btests (Christian Kreibich, Corelight) * Set PYTHONPATH in Docker to automatically find Zeek's modules (Christian Kreibich, Corelight) * Docker image fix: libpython needs to be 3.9 on Debian Bullseye (Christian Kreibich, Corelight) 4.2.0-dev.240 | 2021-10-01 13:28:39 -0700 * Bump highwayhash submodule to pull in another FreeBSD header fix (Christian Kreibich, Corelight) * CI tweak: upgrade curl on FreeBSD to avoid cert expiration (Christian Kreibich, Corelight) * CI tweak: add a datestamp to Debian 9 Dockerfiles to invalidate Cirrus build cache (Christian Kreibich, Corelight) 4.2.0-dev.235 | 2021-09-28 10:04:46 -0700 * GHI-1766: Remove address from Site::private_address_space that converts into 0.0.0.0/0 (Tim Wojtulewicz, Corelight) 4.2.0-dev.233 | 2021-09-27 12:06:27 -0700 * Fix a number of issues with the initial pass of clang-format (Tim Wojtulewicz, Corelight) 4.2.0-dev.224 | 2021-09-26 10:27:05 -0700 * Fix space issue in docker tag computation (Benjamin Bannier, Corelight) 4.2.0-dev.222 | 2021-09-25 11:47:46 -0700 * Fixing homebrew, again. (Tim Wojtulewicz, Corelight) Homebrew apparently requires you to now specify the version of the package you're updating. We previously could just run `brew update openssl` and it would work. Now we must run `brew update openssl@1.1` for the command to succeed. 4.2.0-dev.220 | 2021-09-24 20:49:08 -0700 * Add .dockerignore to suppress btest artifacts (Christian Kreibich, Corelight) We run btest in CI between image build and upload, and since its artifacts weren't suppressed, the discrepancy caused the Dockerfile COPY to cache-miss. 4.2.0-dev.218 | 2021-09-24 11:10:30 -0700 * Add a minimal containerized Docker environment (Benjamin Bannier, Corelight) This patch adds a minimal Zeek environment packaged as a container. Since this is intended both as a base layer for other images and as a quick way to explore Zeek we install only zeek and zkg as basic functionality. 4.2.0-dev.214 | 2021-09-24 10:31:34 -0700 * script simplification that removes an unnecessary &is_assigned (Vern Paxson, Corelight) * removing -uu functionality and associated script analysis now no longer needed (Vern Paxson, Corelight) 4.2.0-dev.208 | 2021-09-23 17:48:13 +0200 * Fix and extend protocol forwarding/logging in dce_rpc-auth. (FOX-DS) 4.2.0-dev.205 | 2021-09-23 12:24:06 +0200 * Avoid allocation of duplicate zero-length strings for new connections. (Justin Azoff, Corelight) * Sanity-check the method passed into ActiveHTTP. Reported by Pierre Gaulon. (Robin Sommer, Corelight) 4.2.0-dev.203 | 2021-09-21 15:16:49 -0700 * Sanitize log files names before they go into system(). (Robin Sommer, Corelight) In principle, an attacker who's controlling the Zeek scripts being loaded could have set log paths to include non-safe characters leading to arbitrary command execution during log rotation. This fix avoids that be sanitizing the file names / command lines. Note, though, that this isn't a problem that we can really solve: somebody controlling scripts can just as well inject custom `system()` calls to begin with. Closes #54. 4.2.0-dev.201 | 2021-09-21 15:15:57 -0700 * PIA - switch size to int64_t (Johanna Amann, Corelight) This brings the PIA size counter in line with the actual datatype used on the scripting layer - both now use an int64_t. * Introduce dpd_max_packets (Johanna Amann, Corelight) dpd_max_packets is an additional setting that limits the maximum amount of packets that dpd will cache; before dpd was only limited by buffer size (but could cache a limitless amount of data-less packets). 4.2.0-dev.198 | 2021-09-21 14:13:21 -0700 * Use `brew update` instead of `brew update-reset` in CI macOS prepare script (Tim Wojtulewicz, Corelight) Using `brew update-reset` causes homebrew to reset to homebrew's HEAD commit, which may be buggy and broken. It appears whatever Cirrus was doing previously on their Catalina VM is no longer a problem, and so update-reset isn't required anymore. Switch to `brew update` to make sure we still get newer versions of the packages, but is actually a versioned release of homebrew. 4.2.0-dev.196 | 2021-09-21 09:44:26 -0700 * Disable script profiling/coverage on sanitizer builds (Tim Wojtulewicz, Corelight) 4.2.0-dev.194 | 2021-09-21 17:56:14 +0200 * Optimize software framework version parsing by adding a small cache in front of the parse method, and moving the parsing itself to the proxies where the caching can be more efficient. (Justin Azoff, Corelight) * Add further mappings for MS-OAUT IDispatch methods. (FOX-DS) 4.2.0-dev.189 | 2021-09-21 07:45:11 -0700 * Add btests for new functionality (Christian Kreibich, Corelight) - Expand language.set to cover sets of sets - Expand language.table to cover tables indexed with tables - Add language.table-nested-set-ordering to capture the reproducer from GHI-1753 * Remove unused HashKey constructor and reorder for consistency (Christian Kreibich, Corelight) One of the HashKey constructors was only used in the old CompHash code. This aso reorders some constructors and the destructor for readability. * Refactor CompHash class to use new HashKey buffering features (Christian Kreibich, Corelight) This preserves the previous hash key buffer layout (so the testsuite still passes) and overall approach but gets rid of the codepath for writing singleton serializations. This code path required a fourth switch block over all types (besides reads, writes, and size computation) and was inconsistent with the one for writing non-atomic types. * Add debug string and ODesc support to HashKey class (Christian Kreibich, Corelight) This allows tracing of hash key buffer reservations, reads, and writes via a new debug stream, and supports printing a summary of a HashKey object via Describe(). The latter comes in handy e.g. in TableVal::Describe() (where including the hash key is now available but commented out). * Refactor HashKey class to support read/write operations (Christian Kreibich, Corelight) This preserves the optimization of storing values directly in the key_u member union when feasible, and using a variable size buffer otherwise. It also adds bounds-checking for that buffer, moves size arguments to size_t, decouples construction from hash computation, emulates the tagging feature found in SerializationFormat to assist troubleshooting, and switches feasible reinterpret_casts to static_casts. * Add unit tests for memory helpers (Christian Kreibich, Corelight) * Add memory sizing/alignment helpers to util.cc/h (Christian Kreibich, Corelight) This functionality previously lived in the CompHash class, with one difference: this removes a discrepancy between the offset aligner and the memory pointer aligner/padder. The size aligner used to align the provided offset and then add an additional alignment size (for example, 1 aligned to 4 wouldn't yield 4 but 8). Like the memory aligners it now only rounds up as needed. Includes unit tests. * Ensure table/set HashKey buffer reservation and writes happen in same order (Christian Kreibich, Corelight) This takes the existing sorting for table index hashkeys we had in place during hash key writes and applies it also during buffer size reservation. It changes the approach slightly: the underlying map now points to the TableVal entry index vals directly, rather than to the numerical index into an additional list that gets built up to store those indexes. Doing so removes the need for that list. 4.2.0-dev.179 | 2021-09-20 11:20:50 +0200 * Add TSval and TSecr to "SYN_packet" record. (gpotter2) 4.2.0-dev.174 | 2021-09-16 15:36:52 -0700 * Minor fixes for build problems after reformatting (Tim Wojtulewicz, Corelight) * Reformat the world (Tim Wojtulewicz, Corelight) * Fixes for includes ahead of reformatting (Tim Wojtulewicz, Corelight) * GH-386: Add clang-format config and scripts (Tim Wojtulewicz, Corelight) 4.2.0-dev.169 | 2021-09-16 11:15:14 +0200 * Added four new opnum mappings for MS-OAUT IDispatch methods. (FOX-DS) * Reformat docs in addrs.zeek to fix doc generation. (Tim Wojtulewicz, Corelight) 4.2.0-dev.165 | 2021-09-14 18:39:34 -0700 * Addressed a number of low-level issues found by Coverity and compiler warnings (Vern Paxson, Corelight) * Fixed https://github.com/zeek/zeek/issues/1776 (Vern Paxson, Corelight) 4.2.0-dev.159 | 2021-09-14 13:00:50 -0700 * ignore_checksums_nets: Add test for multiple subnets (Arne Welzel, Corelight) This test fails with Zeek 4.1 and Zeek 4.0.3. 4.2.0-dev.157 | 2021-09-08 12:04:45 -0700 * fix for looping over vectors with holes per https://github.com/zeek/zeek/issues/1763 (Vern Paxson, Corelight) * btest for https://github.com/zeek/zeek/issues/1763 (Vern Paxson, Corelight) 4.2.0-dev.154 | 2021-09-08 12:03:51 -0700 * Fix mis-usage of string::append that leads to an overflow (Tim Wojtulewicz, Corelight) * Use json_escape_utf8 for all utf8 data in ODesc (Tim Wojtulewicz, Corelight) * Switch the TSV Zeek logs to be UTF8 by default. (Seth Hall, Corelight) There is a paired zeek-testing branch for some updates there. 4.2.0-dev.150 | 2021-09-08 11:44:15 -0700 * Fixes and btests for vector coercion overflows, typing, and holes (Vern Paxson, Corelight) * Improve -u reporting, distinguishing between maybe-vs-definite problems (Vern Paxson, Corelight) * "-a ZAM" testing baseline alternative (Vern Paxson, Corelight) * The main ZAM code, now available via -O ZAM (Vern Paxson, Corelight) * Rewrote tracking of variable definition ranges to be much more performant (Vern Paxson, Corelight) * Extend ID/Expr/Stmt classes to have associated optimization information (Vern Paxson, Corelight) 4.2.0-dev.112 | 2021-09-03 18:12:12 +0000 * Add btests for DNS WKS and BINDS (Vlad Grigorescu) * Add btest for DNS WKS RR. (Vlad Grigorescu) * Add btest for DNS NSEC3PARAM RR. (Vlad Grigorescu) 4.2.0-dev.106 | 2021-09-03 18:10:31 +0000 * Code modernization: use ranged-based for loop where possible (Vlad Grigorescu) * Improve performance of asn1_oid_to_val string conversions. (Vlad Grigorescu) 4.2.0-dev.103 | 2021-09-03 18:08:57 +0000 * Disable the scripts.base.frameworks.logging.sqlite.simultaneous-writes test under TSan (Tim Wojtulewicz, Corelight) Due to a bug (or intentional code) in SQLite, we disabled enabling the shared cache in sqlite3 if running under ThreadSanitizer (see cf1fefbe0b0a6163b389cc92b5a6878c7fc95f1f). Unfortunately, this has the side-effect of breaking the simultaneous-writes test because the shared cache is disabled. This is hopefully a temporary fix until SQLite fixes the issue on their side. * Mark MsgThread::cnt_sent_{in,out} as atomic to avoid a data race (Tim Wojtulewicz, Corelight) * Disable call to sqlite3_enable_shared_cache under ThreadSanitizer (Tim Wojtulewicz, Corelight) See https://sqlite.org/forum/forumpost/54424d80ee for details. 4.2.0-dev.99 | 2021-09-03 17:36:09 +0000 * GH-1589: Avoid extracting IP-like strings from SMTP headers (Tim Wojtulewicz, Corelight) * Minor updates to the external-testsuite scripts (Christian Kreibich, Corelight) Delete the unused Baseline folder, remove a Bro-era name from gitignore, change create-new-repo to work with a second argument that is actually a URL, and tweak whitespace. Expand description in README and update the explanation of OPENSSL_ENABLE_MD5_VERIFY in subdir-btest.cfg. 4.2.0-dev.94 | 2021-08-31 15:58:00 +0200 * GH-1709: Fix signed integer overflow in PIA on big sequence number holes. (Johanna Amann, Corelight) 4.2.0-dev.92 | 2021-08-31 15:55:29 +0200 * Allow `-B` flag in non-debug builds, but ignore it (unless it's `help`). (Benjamin Bannier, Corelight) 4.2.0-dev.89 | 2021-08-26 14:35:28 -0700 * CI support refresh. (Christian Kreibich, Corelight) - Add Debian 11 (Bullseye) - Drop Ubuntu 16.04 * Remove unneccessary >= 0 check in a UTF32 comparison Resolves Coverity CID 1461523. (Christian Kreibich, Corelight) * Trivial signedness warning fix. (Christian Kreibich, Corelight) * Fix addr/string type confusion in Broker::peers(). (Christian Kreibich, Corelight) * Simplify the supervisor's listen() on default address/port. (Christian Kreibich, Corelight) 4.2.0-dev.78 | 2021-08-19 09:39:23 -0700 * Return fully-escaped string if utf8 conversion fails (Tim Wojtulewicz, Corelight) This adds a new function for validating UTF-8 sequences by converting to UTF-32. This allows us to also check for various blocks of codepointsi that we consider invalid while checking for valid sequences in general. 4.2.0-dev.76 | 2021-08-18 08:40:41 -0700 * Fix option length computation in Geneve analyzer. (Benjamin Bannier, Corelight) We previously computed the length of the Geneve options field incorrectly which lead to us passing data at an incorrect offset to inner analyzers. With this patch we now interpret the length field correctly, according the the spec https://datatracker.ietf.org/doc/html/rfc8926#section-3.4. Closes #1726. 4.2.0-dev.73 | 2021-08-12 09:57:59 -0700 * GH-1713: Avoid unneeded reallocs in SerializationFormat::WriteData (Tim Wojtulewicz, Corelight) 4.2.0-dev.70 | 2021-08-06 13:29:30 -0700 * Do not lookup ignore_checksums_nets for every packet (Johanna Amann, Corelight) This could lead to a noticeable (single-percent) performance improvement. Most of the functionality for this is in the packet analyzers that now cache ignore_chesksums_nets. Based on a patch by Arne Welzel (Corelight). 4.2.0-dev.68 | 2021-08-04 09:57:32 +0100 * Use unordered_map to store sessions for performance reasons. This might lead to an 8-9% speedup of Zeek. See GH-1706 for details. (Tim Wojtulewicz, Corelight) 4.2.0-dev.64 | 2021-08-03 10:23:41 +0100 * Under certain circumstances, Zeek processes could get into an infinite looping state inside RotationTimer. This is fixed by handling special cases of rotation happening exactly on the time boundary, and fixing a special case of timer expiration. Fixes GH-1689. (Sowmya Ramapatruni, Corelight) 4.2.0-dev.62 | 2021-08-03 10:21:18 +0100 * Fix some HTTP evasions. Now HTTP packets are correctly parsed, when CRLF is missing on a multipart boundary / at packet boundaries. Fixes GH-1598 (jerome Grandvalet) 4.2.0-dev.59 | 2021-08-02 11:10:04 -0700 * Fix failing test due to patricia change (Tim Wojtulewicz, Corelight) * Rebase patricia code on upstream version (Tim Wojtulewicz, Corelight) 4.2.0-dev.56 | 2021-07-30 08:52:49 -0700 * GH-1654: Exclude the .git directory when installing cmake files (Tim Wojtulewicz, Corelight) 4.2.0-dev.54 | 2021-07-29 13:23:51 -0700 * GH-1692: Add some safety to ASN's binary_to_int64 to avoid bad shifts (Tim Wojtulewicz, Corelight) 4.2.0-dev.50 | 2021-07-27 09:36:13 -0700 * Fix a use-after-free during shutdown (Tim Wojtulewicz, Corelight) 4.2.0-dev.48 | 2021-07-26 13:03:01 -0700 * GH-1693: Fix potential crash with elements being modified during robust iteration (Tim Wojtulewicz, Corelight) * Update HMAC key used for benchmarking service (Tim Wojtulewicz, Corelight) 4.2.0-dev.45 | 2021-07-23 09:28:49 -0700 * GH-1684: Ensure that the time gets updated every pass if we're reading live traffic (Tim Wojtulewicz, Corelight) This is necessary for e.g. packet sources that don't have a selectable file descriptor. They'll always be ready on a very short timeout, but won't necessarily have a packet to process. In these case, sometimes the time won't get updated for a long time and timers don't function correctly. 4.2.0-dev.43 | 2021-07-21 11:41:19 -0700 * Fix handling of timers when cloning TableVals (Johanna Amann, Corelight) When cloning TableVals, a new timer was created for the wrong object (the existing TableVal, not the clone). This lead to the already existing timer being no longer accessible. Which, in turn, leads to an abandoned timer reading into no longer allocated data when the original TableVal is deleted. Fixes GH-1687 4.2.0-dev.40 | 2021-07-20 09:58:14 -0700 * Cover in NEWS the inclusion of btest tooling in the installation (Christian Kreibich, Corelight) [skip ci] 4.2.0-dev.38 | 2021-07-20 09:55:59 -0700 * Fix generate-docs github action to send email when it fails (Tim Wojtulewicz, Corelight) * Use Cirrus's new greedy mode for parallelizing builds and tests (Christian Kreibich, Corelight) This oversubscribes our cores 2x, which testing shows we actually run with at times: speedup is around a third on average for builds, and a bit more than that for testing. Also some light Bashification in ci/build.sh, for consistency. 4.2.0-dev.34 | 2021-07-19 08:56:45 -0700 * Bump highwayhash to pull in FreeBSD 14 fix (Christian Kreibich, Corelight) 4.2.0-dev.32 | 2021-07-19 08:55:06 -0700 * Update 3rdparty submodule to pull in doctest to 2.4.6. (Robin Sommer, Corelight) Our old version didn't support compilation on Apple's M1 yet. 4.2.0-dev.30 | 2021-07-19 08:54:30 -0700 * Fix registration of protocol analyzers from inside plugins. (Robin Sommer, Corelight) With the recent packet manager work, it broke to register a protocol analyzer for a specific port from inside a plugin's initialization code. That's because that registration now depends on the packet manager being set up, which isn't case at that time a plugin's `InitPostInit()` runs. This fix contains two parts: - Initialize the packet manager before the analyzer manager, so that the latter's `InitPostScript()` can rely on the former being ready. - Change the analyzer manager to (only) record port registrations happening before it's fully initialized. Its `InitPostScript()` then performs the actual registrations, knowing it can use the packet manager now. This comes with a `cmake/` to add a missing include directory. 4.2.0-dev.28 | 2021-07-16 22:43:35 -0700 * Update broker submodule for new CAF version (Tim Wojtulewicz, Corelight) * Setup generate-docs workflow to run on push to test branch (Tim Wojtulewicz, Corelight) 4.2.0-dev.24 | 2021-07-15 11:33:31 -0700 * Add a TODO to return a correct status for ParseRR_WKS (Vlad Grigorescu) * Update scripts/site/local-compat test for 4.2 cycle (Tim Wojtulewicz) * Remove tests for deprecated operators. (Robin Sommer, Corelight) * Add tests for operations on vectors of string. (Robin Sommer, Corelight) 4.2.0-dev.18 | 2021-07-15 09:50:25 -0700 * Remove tests for deprecated operators. (Robin Sommer, Corelight) * Add tests for operations on vectors of string. (Robin Sommer, Corelight) 4.2.0-dev.14 | 2021-07-14 15:01:20 -0700 * Fix race conditions in scripts.base.frameworks.input.reread (Christian Kreibich, Corelight) On FreeBSD, this test showed two problems: (1) reordering problems based on writing the predicate, event, and end-of-data updates into a single file, (2) a race condition based on printing the entirety of the table description argument in update events. The description contains the destination table, and its content at the time an update event gets processed isn't deterministic: depending on the number of updates the reader thread has sent, the table will contain a varying number of entries. * Fix race condition in scripts.base.frameworks.input.invalid* (Christian Kreibich, Corelight) The invalidset and invalidtext tests loaded an input file via table and event reads, in parallel. On FreeBSD this triggers an occasional reordering of messages coming out of the reader thread vs the input managers. This commit makes the table and event reads sequential, avoiding the race. 4.2.0-dev.11 | 2021-07-14 15:00:11 -0700 * Reorder fields in some classes for more compact memory layout (Tim Wojtulewicz) * Redo how reassembled flag is accessed in IP_Hdr, filling in a memory hole (Tim Wojtulewicz) 4.2.0-dev.7 | 2021-07-13 12:23:14 -0700 * Zeekygen doesn't support comments on set members (Vlad Grigorescu) * Add 6to4 addresses, if the matching IPv4 address is private (Vlad Grigorescu) * Update Site::private_address_space to latest IANA networks (Vlad Grigorescu) 4.1.0-dev.921 | 2021-07-09 12:01:40 -0700 * docs: escape * in documentation (Vlad Grigorescu) * Update the mail_dest documentation with the new email_dest field (Vlad Grigorescu) 4.1.0-dev.918 | 2021-07-08 18:00:26 -0700 * Fix a compile warning on FreeBSD (Tim Wojtulewicz, Corelight) 4.1.0-dev.916 | 2021-07-08 16:49:55 -0700 * Add a cluster controller testcase for agent-controller checkin (Christian Kreibich, Corelight) This verifies that in a setup with the supervisor creating both controller and agent, the agent successfully checks in with the controller. * Add zeek-client via new submodule (Christian Kreibich, Corelight) The new module resides in auxil/zeek-client. It does not get installed unless one configures with --enable-zeek-client. * Introduce cluster controller and cluster agent scripting (Christian Kreibich, Corelight) This is a preliminary implementation of a subset of the functionality set out in our cluster controller architecture. The controller is the central management node, existing once in any Zeek cluster. The agent is a node that runs once per instance, where an instance will commonly be a physical machine. The agent in turn manages the "data cluster", i.e. the traditional notion of a Zeek cluster with manager, worker nodes, etc. Agent and controller live in the policy folder, and are activated when loading policy/frameworks/cluster/agent and policy/frameworks/cluster/controller, respectively. Both run in nodes forked by the supervisor. When Zeek doesn't use the supervisor, they do nothing. Otherwise, boot.zeek instructs the supervisor to create the respective node, running main.zeek. Both controller and agent have their own config.zeek with relevant knobs. For both, controller/types.zeek provides common data types, and controller/log.zeek provides basic logging (without logger communication -- no such node might exist). A primitive request-tracking abstraction can be found in controller/request.zeek to track outstanding request events and their subsequent responses. * Establish a separate init script when using the supervisor (Christian Kreibich, Corelight) The supervisor does not require the full weight of scripts that init-default.zeek brings with it. The new file, init-supervisor.zeek, contains only what's required by the supervisor in addition to the other always-loaded init files. * Add optional bare-mode boolean flag to Supervisor's node configuration (Christian Kreibich, Corelight) When omitted, the node inherits the Supervisor's bare-mode status. When true/false, the new Zeek node will enable/disable bare mode, respectively. It continues to load any scripts passed at the command line and in the additional scripts list already provided in the node configuration. Includes testcase. * Add support for making the supervisor listen for requests (Christian Kreibich, Corelight) The supervisor now starts listening on the configured Broker default address and port when the new boolean SupervisorControl::enable_listen is T. Listening remains disabled by default. Listening allows nodes to communicate with the supervisor via the events laid out in control.zeek, to conduct further node management. * Add support for setting environment variables via supervisor (Christian Kreibich, Corelight) The NodeConfig record now has a table for specifying environment variable names and values, which the supervisor sets in the created node. This also repositions the cpu_affinity member to keep the order the same in the corresponding script-layer and in-core types. Includes testcase. 4.1.0-dev.907 | 2021-07-08 16:00:06 -0700 * Fix reference in the logging framework docs re. Log::StreamPolicyHook (Christian Kreibich, Corelight) 4.1.0-dev.905 | 2021-07-08 12:59:01 -0700 * Update broker submodule for process metrics (Tim Wojtulewicz, Corelight) This also updates CAF to 0.18.4 4.1.0-dev.903 | 2021-07-07 10:10:26 -0700 * GH-1644: Pass a larger structure to be calloc'd in patricia.c (Tim Wojtulewicz, Corelight) * Fix multiple spelling errors in debug output (Tim Wojtulewicz, Corelight) 4.1.0-dev.900 | 2021-07-06 08:58:48 -0700 * Removing a fix from another branch (Seth Hall, Corelight) * Update binpac pointer (Seth Hall, Corelight) * Update bifcl and binpac for building Zeek as a submodule (Seth Hall, Corelight) * Updated the cmake pointer (Seth Hall, Corelight) * Fix the builtin plugin preload mechanism (Seth Hall, Corelight) * Update submodule pointer for cmake to point at the appropriate branch (Seth Hall, Corelight) * Switch to more accurate source and binary references in cmake (Seth Hall, Corelight) This is mostly involved with moving CMAKE_SOURCE_DIR and CMAKE_BINARY_DIR to PROJECT_SOURCE_DIR and PROJECT_BINARY_DIR because it enables zeek to be built as a subproject of another. There is another branch in the cmake repository with the same name that is also required for this to fully work. 4.1.0-dev.891 | 2021-07-06 08:49:50 -0700 * Fix a number of Coverity findings (Tim Wojtulewicz, Corelight) - 1458048: Use-after-free in the SQLite logger - 1457823: Missing a break statement in script-opt reduction - 1453966: Dead code in CompHash - 1445417: Unintialized variable in StaticHash64 - 1437716: Unintialized variables in FileInfo in scan.l 4.1.0-dev.889 | 2021-07-06 08:48:50 -0700 * Add a few DNS query types. (Johanna Amann, Corelight) 4.1.0-dev.887 | 2021-07-06 08:47:22 +0200 * Add some more items to NEWS. (Robin Sommer, Corelight) 4.1.0-dev.884 | 2021-07-05 10:06:59 +0100 * Add ability to check if hostname is valid for a specific cert (Johanna Amann, Corelight) This commit adds two new bifs, x509_check_hostname and x509_check_cert_hostname. These bifs can be used to check if a given hostname which can, e.g., be sent in a SNI is valid for a specific certificate. This PR furthermore modifies the ssl logs again, and adds information about this to the log-file. Furthermore we now by default remove the server certificate information from ssl.log - I doubt that this is often looked at, it is not present in TLS 1.3, we do still have the SNI, and if you need it you have the information in x509.log. This also fixes a small potential problem in X509.cc assuming there might be SAN-entries that contain null-bytes. * Add ssl_history field to ssl.log (Johanna Amann) This is the equivalent to a connection history for SSL - and contains information about which protocol messages were exchanged in which order. * Add policy script suppressing certificate events (Johanna Amann, Corelight) The added disable-certificate-events-known-certs.zeek disables repeated X509 events in SSL connections, given that the connection terminates at the same server and used the samt SNI as a previously seen connection with the same certificate. For people that see significant amounts of TLS 1.2 traffic, this could reduce the amount of raised events significantly - especially when a lot of connections are repeat connections to the same servers. The practical impact of not raising these events is actually very little - unless a script directly interacts with the x509 events, everything works as before - the x509 variables in the connection records are still being set (from the cache). * Add new ssl-log-ext policy script (Johanna Amann, Corelight) This policy script significantly extends the details that are logged about SSL/TLS handshakes. * Deprecate extract-certs-pem.zeek and add log-certs-base64.zeek (Johanna Amann, Corelight) Extract-certs-pem writes pem files to a dedicated file; since it does not really work in cluster-environments it was never super helpful. This commit deprecates this file and, instead, adds log-certs-base64.zeek, which adds the base64-encoded certificate (which is basically equivalent with a PEM) to the log-file. Since, nowadays, the log-files are deduplicates this should not add a huge overhead. * Implement X509 certificate log caching (Johanna Amann, Corelight) By default, each certificate is now output only once per hour. This also should work in cluster mode, where we use the net broker-table-syncing feature to distribute the information about already seen certificates across the entire cluster. Log caching is also pretty configureable and can be changed using a range of confiuration options and hooks. Note that this is currently completely separate from X509 events caching, which prevents duplicate parsing of X509 certificates. * Deprecate ICSI SSL notary script. (Johanna Amann, Corelight) The ICSI notary is pretty much inactive. Furthermore - this approach does no longer make much sense at this point of time - performing, e.g., signed certificate timestamp validation is much more worthwhile. * Change SSL and X.509 logging format (Johanna Amann, Corelight) This commit changes the SSL and X.509 logging formats. Logs are now indexed by their hash and no longer by the file ID. This commit introduces two new options, which determine if certificate issuers and subjects are still logged in ssl.log. The default is to have the host subject/issuer logged, but to remove client-certificate information. Client-certificates are not a typically used feature nowadays. * Enable OCSP logging by default. (Johanna Amann, Corelight) It turns out that this can actually contain a slew of interesting information - like operating systems querying for the revocation of software signing certificates, e.g. * Split the code that handles X509 event hashing into its own file (Johanna Amann, Corelight) This also improves documentation of this feature. 4.1.0-dev.864 | 2021-07-02 12:50:40 -0700 * Add a global log policy hook to the logging framework (Christian Kreibich, Corelight) This addresses the need for a central hook on any log write, which wasn't previously doable without a lot of effort. The log manager invokes the new Log::log_stream_policy hook prior to any filter-specific hooks. Like filter-level hooks, it may veto a log write. Even when it does, filter-level hooks still get invoked, but cannot "un-veto". Includes test cases. 4.1.0-dev.861 | 2021-07-02 11:58:01 -0700 * Use mallinfo2() instead of mallinfo() when available (Christian Kreibich, Corelight) glibc 2.33 deprecates mallinfo in favor of a struct that returns its members as size_ts instead of ints. 4.1.0-dev.859 | 2021-07-02 10:16:25 -0700 * Update Broker submodule (Dominik Charousset, Corelight) 4.1.0-dev.856 | 2021-07-02 13:01:05 +0100 * Update Mozilla CA list and CT anchors. (Johanna Amann, Corelight) This commit switches to only allowing the CT logs that are currently accepted by Google Chrome - which makes much more sense for us since this is (potentially) used for validation. Additional CT logs can be added in user-scripts. * Minor NEWS formatting fixes (Tim Wojtulewicz, Corelight) 4.1.0-dev.852 | 2021-07-01 08:46:41 -0700 * low-level coding style fixes (Vern Paxson, Corelight) * support for standalone compiled scripts to export globals with module qualifiers (Vern Paxson, Corelight) * updates for documentation of functionality for compiling scripts to C++ (Vern Paxson, Corelight) * fixes for standalone C++ scripts making types & variables/functions available (Vern Paxson, Corelight) * fixed bug limiting availability of load_CPP() BiF (Vern Paxson, Corelight) * updates to development helper scripts to support new workflow (Vern Paxson, Corelight) * simpler workflow for -O gen-C++ ; also some hooks for -O gen-standalone-C++ (Vern Paxson, Corelight) * ReplaceBody now deletes a body if the replacement is nil (Vern Paxson, Corelight) * removal of can't-actually-be-executed code (Vern Paxson, Corelight) 4.1.0-dev.842 | 2021-06-30 20:32:37 -0700 * Skip input framework entries with missing but non-optional fields (Christian Kreibich, Corelight) The framework so far populated data structures with missing fields even when those fields are defined without the &optional attribute. When using the attribute, such entries continue to get populated. Update tests to reflect focus on unset fields. * Fix segfault in input framework when reading unset fields (Christian Kreibich, Corelight) Unset fields could trigger unexpected null pointers in the input manager. This also adds a warning such fields come up. 4.1.0-dev.837 | 2021-06-30 14:10:58 -0700 * Fix tests (Seth Hall, Corelight) * Removed a non-functional builtin plugin preload loading mechanism (Seth Hall, Corelight) * Load the builtin-plugin preload files and fix the dev path. (Seth Hall, Corelight) 4.1.0-dev.833 | 2021-06-30 08:50:39 -0700 * GH-1500: Mark BasicThread::Done to be ignored by ThreadSanitizer (Tim Wojtulewicz, Corelight) There's a known false positive with the atomic variables in this method that triggers a complaint from ThreadSanitizer. Marking it as ignored avoids the warning. 4.1.0-dev.831 | 2021-06-30 13:17:01 +0100 * Update the bundled SQLite to 3.36.0 (Johanna Amann, Corelight) 4.1.0-dev.830 | 2021-06-30 07:51:08 +0200 * GH-1406: Fix SMB tests on Apple M1. (Robin Sommer, Corelight) 4.1.0-dev.828 | 2021-06-29 11:11:22 -0700 * Remove remaining vestigial IDMEF support code (Christian Kreibich, Corelight) 4.1.0-dev.826 | 2021-06-29 10:39:18 -0700 * made RecordVal::AppendField protected: it's low-level & requires knowledge of internals (Vern Paxson, Corelight) 4.1.0-dev.821 | 2021-06-29 08:38:37 -0700 * Fix package name for CMake on CentOS 8 (Dominik Charousset, Corelight) 4.1.0-dev.818 | 2021-06-28 13:50:13 -0700 * GH-1216: Enable Mobile IPv6 support by default (Tim Wojtulewicz, Corelight) This removes the ENABLE_MOBILE_IPV6 #define variable. It also marks the --enable-mobile-ipv6 configure argument as deprecated. 4.1.0-dev.816 | 2021-06-28 11:08:29 -0700 * GH-572: Mark MemoryAllocation() and related methods deprecated (Tim Wojtulewicz, Corelight) 4.1.0-dev.814 | 2021-06-28 11:06:39 -0700 * Check for -1 return from FieldOffset() in Val::HasField() Fixes Coverity 1457804 (Tim Wojtulewicz, Corelight) 4.1.0-dev.812 | 2021-06-28 11:02:46 -0700 * whoops overlooked the need to canonicalize filenames (Vern Paxson, Corelight) * another set of tweaks per review comments (Vern Paxson, Corelight) * addressed a number of code review comments (Vern Paxson, Corelight) * baseline updates for merge (Vern Paxson, Corelight) * Merge remote-tracking branch 'origin/master' into topic/vern/ZAM-prep (Vern Paxson, Corelight) * support "any" coercions for "-O gen-C++" (Vern Paxson, Corelight) * better descriptions for named record constructors (Vern Paxson, Corelight) * test suite baseline updates for "-a opt" optimize-AST alternative (Vern Paxson, Corelight) * test suite baseline updates for "-a xform" alternative / AST transformation (Vern Paxson, Corelight) * error propagation fix for AST reduction (Vern Paxson, Corelight) * updates to "-a inline" test suite alternative baseline (Vern Paxson, Corelight) * updates for the main test suite baseline (Vern Paxson, Corelight) * updates to test suite tests for compatibility with upcoming ZAM functionality (Vern Paxson, Corelight) * "-O compile-all" option to specify compilation of inlined functions (Vern Paxson, Corelight) * compile inlined functions if they're also used indirectly (Vern Paxson, Corelight) * provide ZAM-generated code with low-level access to record fields (Vern Paxson, Corelight) * fix for cloning records with fields of type "any" (Vern Paxson, Corelight) * direct access for ZAM to VectorVal internal vector (Vern Paxson, Corelight) * ZVal constructors, accessors & methods in support of ZAM (Vern Paxson, Corelight) * switch ZVal representation of types from Type objects to TypeVal's (Vern Paxson, Corelight) * revised error-reporting interface for ZVal's, to accommodate ZAM inner loop (Vern Paxson, Corelight) * faster construction of records by factoring static decisions into RecordType's (Vern Paxson, Corelight) * make "switch" internals accessible to ZAM; tidying of same (Vern Paxson, Corelight) * factor out "cast" functionality to make available to lower-level ZAM access (Vern Paxson, Corelight) * tidying for check_and_promote_expr (Vern Paxson, Corelight) * employ explicit conversions to/from "any" and "vector of any" types (Vern Paxson, Corelight) * more robust treatment of arithmetic coercions (Vern Paxson, Corelight) * support for constructing VectorVal's directly from underlying ZVal vectors (Vern Paxson, Corelight) * support for ensuring that a vector can be treated as having a homogeneous type (Vern Paxson, Corelight) * factoring out of low-level vector indexing to make available to ZAM (Vern Paxson, Corelight) * minor changes for more robust behavior in the face of errors (Vern Paxson, Corelight) * gracefully deal with "eval" exceptions that occur during AST reduction (Vern Paxson, Corelight) * directly construct records of known types, rather than requiring coercion (Vern Paxson, Corelight) * fixes for treating WhileStmt's "loop_cond_pred_stmt" as a first-class citizen (Vern Paxson, Corelight) * support for profiling function bodies w/o needing accompanying ScriptFunc object (Vern Paxson, Corelight) * support for Frame's having call locations even if no associated CallExpr (Vern Paxson, Corelight) * fix for AST optimization altering top-level body statement (Vern Paxson, Corelight) * fix for analyzing variable usage inside of table initializers (Vern Paxson, Corelight) * fix for inlining type-based switch statements (Vern Paxson, Corelight) * fix for computing |size| of files and subnets (Vern Paxson, Corelight) * fix for tracking the effects of += operations (Vern Paxson, Corelight) * Expr method to invert the sense of a relational (Vern Paxson, Corelight) * Trigger constructor (and factoring) to support lower-level constructions (Vern Paxson, Corelight) * fixed / removed out-of-date comments, tidied check_and_promote_args() interface (Vern Paxson, Corelight) * convert scopes to be managed using IntrusivePtr's (Vern Paxson, Corelight) * various accessors used by ZAM compiler (Vern Paxson, Corelight) * Update submodule(s) [nomail] (Tim Wojtulewicz, Corelight) 4.1.0-dev.764 | 2021-06-27 10:50:19 -0700 * Remove unnecessary -B arguments from Zeek invocations in testsuite Now that Zeek no longer silently accepts -B when not compiled in debug mode, these tests were failing. (Christian Kreibich, Corelight) * Fix perftools-enabled build (Christian Kreibich, Corelight) * Minor tweaks to single-character command line option handling - Use of "-B" now triggers usage output and errors out when Zeek wasn't built with debugging support. - Always keep the perftools options (-m, -M) in the optparse string, for consistency with other flags dependent on configuration. We still fall through to usage and erroring out as before. - Minor indentation fix. (Christian Kreibich, Corelight) 4.1.0-dev.760 | 2021-06-27 10:46:01 -0700 * Add btest test case (Luke Cesarz) * Fix typo (Luke Cesarz) * Fix segfault with incomplete connection Add required HasField check before GetFieldAs call (Luke Cesarz) 4.1.0-dev.755 | 2021-06-23 13:53:54 -0700 * Call brew update-reset in ci/macos/prepare.sh This fixes some issues with the Catalina builds when it prepares the image and the base Cirrus image has old recipes for Homebrew. The VM then has to build a bunch of packages it shouldn't need to. (Tim Wojtulewicz, Corelight) * GH-1368: Use --osx-sysroot for macOS CI builds (Tim Wojtulewicz, Corelight) 4.1.0-dev.750 | 2021-06-21 16:14:03 -0700 * Remove the Stepping Stone analyzer This commit removes the stepping stone analyzer. It has been deactivated by default since at least Zeek 2.0, is dysfunctional in cluster settings and has a bunch of other issued. Relates to GH-1573 (Johanna Amann, Corelight) 4.1.0-dev.748 | 2021-06-21 15:41:29 -0700 * Drop Fedora 32 from CI, now past EOL (Christian Kreibich, Corelight) * Remove a double-defined TLS cert from a Broker btest (Christian Kreibich, Corelight) 4.1.0-dev.744 | 2021-06-21 09:17:36 +0200 * Add a new field `email_dest` to NOTICEs, which defines where to send email to. The email-related NOTICE actions fill this now, and then emails will be sent to all recorded addresses at the end of NOTICE processing. This makes email generation more consistent and extensible. (Vlad Grigorescu) * Add page and email administrator to mails processed by hostnames extension. (Vlad Grigorescu) 4.1.0-dev.731 | 2021-06-17 10:40:58 +0100 * Change SSH version field to be `&optional`. In version 3.3.0-dev.537 we added handling for SSH version 1.99 which used a SSH version of 0 to indicate weird cases where no version could be determined. This patch is a fixup for that patch. We now use an `&optional` version value. If no SSH version can be eixtracted the version will be unset; additionally a `conn_weird` event will be raised. See GH-1590. (Benjamin Bannier, Corelight) 4.1.0-dev.727 | 2021-06-14 16:19:34 -0700 * Bump Highwayhash submodule to pull in fix for FreeBSD (Christian Kreibich, Corelight) 4.1.0-dev.725 | 2021-06-11 11:54:30 -0700 * Fixes for the builtin plugin functionality (Seth Hall, Corelight) 4.1.0-dev.722 | 2021-06-10 10:42:57 -0700 * Added --include-plugins configure argument (Seth Hall, Corelight) 4.1.0-dev.720 | 2021-06-10 11:29:19 +0100 * Introduce script-land option LogAscii::logdir that can be used to set the logging directory. (Henrik Kramselund Jereminsen) 4.1.0-dev.715 | 2021-06-09 09:12:26 -0700 * Fix macOS Big Sur builds on Cirrus - Upgrade the Big Sur VM to use the Xcode 12.5 version. This has a newer version of brew installed on it that fixes an issue with an EOL package host that finally shut down for good recently. - Use 'brew upgrade' for openssl and cmake, since those are both present on the base VM. This prevents 'brew install' from printing an error if the package exists but is out of date. (Tim Wojtulewicz, Corelight) 4.1.0-dev.713 | 2021-06-08 13:54:28 -0700 * Add OpenSUSE Leap 15.3 to testing (Johanna Amann, Corelight) 4.1.0-dev.709 | 2021-06-07 09:41:28 +0200 * Improve assignment operators for IntrusivePtr. (Dominik Charousset, Corelight) * Fix docs for `ProcStats`: `mem` is in bytes, not KB. (Arne Welzel, Corelight) 4.1.0-dev.704 | 2021-06-04 08:29:18 -0700 * Add deprecated headers for UDP and ICMP analyzers (Tim Wojtulewicz, Corelight) * Fix handling of IP packets with bogus IP header lengths Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=34711 (Link to details becomes public 30 days after patch release) (Tim Wojtulewicz, Corelight) 4.1.0-dev.700 | 2021-06-03 09:27:57 -0700 * Make update-traces fail when the curl invocation fails (Christian Kreibich, Corelight) 4.1.0-dev.697 | 2021-06-02 15:08:12 -0700 * Add FreeBSD 13 to CI (Christian Kreibich, Corelight) * Add Fedora 34 to CI (Christian Kreibich, Corelight) 4.1.0-dev.693 | 2021-06-02 13:22:09 -0700 * Label session adapters in the output of zeek -NN (Tim Wojtulewicz, Corelight) * Split session adapter code into separate files from the analyzers (Tim Wojtulewicz, Corelight) * Move adapter-specific code back into the adapter (Tim Wojtulewicz, Corelight) * Move ICMP counterpart methods outside of ICMPAnalyzer class These were previously global methods in the old analyzer, and moving them to be private members of ICMPAnalyzer broke the usage of them by at least one external plugin. (Tim Wojtulewicz, Corelight) * Remove obsolete Skipping()/SetSkip() from Connection (Tim Wojtulewicz, Corelight) * Remove some code from IPBasedAnalyzer and children that was waiting for TCP to be implemented (Tim Wojtulewicz, Corelight) * Move TCPStateStats object out of session_mgr (Tim Wojtulewicz, Corelight) * Move analyzer-to-port mapping out of analyzer::Manager into packet analyzers (Tim Wojtulewicz, Corelight) * Move packet parsing code out of adapter into analyzer (Tim Wojtulewicz, Corelight) * Move old TCP analyzer into analyzer adapter in packet analysis tree (Tim Wojtulewicz, Corelight) 4.1.0-dev.681 | 2021-06-02 09:57:41 -0700 * Add some extra length checking when parsing mobile ipv6 packets Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=34263 (Link to details becomes public 30 days after patch release) (Tim Wojtulewicz, Corelight) 4.1.0-dev.679 | 2021-06-01 19:11:40 -0700 * Replace toupper() usages in netbios decoding BIFs (Jon Siwek, Corelight) This avoids potential for locale-dependent results of toupper() by instead using a function that simply maps ASCII characters a-z to A-Z. 4.1.0-dev.676 | 2021-06-01 10:19:19 -0700 * Integrate review feedback (Dominik Charousset, Corelight) * Sync new broker options, fix name inconsistencies (Dominik Charousset, Corelight) * Integrate new Broker metric exporter parameters (Dominik Charousset, Corelight) 4.1.0-dev.671 | 2021-06-01 09:51:38 -0700 * Update detect-MHR.zeek (Chris C) Update Virustotal URL to current option match_sub_url = "https://www.virustotal.com/gui/search/%s" 4.1.0-dev.669 | 2021-06-01 09:39:30 -0700 * GH-839: Fix use of &optional sub-records within table/set indices (Jon Siwek, Corelight) 4.1.0-dev.666 | 2021-05-26 10:51:51 -0700 * Ensure SessionAdapter members are initialized (Tim Wojtulewicz, Corelight) Fixes Coverity #1453273 4.1.0-dev.665 | 2021-05-26 08:07:26 +0200 * Extend the file analyzer API to set source manually. (Robin Sommer, Corelight) 4.1.0-dev.661 | 2021-05-24 15:03:54 -0700 * Update Broker submodule for bump of embedded CAF to 0.18.3 (Jon Siwek, Corelight) 4.1.0-dev.660 | 2021-05-24 12:38:44 -0700 * Add type field to session::Key to help avoid collisions in map (Tim Wojtulewicz, Corelight) * Move bad UDP checksum handling into adapter object (Tim Wojtulewicz, Corelight) * Rename IPBasedTransportAnalyzer to SessionAdapter (Tim Wojtulewicz, Corelight) This also also combines the old TransportLayerAnalyzer class into SessionAdapter, and removes the old class. This requires naming changes in a few places but no functionality changes. * Move building session analyzer tree out of analyzer::Manager (Tim Wojtulewicz, Corelight) * Rework the packet flow through the IP-based analyzers (Tim Wojtulewicz, Corelight) * Add new UDP packet analyzer, remove old one (Tim Wojtulewicz, Corelight) * Add new ICMP packet analyzer, remove old one (Tim Wojtulewicz, Corelight) * Add base class for IP-based packet analyzers (Tim Wojtulewicz, Corelight) * Move SessionManager::ParseIPPacket to IP analyzer's namespace (Tim Wojtulewicz, Corelight) * Added skeletons for TCP/UDP/ICMP packet analysis plugins. (Tim Wojtulewicz, Corelight) This includes integration into the IP plugin and calling of the sessions code from each plugin. 4.1.0-dev.646 | 2021-05-18 11:47:25 -0700 * Omit unneeded decimal points in modp_dtoa2() scientific notation output (Jon Siwek, Corelight) For example, "1e-13" is now used instead of "1.e-13". * GH-1244: Change modp_dtoa2() to use scientific notation for small values (Jon Siwek, Corelight) This fixes problems where printing floating point numbers less than 10^-6 output as "0.0". Such numbers now use using scientific notation and preserve the value's actual floating point representation. 4.1.0-dev.643 | 2021-05-17 11:57:58 -0700 * GH-1546: Make DictIterator() public, add copy/move operators (Tim Wojtulewicz, Corelight) 4.1.0-dev.641 | 2021-05-17 11:28:11 -0700 * GH-1558: Fix reading `vector of enum` types from config files (Jon Siwek, Corelight) * GH-1555: Fix reading empty set[enum] values from config files (Jon Siwek, Corelight) 4.1.0-dev.638 | 2021-05-17 13:08:28 +0100 * Manual page updates (Henrik Kramselund Jereminsen) 4.1.0-dev.631 | 2021-05-11 09:26:37 -0700 * Add unit tests to ZeekString.cc (Tim Wojtulewicz) 4.1.0-dev.628 | 2021-05-10 12:44:25 -0700 * Add experimental support for translating Zeek scripts to equivalent C++ (Vern Paxson, Corelight) The generated C++ can then be compiled directly into the `zeek` binary, replacing use of the interpreter and producing better runtime performance. See `src/script_opt/CPP/README.md` for a guide on how to use this feature. * Add new "-a cpp" btest alternative (Vern Paxson, Corelight) * Add VectorVal methods to leverage ZVal representation (Vern Paxson, Corelight) * Fix backtrace BiF to avoid iterator invalidation & support compiled code (Vern Paxson, Corelight) 4.1.0-dev.593 | 2021-05-10 10:17:34 +0100 * Explain zeek-config options in help output (Christian Kreibich, Corelight) * Sort variables at top of zeek-config alphabetically (Christian Kreibich, Corelight) * Install Zeek's btest tooling with the distribution This creates $PREFIX/share/btest in the install tree, with the following folders: - scripts/ for the canonifiers - data/ for random.seed - data/pcaps for the test pcaps The pcaps can be skipped by configuring with --disable-btest-pcaps. (Christian Kreibich, Corelight) 4.1.0-dev.587 | 2021-05-05 14:05:51 +0000 * Merge remote-tracking branch 'origin/topic/timw/session-coverity' * origin/topic/timw/session-coverity: Minor cleanup in IPAddr.h Fix a few Coverity warnings from the session manager work (Tim Wojtulewicz) * Minor cleanup in IPAddr.h (Tim Wojtulewicz, Corelight) * Fix a few Coverity warnings from the session manager work - Be explicit about setting the copied flag in session::Key. Coverity seems confused about when that flag is set if it gets set by default initialization. This should fix 1452757 and 1452759. - Explicitly copy the fields in ConnKey instead of using memcpy. Fixes 1452758. (Tim Wojtulewicz, Corelight) 4.1.0-dev.583 | 2021-05-03 18:21:33 -0700 * switched RecordVal's to use std::optional for tracking missing fields (Vern Paxson, Corelight) * added constructors for directly building ZVal's (Vern Paxson, Corelight) 4.1.0-dev.580 | 2021-04-30 18:29:22 -0700 * Add missing zeek/ prefix to a telemetry header's includes (Jon Siwek, Corelight) 4.1.0-dev.578 | 2021-04-30 09:42:26 -0700 * GH-1534: Fix excessive coredump for duplicate enum definitions An adequate error message was previously reported for duplicate enum definitions, this just now prevents trying to access it as a constant in subsequent parsing and further generating a coredump. (Jon Siwek, Corelight) 4.1.0-dev.576 | 2021-04-30 09:40:18 -0700 * Fixes to `decode_netbios_name` and `decode_netbios_name_type` BIFs Fixes to `decode_netbios_name`: * Improve validation that input string is a NetBIOS encoding (32 bytes, with characters ranging from 'A' to 'P'). This helps prevent Undefined Behavior of left-shifting negative values. Invalid encodings now cause a return-value of an empty string. * More liberal in what decoded characters are allowed. Namely, spaces are now allowed (but any trailing null-bytes and spaces are trimmed, similar to before). Fixes to `decode_netbios_name_type`: * Improve validation that input string is a NetBIOS encoding (32 bytes, with characters ranging from 'A' to 'P'). This helps prevent Undefined Behavior of left-shifting negative values and a heap-buffer-overread when the input string is too small. Invalid encodings now cause a return-value of 256. (Jon Siwek, Corelight) 4.1.0-dev.573 | 2021-04-29 11:29:39 -0700 * Rename ConnID and ConnIDKey (Tim Wojtulewicz, Corelight) * Remove Session prefix from some session-related classes and files (Tim Wojtulewicz, Corelight) * Move session code into new directory and into zeek::session namespace (Tim Wojtulewicz, Corelight) * Move SessionKey into a separate file, added comments (Tim Wojtulewicz, Corelight) * Review cleanup - Add constructors for ConnIDKey, remove BuildConnIDKey() - Rename protocol stats classes and move to implementation file - Rename "num" field of protocol stats to "active" - Explicitly delete copy operations for SessionKey - Change argument for ProtocolStats methods to const-reference - Make key validity methods in Session not be virtual - Rename Session::ClearKey and Session::IsKeyValid (Tim Wojtulewicz, Corelight) * Rework stats to store handles to the counters (Tim Wojtulewicz, Corelight) * Rename some connection-specific methods in Session and SessionManager (Tim Wojtulewicz, Corelight) * Add deprecated version of Sessions.h (Tim Wojtulewicz, Corelight) * Rename NetSessions to SessionManager This also includes: - Deprecating the NetSessions name. - Renaming the zeek::sessions global to zeek::session_mgr and deprecating the old name. - Renaming Sessions.{h,cc} to SessionManager.{h,cc}. (Tim Wojtulewicz, Corelight) * Store a single map of Sessions instead of split maps of Connections. This commit also includes: - Storing the transport protocol in ConnID and ConnIDKey to allow tcp and udp connections from the same IP/Port combinations. This happens in the core.cisco-fabric-path test, for example. - Lots of test updates. The reasons for these are two fold. First, with the change to only store a single map means that TCP, UDP, and ICMP connections are now mixed. When Zeek drains the map at shutdown, it drains each of those protocols together instead of separately. The second is because of how Sessions are stored in the map. We're now storing them keyed by the hash of the key stored by the Session objects, which causes them to again be in the map in a different order. (Tim Wojtulewicz, Corelight) * Convert session stats to use the new telemetry API (Tim Wojtulewicz, Corelight) * Add test for get_conn_stats BIF before reworking session stats (Tim Wojtulewicz, Corelight) * Add new Session base class This is mostly code copied from the existing Connection class, as that class now inherits from Session. (Tim Wojtulewicz, Corelight) * Move packet filter out of NetSessions (Tim Wojtulewicz, Corelight) * Move some code out of NetSessions - TCPStateStats update when a session is removed was moved to Connection - Stepping Stone manager moved to a singleton object in SteppingStoneManager (Tim Wojtulewicz, Corelight) 4.1.0-dev.557 | 2021-04-29 09:00:51 -0700 * Add basic testing for Geneve protocol analyzer The added pcap file was downloaded from an attachment to https://gitlab.com/wireshark/wireshark/-/issues/10193 without explicit license. (Benjamin Bannier, Corelight) * GH-1517: Add Geneve decap support This patch adds the ability to decap Geneve packets to process the inner payload. The structure of the analyzer borrows heavily from the VXLAN analyzer. (Benjamin Bannier, Corelight) 4.1.0-dev.554 | 2021-04-28 13:45:53 -0700 * Add a fatal error condition for invalid Dictionary insertion distances (Jon Siwek, Corelight) When choosing poor/aggressive values for `table_expire_interval`, `table_expire_delay`, and/or `table_incremental_step` that tend to leave tables in state of constant table-expiry-iteration, the underlying Dictionary is never allowed the chance to complete remapping operations which re-position entries to more ideal locations (e.g. after reallocating the table to be able to store more entries). That situation not only leads to the Dictionary generally having a less efficient structure, but eventually, the lack of re-positioning may cause an insertion to calculate the new entry's distance-from-ideal-position to be a value requiring a full 16-bits or more (>=65535), but an entry only allows storing 16-bit distance values, with 65535 being a sentinel value that is supposed to indicate an empty entry. Dictionary operations may start misbehaving if that's allowed to happen. * Fix using clear_table() within an &expire_func (Jon Siwek, Corelight) This previously crashed since clear_table()/TableVal::RemoveAll() left behind a stale iterator to the old table causing a heap-use-after-free when resuming table expiry iteration in TableVal::DoExpire(). * Remove saving/restoring of value pointer after calling expire_func (Jon Siwek, Corelight) It's no longer used for anything. Previously, it was used to detect whether the expiry batch finished iterating the entire table or not, but that's now determined by directly checking if the iterator itself signifies the end of the table. * Avoid allocating a HashKey for no-op table expiry iterations (Jon Siwek, Corelight) 4.1.0-dev.549 | 2021-04-28 13:09:30 -0700 * Fix -Wsign-compare warnings in Debug{Cmds}.cc (Jon Siwek, Corelight) 4.1.0-dev.547 | 2021-04-28 09:27:15 -0700 * GH-1528: Remove broken Queue/PQueue class, replace with std::deque (Tim Wojtulewicz, Corelight) 4.1.0-dev.545 | 2021-04-26 11:14:49 -0700 * GH-1483: Improve error for mismatched container initialization types (Jon Siwek, Corelight) For example, trying to assign a vector, table, set, or record constructor expression to a global variable of a different type now provides a more explanatory error message than the previous "Val::CONVERTER" fatal-error and coredump. 4.1.0-dev.543 | 2021-04-26 09:53:53 -0700 * Update btest baselines for --enable-mobile-ipv6 builds (Tim Wojtulewicz, Corelight) * Build ubuntu18 on CI with --enable-mobile-ipv6 (Tim Wojtulewicz, Corelight) 4.1.0-dev.539 | 2021-04-20 15:21:03 -0700 * Fix LGTM finding in zeek-cut with uses of localtime and gmtime (Tim Wojtulewicz, Corelight) 4.1.0-dev.537 | 2021-04-20 14:34:48 -0700 * Restore support for vectors with holes and improve test cases (Vern Paxson, Corelight) There's no longer breaking behavior changes to how vector-holes work from previous Zeek versions. * Fix cloning/copying vectors that contain holes (Vern Paxson, Corelight) * fix using ++/-- to vectors that contain holes (Vern Paxson, Corelight) 4.1.0-dev.526 | 2021-04-16 16:03:06 -0700 * GH-1506: Fix Broker unserialization of set/table function indices (Jon Siwek, Corelight) Zeek function types are serialized as a broker::vector, but the unserialization logic for Zeek set/table types that use a function for as an index incorrectly identified it as a composite-index, which also use broker::vector, rather than a singleton-index, and makes such unserialization fail. A general example where this failure can happen in practice is when trying to unserialize a connection record for which there's a Conn::RemovalHook, since that's a set[function], and a specific case of that is use of the Intel Framework in a Zeek cluster. 4.1.0-dev.524 | 2021-04-16 08:08:38 -0700 * Move an assert() in input/Manager.cc to account for ValueToVal errors (Jon Siwek, Corelight) * Add test for config framework (Tim Wojtulewicz, Corelight) * Fix similar issues with ValueTo* methods in the input framework (Tim Wojtulewicz, Corelight) * GH-1487: Handle error from ValueToVal instead of ignoring it (Tim Wojtulewicz, Corelight) 4.1.0-dev.519 | 2021-04-15 17:44:16 -0700 * Fix indexing of set/table types with a vector (Jon Siwek, Corelight) That previously caused an internal error/crash * GH-1507: Tolerate junk data before SIP requests (Jon Siwek, Corelight) This allows for data that won't match a SIP request method to precede an actual request and generates a new 'sip_junk_before_request' weird when encountering such a situation. * GH-1503: Improve &expire_func compatibility type-checking (Jon Siwek, Corelight) Previously, incompatible &expire_funcs could mistakenly be used, such as when using that attribute on the unspecified table()/set() initializations/assignments, resulting in invalid function calls that eventually crash Zeek. 4.1.0-dev.512 | 2021-04-15 11:54:44 -0700 * GH-1496: Fix build on armv7 architecture (Jon Siwek, Corelight) 4.1.0-dev.510 | 2021-04-14 16:05:55 -0700 * GH-1450: Improve printing/logging of large double/interval/time values The modp_dtoa/modp_dtoa2 functions aren't capable of handling double values larger than INT_MAX and fallback on using sprintf() in that situation. Previously, the format string to that sprintf() was "%e", defaulting to a precision of 6, which is already too few digits to represent a number known to be larger than INT_MAX. Now, an sprintf() is still performed for values larger than INT_MAX and still uses a scientific notation format, but in a way that uses as many decimal digits as needed to preserve information. (Jon Siwek, Corelight) 4.1.0-dev.508 | 2021-04-14 15:54:03 -0700 * Add btest for unterminated pattern parsing error behavior And clarified the error message that it's more about finding an unterminated pattern than knowing for sure there's remaining pattern text spanning multiple lines. (Jon Siwek, Corelight) * GH-1497: Support CRLF line-endings in Zeek scripts and signature files (Jon Siwek, Corelight) * Escape non-printables in "unrecognized character" parser error messages (Jon Siwek, Corelight) 4.1.0-dev.504 | 2021-04-14 15:51:39 -0700 * Add a check for null packet data in pcap IOSource Some libpcaps (observed in Myricom's) may claim to have read a packet, but either did not really read a packet or at least provide no way to access its contents, so this adds a check for null-data to handle those cases. (Jon Siwek, Corelight) 4.1.0-dev.501 | 2021-04-12 18:03:53 -0700 * Add explanation of vector holes/in-operator changes to NEWS (Jon Siwek, Corelight) 4.1.0-dev.500 | 2021-04-09 12:18:06 -0700 * Reference the manual's explanation of originator and responder from conn_id (Christian Kreibich, Corelight) 4.1.0-dev.498 | 2021-04-07 18:30:58 -0700 * Disable Broker tests in Coverity Scan cronjob build (Jon Siwek, Corelight) 4.1.0-dev.497 | 2021-04-07 15:54:21 -0700 * GH-1493: Fix build with -DENABLE_MOBILE_IPV6 (Tim Wojtulewicz, Corelight) 4.1.0-dev.495 | 2021-04-07 11:12:13 -0700 * Change TableVal::ToMap() to return ValPtr-indexed maps (Jon Siwek, Corelight) * Add std::hash specialization for IntrusivePtr (Jon Siwek, Corelight) * Move IntrusivePtr relational operators to zeek namespace Otherwise some cases relying on argument-dependent lookup (ADL) fail. (Jon Siwek, Corelight) 4.1.0-dev.490 | 2021-04-05 14:13:48 -0700 * Add Telemetry API (C++/BIFs) for gathering runtime metrics (Dominik Charousset, Corelight) This relies on the CAF metrics API/implementation and allows potential export to Prometheus. These typical metric types are supported: counters, gauges, histograms, timers. 4.1.0-dev.475 | 2021-04-03 09:39:10 -0700 * Increase timeout of plugins.reader btest (Jon Siwek, Corelight) * optionally hash original, not extended, form of records for profiling (Vern Paxson, Corelight) profile types associated with global initializations incorporate type name, if present, into type hash * fixed out-of-range enum constant: OpenFlow::INVALID_COOKIE (Vern Paxson, Corelight) (Though it's not currently an enum value) * Add method to add a fully-qualified enum name (module name included) (Vern Paxson, Corelight) * enum types track whether they've had values added via "redef" (Vern Paxson, Corelight) * tracking of original size of records (pre redef'ing) (Vern Paxson, Corelight) * Added TableVal::ToMap to retrieve a table's entire contents as a unordered_map (Vern Paxson, Corelight) * factoring out some replicated code in the parser (Vern Paxson, Corelight) * some micro-preening of parse.y; no semantic changes (Vern Paxson, Corelight) * for readability, removed explicit scoping in parse.y; no semantic changes (Vern Paxson, Corelight) 4.1.0-dev.461 | 2021-04-01 14:11:44 -0700 * function profiling rewritten - more detailed info, supports global profiling (Vern Paxson, Corelight) Hashes for Zeek script types are now done globally rather than per-function-body, which can save considerable time due to the complexity of some commonly used types (such as connection records). Hashing has been expanded to provide more robust distinctness (lack of collisions in practice) and determinism (consistently computing the same hash across compilations). * track whether a given function/body should be included/skipped for optimization (Vern Paxson, Corelight) 4.1.0-dev.451 | 2021-03-31 11:58:08 -0700 * Add ssh to Alpine Dockerfile for retrieving external test repos (Jon Siwek, Corelight) 4.1.0-dev.449 | 2021-03-31 10:47:22 -0700 * Change ci/init-external-repos.sh to use `base64 -d` (Jon Siwek, Corelight) The BusyBox version of `base64` does not have `--decode`. 4.1.0-dev.448 | 2021-03-31 10:29:17 -0700 * Teach ci/init-external-repos.sh to consider user permissions (Jon Siwek, Corelight) Any errors while setting up external/private test repo should fail the task for PRs submitted by a user with write/admin permission. 4.1.0-dev.447 | 2021-03-31 09:55:05 -0700 * CI: Add OpenSUSE Leap 15.2 (Johanna Amann, Corelight) * Add CI task for Alpine Linux (Jon Siwek, Corelight) * Separate stdout from stderr in btest baselines (Jon Siwek, Corelight) Redirecting both to the same file can show platform-specific differences (e.g. Alpine), likely due to different buffering defaults. * Remove newline-eof canonification attempt in diff-remove-timestamps (Jon Siwek, Corelight) * Change a include to (Jon Siwek, Corelight) Some systems (e.g. Alpine) may warn that that the former is incorrect. 4.1.0-dev.440 | 2021-03-30 14:57:07 -0700 * Fix incomplete-type for struct timeval (Andrew Benson) This fixes building on musl (e.g. Void, Alpine, etc.) 4.1.0-dev.438 | 2021-03-29 15:42:25 -0700 * "balance" tests with multiple Zeek scripts to load the same elements (Vern Paxson, Corelight) * put global statements into a quasi-function to support script optimization (Vern Paxson, Corelight) 4.1.0-dev.434 | 2021-03-29 13:18:18 -0700 * Fix sign-compare compiler warning in coerce_to_record() (Jon Siwek, Corelight) * Fix maybe-uninitialized warning in ZVal::ToVal() (Jon Siwek, Corelight) * Change RecordVal::GetFieldAs() to use std::vector::operator[] (Jon Siwek, Corelight) Since the method claims it's up to the user to ensure the field exists before calling, the extra bounds-checking done by std::vector::at() isn't needed. * Add RecordVal::AssignField() and use it in supervisor code (Jon Siwek, Corelight) This is a convenience method to assign a known record field value by field name. May also be useful to reduce warnings from static analysis (e.g. Coverity) about not checking for negative return values before assigning since that now flows through a [[noreturn]] error path. * GH-960: Fix include order of bundled header files (Jon Siwek, Corelight) Previously, a system-wide installation of any bundled auxil/ software (like CAF) may get found/included rather than the bundled version and possibly break the build. 4.1.0-dev.427 | 2021-03-27 14:18:16 -0700 * Update bundled CAF to 0.18.2 (Jon Siwek, Corelight) 4.1.0-dev.426 | 2021-03-26 17:17:45 -0700 * GH-1463: Rename supervisor_rotation_format_func to archiver_rotation_format_func (Vlad Grigorescu) And expose it for non-supervised setups. Also deprecates the old name. 4.1.0-dev.423 | 2021-03-26 16:44:26 -0700 * GH-1454: Heartbleed: fix substraction order. (Johanna Amann) Fixes incorrect/overflowed `n` value for `SSL_Heartbeat_Many_Requests` notices where number of server heartbeats is greater than number of client heartbeats. The larger number was substracted from the smaller one leading to an integer overflow. However, no information was lost due to everything also being present in the notice message. 4.1.0-dev.421 | 2021-03-26 16:39:22 -0700 * Fix crash in Analyzer::ForwardPacket due to recursive analyzer calls. (Tim Wojtulewicz, Corelight) The change in 44f558df7b5a85bae40945de653bcb2448e0a7f4 that made analyzer_list a std::vector instead of a std::list doesn't take into account that in some cases an analyzer may chain back into itself, such as with UDP-in-UDP tunnels. In these cases, the second call to ForwardPacket may cause iterator invalidation, leading to a crash, so this reverts back to using an std::list. * Include git sha in request to benchmark host (Tim Wojtulewicz, Corelight) 4.1.0-dev.417 | 2021-03-25 11:37:55 -0700 * test suite update due to factoring out coerce_to_record() (Vern Paxson, Corelight) * removal of vestigial #include's (breaking an include loop in the process) (Vern Paxson, Corelight) * better method name: HasCopySemantics() (Vern Paxson, Corelight) * Use STL functionality to initialize coercion map (Vern Paxson, Corelight) * comments for factored-out index slice functions (Vern Paxson, Corelight) * support for subclassing ScriptFunc's, esp. for alternate lambda closures (Vern Paxson, Corelight) * factor out record coercion; modernize management of coercion "map" (Vern Paxson, Corelight) * lower-level method for adding fields to records (Vern Paxson, Corelight) * factor out "print" statement's execution functionality (Vern Paxson, Corelight) * functions for indexing slices and strings (Vern Paxson, Corelight) * new function for getting the location, if any, associated with the current call (Vern Paxson, Corelight) 4.1.0-dev.404 | 2021-03-24 16:58:50 -0700 * Fix missing `user_agent` existence check in smtp/software.zeek (Michael Dopheide) Without the check there can be extra reporter.log error noise, but no actual difference in functionality. 4.1.0-dev.402 | 2021-03-24 15:49:52 -0700 * Add `bin_dir` setting for default zkg config file (Jon Siwek, Corelight) Which defaults to `${zeek_install_prefix}/bin`, same as what a standalone `zkg autoconfig` would use. 4.1.0-dev.400 | 2021-03-23 20:44:19 -0700 * whitespace tweaks (Vern Paxson, Corelight) * resolved some TODO comments (Vern Paxson, Corelight) * remove unnecessary casts, and change necessary ones to use static_cast<> (Vern Paxson, Corelight) * explain cmp_func default (Vern Paxson, Corelight) * change functions for ZVal type management to static members (Vern Paxson, Corelight) * fix some unsigned/signed integer warnings (Vern Paxson, Corelight) * address lint concern about uninitialized variable (Vern Paxson, Corelight) * Remove use of obsolete forward-declaration macros (Tim Wojtulewicz, Corelight) * Merge branch 'topic/vern/zval' of github.com:zeek/zeek into topic/vern/zval (Vern Paxson, Corelight) * fix #include's that lack zeek/ prefixes (Vern Paxson, Corelight) * overlooked another way in which vector holes can be created (Vern Paxson, Corelight) * initialize vector holes to the correct corresponding type (Vern Paxson, Corelight) * explicitly populate holes created in vectors (Vern Paxson, Corelight) * fix other instances of GetField().get() assuming long-lived ValPtr's (Vern Paxson, Corelight) * fix for now-incorrect assumption that GetField always returns an existing ValPtr (Vern Paxson, Corelight) * ... and porting over memory management for assignment to vector elements (Vern Paxson, Corelight) * ... porting over memory management for assignment to record fields (Vern Paxson, Corelight) * dagnabbit, forgot to port over destructor cleanup from ZAM_vector/ZAM_record (Vern Paxson, Corelight) * fix #include's that lack zeek/ prefixes (Vern Paxson, Corelight) * explicitly populate holes created in vectors (Vern Paxson, Corelight) * fixes for now-incorrect assumption that GetField always returns an existing ValPtr (Vern Paxson, Corelight) * memory management for assignment to vector elements (Vern Paxson, Corelight) * memory management for assignment to record fields (Vern Paxson, Corelight) * destructor cleanup from ZAM_vector/ZAM_record (Vern Paxson, Corelight) * test suite update for Smith-Waterman now using 0-based vectors (Vern Paxson, Corelight) * test suite updates for expanded sort() BiF semantics (bools, doubles) (Vern Paxson, Corelight) * test suite updates reflecting that vectors no longer support arbitrary "holes" (Vern Paxson, Corelight) * migration to using new differentiated methods for setting record fields (Vern Paxson, Corelight) * RecordVal methods for differentiated setting of record fields (Vern Paxson, Corelight) * fix for fencepost error when looping over vectors (Vern Paxson, Corelight) * migrate to using RecordVal's new Remove() method (Vern Paxson, Corelight) * new RecordVal Remove() method to clear a field (Vern Paxson, Corelight) * use differentiated record field accessors (Vern Paxson, Corelight) * base support for differentiated record field accessors (Vern Paxson, Corelight) * migrate to differentiated vector "At" accessors to support future efficiency (Vern Paxson, Corelight) * fix for subtle bug due to GetField now potentially returning a new ValPtr (Vern Paxson, Corelight) * use HasField() rather than GetField() where appropriate (Vern Paxson, Corelight) * moving sort()/order() functionality into VectorVal (Vern Paxson, Corelight) * convert VectorVal's to use vector as internal representation (Vern Paxson, Corelight) * convert RecordVal's to use vector as internal representation (Vern Paxson, Corelight) * added ZVal for low-level representations of Zeek script values (Vern Paxson, Corelight) * fix mis-typed record fields that happened to work in original RecordVal impl. (Vern Paxson, Corelight) * fix off-by-one assumption regarding vector indexing dating to 2011 (Vern Paxson, Corelight) 4.1.0-dev.356 | 2021-03-23 19:33:44 -0700 * Increase timeout of openflow.log-cluster btest (Jon Siwek, Corelight) 4.1.0-dev.355 | 2021-03-23 19:05:10 -0700 * add option for deterministic descriptions of sets & tables (Vern Paxson, Corelight) * determinism for concurrent Zeek test suite invocations; split out deprecations (Vern Paxson, Corelight) * disambiguate descriptions of enum types; include attributes when describing record types (Vern Paxson, Corelight) * more liberal view of attribute equality; allow suppressing attr type-checking (Vern Paxson, Corelight) * support for operations on sets that return new values (Vern Paxson, Corelight) * add low-level method for adding new enum values (Vern Paxson, Corelight) * sundry accessors/cast-ers; RE_Matcher's track their construction values (Vern Paxson, Corelight) * add convenience functions for comparing IP addresses (Vern Paxson, Corelight) 4.1.0-dev.346 | 2021-03-23 17:25:02 -0700 * Add new ``./configure --plugindir`` option (Vlad Grigorescu) Useful for customizing the default installation path for plugins. 4.1.0-dev.344 | 2021-03-23 16:29:27 -0700 * Clean up iterator usage in Analyzer (Tim Wojtulewicz, Corelight) The ``zeek::analyzer::analyzer_list`` type-alias changed from an ``std::list`` to ``std::vector`` which, in practice, is not expected to be used from plugins in API-incompatible way and may result in ~1-2% overall performance benefit. 4.1.0-dev.341 | 2021-03-18 12:45:04 -0700 * fix for associating current scope with the name of enums; name tidying (Vern Paxson, Corelight) * avoid infinite recursion in same_type() if it is analyzing recursive types (Vern Paxson, Corelight) * remove iffy reliance on type punning that relies on interpreter's behavior (Vern Paxson, Corelight) * fixes for propagating optimization options, and pruning script function analysis (Vern Paxson, Corelight) * tidying with respect to "const", and streamlining OuterIDBindingFinder (Vern Paxson, Corelight) * making Exec methods non-const, so execution can manage side-information (Vern Paxson, Corelight) * removing unneeded BoolVal constructor that can cause overload ambiguity issues (Vern Paxson, Corelight) * lint fixes: ensuring functions return values, robustness to nil Val's (Vern Paxson, Corelight) * correcting some simple typos (Vern Paxson, Corelight) 4.1.0-dev.330 | 2021-03-17 13:48:34 -0700 * Prevent use of LeakSanitizer on FreeBSD (Jon Siwek, Corelight) 4.1.0-dev.328 | 2021-03-17 10:35:59 -0700 * Add tests for ERSPAN Type I patch Co-authored-by: Markel Elorza Alvarez Co-authored-by: Ivan Arrizabalaga Cupido (Gorka Olalde Mendia) * Add ERSPAN Type I patch Co-authored-by: Markel Elorza Alvarez Co-authored-by: Ivan Arrizabalaga Cupido (Gorka Olalde Mendia) 4.1.0-dev.323 | 2021-03-11 12:43:21 -0800 * ts fields in SMB logs now default to network_time() (Seth Hall, Corelight) This avoids a problem identified by amanbansal2709 in pull request #1288. I fixed it in a different way than that pull request by making sure the ts field is always set so that this isssue doesn't return in the future. 4.1.0-dev.320 | 2021-03-10 12:29:13 -0800 * GH-1432: Use buffered IO for file extraction (Jon Siwek, Corelight) This can improve performance significantly: ~3.5x faster when tested on a large file passing data to the file analysis framework in small chunks of 20 bytes. 4.1.0-dev.318 | 2021-03-10 12:05:47 -0800 * Avoid searching a directory for dynamic plugins multiple times (Jon Siwek, Corelight) 4.1.0-dev.316 | 2021-03-10 13:00:27 +0000 * Fix potential mime type detection bug in IRC/FTP file_transferred event The files framework uses strncpy to copy data into the buffer that is used for IRC/FTP mime type detection. From all I can tell that means that, in these cases, currently mime type detection will be messed up if the data being passed in contains zero bytes. (Johanna Amann, Corelight) 4.1.0-dev.314 | 2021-03-08 18:28:22 -0800 * new "opt" btest alternative (Vern Paxson, Corelight) * update to "xform" test suite baseline due to recent line number changes (Vern Paxson, Corelight) * logic for driving AST optimization when requested (Vern Paxson, Corelight) * methods implementing AST optimization (aliasing, constant propagation, CSE) (Vern Paxson, Corelight) * helper class checking if common-subexpression elimination opportunity is valid (Vern Paxson, Corelight) * helper function for comparing collections of definition points (Vern Paxson, Corelight) * track more information about temporary variables (Vern Paxson, Corelight) * simplify and tidy up some script-opt interfaces (Vern Paxson, Corelight) * enable setting "-O optimize-AST" option (Vern Paxson, Corelight) * fix for reducing operations on boolean vectors (Vern Paxson, Corelight) * Add flag to FuncType::CheckArgs() to control type-equivalence warning output (Vern Paxson, Corelight) * remove unneeded IndexExpr::Traverse() virtual method (same as inherited method) (Vern Paxson, Corelight) * accessors for additional Expr subclasses (Vern Paxson, Corelight) * convenience functions to check for aggregate types (Vern Paxson, Corelight) * fix header includes to have proper zeek/ prefixes (Vern Paxson, Corelight) 4.1.0-dev.288 | 2021-03-08 12:29:14 -0800 * GH-1426: Improve handling of Broker data store creation failures (Jon Siwek, Corelight) Broker::create_master() and Broker::create_clone() now return a valid value even when there's a failure to open the backend database (e.g. SQLite filesystem error). In that case, the returned value can still be passed into other data store operations, but they'll fail immediately with an error. Broker::is_closed() can now also be used to determine whether the data store creation calls failed. 4.1.0-dev.286 | 2021-03-08 11:24:38 +0000 * Allow non-TCP based protocols to use SSL analyzer. (Keith Jones) 4.1.0-dev.284 | 2021-03-05 17:37:03 -0800 * Simplify some copy_string() usages in threading::Field (Jon Siwek, Corelight) * Use const-reference in plugin::Manager::MetaHookPost for minor performance gain (Tim Wojtulewicz, Corelight) * Fix missing assigmnent operator/copy constructor pairings reported by LGTM (Tim Wojtulewicz, Corelight) * Fix variable shadowing issues reported by LGTM (Tim Wojtulewicz, Corelight) 4.1.0-dev.277 | 2021-03-05 16:14:23 -0800 * Avoid reporting paths repeatedly in zeek-config --include_dir (Christian Kreibich, Corelight) * Fix comment in IP analyzer (Tim Wojtulewicz, Corelight) 4.1.0-dev.272 | 2021-03-01 11:36:06 -0800 * Change update-zeekygen-docs.sh to set release branch in Sphinx config (Jon Siwek, Corelight) Allows release branches to generate correct links to GitHub source code. * Teach Zeekygen to produce source-code-range information (Jon Siwek, Corelight) Related to https://github.com/zeek/zeek-docs/issues/56 * Add normalize_script_path() zeek::zeekygen::detail namespace (Jon Siwek, Corelight) * Add starts_with()/ends_with() to zeek::util namespace (Jon Siwek, Corelight) 4.1.0-dev.266 | 2021-02-25 08:45:49 -0700 * Avoid superfluous string copies when adding to zeek::detail::sig_files Coverity 1387015 (Jon Siwek, Corelight) * Initialize an RD_Decorate member via std::move (Jon Siwek, Corelight) * Fix invalid iterator comparison in UseDefs::FindSuccUsage() Coverity 1447018 (Jon Siwek, Corelight) 4.1.0-dev.260 | 2021-02-24 15:52:31 -0800 * Remove obsolete ZEEK_FORWARD_DECLARE_NAMESPACED macros (Tim Wojtulewicz, Corelight) 4.1.0-dev.258 | 2021-02-24 15:10:51 -0700 * Minor Fedora 32/33 updates in CI - Switch Fedora package management from yum to dnf, the default - Minimize set of requested RPMs (Christian Kreibich, Corelight) * Add CentOS Stream 8 to CI There's no official Docker image on the Docker Hub, it's on quay.io: https://quay.io/repository/centos/centos?tab=tags At the moment, "Stream" and "Stream 8" are synonymous. (Christian Kreibich, Corelight) 4.1.0-dev.255 | 2021-02-24 12:35:47 -0800 * Support explicit disabling of file analyzers (Jon Siwek, Corelight) This adds various methods/BIFs the enable/disable file analyzers or query whether they're currently enabled. 4.1.0-dev.252 | 2021-02-23 12:01:52 -0800 * "xform" btest alternative baseline update (Vern Paxson, Corelight) * Test baselines updates related to "unused assignment" warnings (Vern Paxson, Corelight) * USE ZEEK_USAGE_ISSUES environment variable in tests rather than -u/-uu (Vern Paxson, Corelight) Since it has precedence over the flag arguments and don't want btest alternatives to override it. * splitting out "usage" test suite alternative into -u/-uu versions (Vern Paxson, Corelight) * adding &is_used attribute for base scripts (Vern Paxson, Corelight) Not actually needed yet, but will be once optimization is added. * removing unused assignments from base scripts (Vern Paxson, Corelight) * enhancements/changes to the Reduce class in preparation for use-defs (Vern Paxson, Corelight) * Add "-Odump-uds" and "ZEEK_DUMP_UDS" flag/environment variable for dumping use-defs (Vern Paxson, Corelight) * Add &is_used attribute to suppress set-but-not-used usage warnings (Vern Paxson, Corelight) * Support for "unused assignment" dead-code detection/warning via -u/-uu analysis (Vern Paxson, Corelight) 4.1.0-dev.229 | 2021-02-16 12:32:23 -0800 * Update Broker submodule for performance/credit-tracking fixes (Jon Siwek, Corelight) 4.1.0-dev.228 | 2021-02-16 11:10:40 -0800 * Fix `major_subsys_version` field in `pe_optional_header` event It was incorrectly set the same as the `minor_subsys_version` field of the `PE::OptionalHeader` record. (Jon Siwek, Corelight) 4.1.0-dev.225 | 2021-02-12 14:40:14 -0800 * Fix CentOS 8 CI Dockerfile (Jon Siwek, Corelight) The "PowerTools" repoid changed to "powertools": https://bugs.centos.org/view.php?id=17920 4.1.0-dev.224 | 2021-02-12 11:18:32 -0800 * GH-1398: Fix buffer overread in ascii formatter (Johanna Amann, Corelight) When a text with an (escaped) zero byte was passed to ParseValue, only the part of the string up to the zero byte was copied, but the length of the full string was passed to the input framework. This leads to the input manager reading over the end of the buffer. 4.1.0-dev.220 | 2021-02-11 11:10:46 -0800 * GH-1399: Remove RocksDB configure script options (Jon Siwek, Corelight) 4.1.0-dev.219 | 2021-02-11 10:56:16 -0800 * Add a script to cleanup the file listing before sending to Coveralls (Tim Wojtulewicz) GCC has a "feature" where it doesn't realize that files in the src/zeek/... tree are the same as files in the normal 'src/' tree. This leads the coverage script to send duplicates to Coveralls and pollute the display with them. The new script scrapes the intermediate output from lcov and de-duplicates anything in src/zeek/ to be just from src/. 4.1.0-dev.216 | 2021-02-08 13:06:07 -0800 * Allow --parse-only to work with --usage-issues flag (Jon Siwek, Corelight) Use of --parse-only would previously exit before --usage-issues had a chance to analyze scripts and report any discovered issues and it can be useful to combine both flags for checking scripts for mistakes without actually executing any code. This also improves the behavior of --parse-only in combination with reporting problems in signature files (previously, it exited before even reading them). 4.1.0-dev.212 | 2021-02-05 10:56:23 -0800 * Support for Zeek script "Reaching Definitions" (RD) analysis (Vern Paxson, Corelight) RD analysis invovles tracking the extent to which a given variable definition (assignment) can be visible elsewhere in the Zeek script. The analysis works on local variables in function/event/hook bodies, but not across them. The code tracks two forms of RDs, "minimal" (what's guaranteed to reach a given point in a function body) and "maximal" (what possibly could reach). Upcoming script optimization will use the latter, but the former currently allows identification of places for which a value is used where it does not appear that it will necessarily be defined. Specifying the ``zeek -u`` option will generate warnings for instances where this holds for local variables. Specifying ``zeek -uu`` turns on additional expensive analysis to report instances where record fields might be used without having previously been set. The base scripts have some places where static analysis lacks sufficient power to tell that values are being used safely (guaranteed to have been identified). In order to enable users to employ ``zeek -u`` on their own scripts without being distracted by these instances, this change also includes a new attribute, ``&is_assigned``, which can be associated with a variable or a record field to inform Zeek's analysis that the script writer asserts the value will be set, suppressing the associated warnings. For the test suite, there's new ``btest -a usage`` alternative to exercise both ``zeek -uu`` and ``&is_assigned`` functionality. 4.1.0-dev.173 | 2021-02-03 11:16:04 -0800 * GH-1389: Add VN-Tag packet analyzer that skips VN-Tag headers (Tim Wojtulewicz, Corelight) 4.1.0-dev.171 | 2021-02-03 09:53:12 -0700 * Prefix #includes of .bif.h files with zeek/ This enables locating the headers within the install-tree using the dirs provided by `zeek-config --include_dir`. To enable locating these headers within the build-tree, this change also creates a 'build/src/include/zeek -> ..' symlink. (Jon Siwek, Corelight) 4.1.0-dev.169 | 2021-02-03 09:50:33 -0700 * Bump macOS jobs in CI from 4 to 12 cores A recent Cirrus upgrade enables this and saves 4-5 minutes for the build and test jobs (combined), for both Catalina and Big Sur. (Christian Kreibich, Corelight) 4.1.0-dev.165 | 2021-02-01 15:53:56 -0800 * Update documentation to include "Book of Zeek" revisions (Jon Siwek, Corelight) 4.1.0-dev.164 | 2021-02-01 12:13:38 -0800 * Deprecate zeekenv() and use getenv() directly (Jon Siwek, Corelight) 4.1.0-dev.162 | 2021-01-29 21:05:41 -0800 * GH-1377: Add missing zeek/ to header includes (Jon Siwek, Corelight) 4.1.0-dev.158 | 2021-01-29 16:37:56 -0800 * Fix language.init-in-anon-function btest due to changes to log filter predicates (Tim Wojtulewicz, Corelight) * Remove deprecated log filter predicates for 4.1 (Christian Kreibich, Corelight) Update the logging framework tests: since hooks operate by name, they cannot be anonymous. I'm also dropping the &optional attribute from the status field, since here know that the values are actually defined, and access to an optional status field should normally be guarded by the existence test operator. * Remove deprecated Plugin::HookCallFunction (Tim Wojtulewicz, Corelight) * Remove deprecated support for .bro script extension and BRO_ environment variables (Tim Wojtulewicz, Corelight) * Remove deprecated ICMP events (Tim Wojtulewicz, Corelight) * Remove some deprected methods/events from bif files (Tim Wojtulewicz, Corelight) * Remove deprecated TYPE_COUNTER (Tim Wojtulewicz, Corelight) * Remove all of the random single-file deprecations (Tim Wojtulewicz, Corelight) * Remove all fully-deprecated files (Tim Wojtulewicz, Corelight) 4.1.0-dev.146 | 2021-01-29 10:20:28 -0800 * Skip negative integers in input framework where not type-permissible (Christian Kreibich, Corelight) Script-layer counts, when provided as negative integers in an input file, got cast to unsigned values because strtoull() does not complain about negative values. For example, input string "-1" would lead to value 18446744073709551615 (an all-ones 64-bit int) on x86_64. This is more likely to be an error than an intent to get very large, platform-dependent values, so these input lines are now skipped with according messaging in the reporter.log/stderr. This also affected ports: -1/tcp got cast to unsigned and only thrown out because PortVal rejects values > 65535, mapping them to 0. We now skip such inputs as well. 4.1.0-dev.144 | 2021-01-29 09:54:01 -0700 * Fix various broken links in script documentation (Jon Siwek, Corelight) 4.1.0-dev.141 | 2021-01-25 12:32:48 -0800 * Update broker submodule, embedded CAF to 0.18 (Jon Siwek, Corelight) 4.1.0-dev.137 | 2021-01-21 17:29:40 -0800 * Update SSL consts (Johanna Amann) There have been a bunch of mostly TLS 1.3 related changes. 4.1.0-dev.135 | 2021-01-21 16:57:01 -0800 * GH-1352: Added flag to stop processing SMTP headers in attached messages (Jon Oakley) Previously, header values from sub-entities could overwrite the top-level MIME entity values, causing the SMTP log to contain misleading information. 4.1.0-dev.132 | 2021-01-21 13:59:35 -0800 * GH-1296: fix type-checks related to list-type equality (Jon Siwek, Corelight) List-types as used in composite table/set indices, for example, previously had incorrect same_type() comparisons due to flattening of the list-type into a single type without checking whether the number and kind of types all match. This patch simply removes the flatten_type() call from same_type() since it was already contradicting/preventing a subsequent full-comparison between elements of two TYPE_LISTs. There was also a superfluous special-case of the `in` operator's type-checking for testing whether a record is in a table/set. It's superfluous because the general case will already do the type-checking from MatchesIndex() after first wrapping the record operand in a ListExpr. The previous logic was incorrectly relying on the flatten_type() for testing equality of a record-type against a list-type, whereas the general case correctly normalizes to testing equality of two list-types. The special-cased type-checking logic for assigning a record value to a table index during its initialization similarly needed minor re-organization in order to maintain the same error messages as before. * Guard against nullptr deref after AST-reduction consistency checks (Jon Siwek, Corelight) 4.1.0-dev.128 | 2021-01-21 13:37:10 -0800 * Install BTest with Zeek (Christian Kreibich, Corelight) Configuring with --disable-btest skips the installation. The main reason for doing this is that many Zeek packages come with testsuites that nearly always require btest, so providing btest out of the box makes sense. Note that zkg's recent additions to its PATH environment variable management mean that this btest instance is automatically found also when the Zeek installation's binary folder isn't in the path. 4.1.0-dev.125 | 2021-01-21 11:27:14 +0000 * More precise type information in a config framework error message When an option's value is a reader-incompatible table or set, Zeek now renders the type as expressed in the script layer (e.g. "set[addr,addr]") as opposed to the internal type tag (which'd here be "table", including for sets). (Christian Kreibich, Corelight) * Explicitly don't support sets with multiple index types in input/config frameworks The input framework's Manager::IsCompatibleType() already rejected sets with multiple index types that aren't all the same (i.e. that are not pure). Pure ones (e.g. "set[addr,addr]") slipped through and could cause Zeek to segfault elsewhere in the config framework due to type comparison subtleties. Note that the ASCII reader can't read such sets anyway, so this method now rejects sets with any kind of index-type tuple. In the config framework, the script-level change handler has a risky conversion from any to set[bool], which can trigger segfaults when the underlying set's index is a type tuple. We now prevent this code path by ensuring it only applies to sets with a single index type. (Christian Kreibich, Corelight) * Fix logo/website link in README (Jon Siwek, Corelight) 4.1.0-dev.120 | 2021-01-19 21:08:11 -0800 * Update Cirrus CI naming for MacOS images (Christian Kreibich, Corelight) The docs on https://cirrus-ci.org/guide/macOS/ changed from "osx_instance" to "macos_instance", so let's reflect that. 4.1.0-dev.118 | 2021-01-18 19:14:30 -0700 * Remove superfluous nullptr checks in Val/RecordVal methods (Jon Siwek, Corelight) 4.1.0-dev.116 | 2021-01-14 19:29:58 -0800 * Implement standard-library-compatible iterators for Dictionary (Tim Wojtulewicz, Corelight) 4.1.0-dev.114 | 2021-01-14 15:00:48 -0800 * Add support for rewriting the ASTs for script functions in "reduced" form In reduced ASTs, expressions have only variables or constants for operands (no subexpressions - those are replaced by temporary variables). In addition, some expressions are transformed into alternatives (new specialized expressions, or in some cases new internal types of statements) to more directly encapsulate their operation. The functionality is enabled by using -O xform. You can dump the before-and-after ASTs using -O dump-xform (very large output!). The new command-line option --optimize-only=X restricts the processing to only the function named X, and also automatically turns on dump-xform. * Add new btest alternative for script transformation (xform) (Vern Paxson, Corelight) * split bifs.string_utils into a non-error test and an only-errors test (Vern Paxson, Corelight) To help control for differing error propagation * Add convenience function, obj_desc(), for accessing object descriptions (Vern Paxson, Corelight) * removed unused EventBodyList subclass (Vern Paxson, Corelight) 4.1.0-dev.82 | 2021-01-14 13:27:23 -0700 * Use static_cast instead of dynamic_cast for Val conversions (Tim Wojtulewicz) * Remove GET_FIELD_AS macro, replace with template methods (Tim Wojtulewicz, Corelight) * Add templated As() method to Val, use in various places we were using dynamic_cast (Tim Wojtulewicz, Corelight) * Move ValImpl classes into detail namespace (Tim Wojtulewicz) * Some review cleanup in Val.cc (Tim Wojtulewicz, Corelight) * Changes addressing pending issues per PR discussion (Vern Paxson, Corelight) * Remove BroValUnion by hoisting underlying Val subclass values into subclasses (Vern Paxson, Corelight) 4.1.0-dev.74 | 2021-01-14 11:57:48 -0700 * Use xcode 11 on the Catalina CI image (Tim Wojtulewicz, Corelight) 4.1.0-dev.71 | 2021-01-11 18:03:25 -0800 * Add []-style variable-capture-list for Zeek lambda functions (Vern Paxson, Corelight) The previous behavior of automatically capturing references to variables outside a lambda's scope is now deprecated. An explicit capture list which also specifies the desired copy-semantics will be required when writing lambda functions that refer to local variables of an outer scope. Examples of the new capture-list syntax are described at https://docs.zeek.org/en/master/script-reference/types.html#type-function * nit: fixed some 0/1 values that should instead be false/true (Vern Paxson, Corelight) * factored some complexity of begin_func() into static functions for clarity (Vern Paxson, Corelight) * error propagation fix: don't complain about "unused" values that themselves are due to errors (Vern Paxson, Corelight) * corrected & reflowed some comments, plus a whitespace tweak (Vern Paxson, Corelight) 4.1.0-dev.52 | 2021-01-11 11:11:13 -0800 * Remove unusable/broken RocksDB code and options (Jon Siwek, Corelight) The Broker RockSDB data store backend was previously unusable and broken, so all code and options related to it are now removed. 4.1.0-dev.51 | 2021-01-07 17:02:28 -0800 * Virtualize Obj::GetLocationInfo() (Vern Paxson, Corelight) Cleaner approach for localizing errors associated with duplicated ASTs. * Add support for inlining of Zeek script functions (Vern Paxson, Corelight) * Add support for duplicating Zeek ASTS (Vern Paxson, Corelight) * Update COPYING to 2021 (Johanna Amann, Corelight) 4.1.0-dev.27 | 2021-01-06 20:42:35 -0800 * GH-1347: Update cmake module to fix ZeekPluginDynamic's find_package(CAF) (Jon Siwek, Corelight) 4.1.0-dev.25 | 2020-12-28 15:37:20 -0700 * Add LGTM config file (Jon Siwek, Corelight) 4.1.0-dev.23 | 2020-12-23 10:36:22 -0800 * Fix subtle race on data store initialization (Dominik Charousset, Corelight) 4.1.0-dev.19 | 2020-12-22 10:58:13 -0800 * Add new TLS 1.3 constants. (Johanna Amann) * Fix TLS 1.3 session resumption detection. (Johanna Amann) Now we detect TLS 1.3 session resumption by looking if both sides have the PSK extension set, which is much more exact than the previous approach. * Introduce ssl_probable_encrypted_handshake_message event (Johanna Amann) This event is raised for messages that (probably) are TLS 1.3 handshake messages, including finished. This allows scripts to examine handshake messages without having to handle all encrypted messages. * SSL Analyzer: ignore CCS for TLS 1.3 (Johanna Amann) In TLS 1.3, the ChangeCipherSpec message is meaningless; it only is included to convince middleboxes that the devices actually are speaking TLS 1.2. Nowadays some TLS 1.3 implementations also just don't send the packet. In a push to unify our handling of TLS 1.3 connections - ignore CCS and always go with application data packet counting to determine if connections are or are not encrypted. * GH-1323: TLS analyzer: change logic to track TLS 1.3 connection establishment (Johanna Amann) This commit changes the logic that is used to tracks connection establishment - and moves it from scriptland into the core. TLS 1.3 connection establishment is much more finnicky for us than the establishment of earlier versions - since we cannot rely on the CCS message anymore (which is meaningless and not sent in a lot of cases). With this commit, the ssl_encrypted_data message gets raised for encrypted TLS 1.3 handshake messages - which is much more correct than the behavior before that just interpreted them as plaintext messages. 4.1.0-dev.12 | 2020-12-21 10:55:13 -0800 * Add MacOS Big Sur to CI (Christian Kreibich, Corelight) 4.1.0-dev.10 | 2020-12-17 16:11:10 -0800 * Remove explicit use of config mode for finding CAF (Dominik Charousset, Corelight) 4.1.0-dev.7 | 2020-12-16 15:10:04 -0800 * logging/ascii: Atomically create .shadow files using rename() (Arne Welzel, Corelight) A logger process being terminated/killed while in the process of creating a new .shadow file may leave an empty (invalid) one around. This in turn causes the logger to error and exit during startup. 4.1.0-dev.4 | 2020-12-15 08:00:44 -0800 * Add test case to cover weird EDNS ECS parsing situations (Jon Siwek, Corelight) * Fix EDNS ECS option parsing bugs (Jon Siwek, Corelight) * The parsing of IPv6 addresses tried to fill a stack-buffer with as much data as supplied in the Option even if it was in excess of the desired prefix or maximum IPv6 address size. This could result in an overflow of that stack-buffer. * The parsing of IPv4 addresses would overwrite the storage used for that address as many times as there were bytes in the Option in excess of the desired prefix length or maximum IPv4 address size. This could cause the resulting IPv4 address to be derived from the incorrect data. * Upon encountering unexpected/excessive option-length or source-prefix parameters, the data pointer used for parsing was also not always advanced to the start of the next alleged option's data. Assuming all other parsing code correctly guards against invalid input, there's no further harm from that other than the subsequent parsing being more likely to encounter unexpected values and emitting more Weirds. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=28336 (Link to details becomes public 30 days after patch release) 4.1.0-dev.1 | 2020-12-14 22:27:57 -0800 * Fix local.zeek compatibility test (Jon Siwek, Corelight) 3.3.0-dev.664 | 2020-12-14 21:23:01 -0800 * Update submodules (Jon Siwek, Corelight) 3.3.0-dev.663 | 2020-12-14 14:27:45 -0800 * Rename a 'do_net_run' variable to 'do_run_loop' For clarity, since the net_run() function was renamed to run_loop(). (Jon Siwek, Corelight) * GH-1329: call Zeek's cleanup function from standalone fuzzer driver (Jon Siwek, Corelight) Otherwise, the global Broker manager object containing CAF/threading logic is never destructed and can result in a heap-use-after-free if it tries to access other global objects after they're cleaned up from __cxa_finalize(). 3.3.0-dev.660 | 2020-12-14 10:55:15 -0800 * Fix typo in table iterator invalidation test comment (Tim Wojtulewicz, Corelight) * GH-1328: Improve behavior of Dictionary iterator invalidation (Jon Siwek, Corelight) Previously, an assertion was triggered in debug builds upon any attempt to insert or remove a Dictionary entry while any iteration of that Dictionary is underway and also even in cases where Dictionary membership was not actually modified (and thus invalidates a loop). Now, it emits run-time warnings regardless of build-type and only when insert/remove operations truly change the Dictionary membership. In the context of a Zeek script causing an invalidation, the warning message also now helps pinpoint the exact expression that causes it. * Add a note about aligning .cirrus.yml to platform support matrix wiki page (Johanna Amann, Corelight) 3.3.0-dev.650 | 2020-12-12 20:24:19 -0800 * Install zkg as part of the Zeek distribution. (Christian Kreibich, Corelight) - Add auxil/package-manager submodule as an optional build source. When the submodule is present, zkg gets installed into the Zeek installation's bin directory, its config file into etc/zkg, and its state into var/lib/zkg. Like zeekctl, zkg finds its own module independently of any PYTHONPATH. Installation via pip remains supported. You can skip zkg explicitly via --disable-zkg. See the NEWS update for details. - Establish a "zeek/python" subdirectory under libdir as the common place for Python modules in the Zeek distribution. This now separates out the Broker Python bindings, ZeekControl, and zkg's Python module. - Add configure flags to allow customizing this Python folder, in three ways: --python-dir, --python-prefix, and --python-home. These differ in the logic they automatically add to the path, and build on the logic already used in Broker. - Include a (commented-out) @load for zkg's packages folder in local.zeek. 3.3.0-dev.648 | 2020-12-12 13:39:28 -0800 * Update Coverity Scan GitHub Action (Jon Siwek, Corelight) * Remove old Python 2 vs. 3 workaround * Build CAF separately to prevent internal errors in Coverity tools 3.3.0-dev.645 | 2020-12-10 14:14:12 -0800 * Fix message ordering of Broker messages (Dominik Charousset, Corelight) Using two separate Broker subscribers for status events and regular messages introduces a race on the two objects. Even if Broker sends all messages in a particular (deterministic) order, Zeek may still process them in a different order as a result. Since several tests rely on a strict ordering of Broker events, these tests could fail sporadically. Using only a single subscriber for all Broker messages makes sure that Zeek observes all messages in the same order as Broker emits them. * Fix UB in shutdown of Broker manager (Dominik Charousset, Corelight) * Migrate to CAF 0.18 (Dominik Charousset, Corelight) 3.3.0-dev.638 | 2020-12-09 17:29:03 -0800 * Update Mozilla Root Store (Johanna Amann, Corelight) 3.3.0-dev.634 | 2020-12-09 10:51:49 -0800 * GH-1321: Prevent compounding of `connection_status_update` event timers (Jon Siwek, Corelight) Particularly for ICMP connections, a new timer got added every time a `connection` record was updated even if there was still a pending timer for that connection. * Update Debug.cc to use setsignal() wrapper (Jon Siwek, Corelight) So as to prefer sigaction() over either sigset() or signal(), which are less favorable due to underspecification issues and some treating them as obsolete/deprecated. 3.3.0-dev.630 | 2020-12-07 16:52:52 -0800 * Fix narrowing conversion compiler error in SMB error-response logic (Jon Siwek, Corelight) 3.3.0-dev.629 | 2020-12-07 16:03:25 -0800 * Add tests for new SMB3 multichannel support (Vlad Grigorescu) * Fix SMB2 response status parsing. Fixes #1286 (Vlad Grigorescu) 3.3.0-dev.624 | 2020-12-07 18:17:34 +0000 * Switch test baselines to canonified baselines All baselines are now using the new btest feature to save baselines in the canonified form (after the canonifiers are run). This should, hopefully, lead to less changes in the future - that are easier to diff. In addition, this cleans up a few small problems of the canonifiers which were over-eager in identifying some numerical and path values. We also use the new btest-diff --binary flag to identify binary files on which no canonifier (not even the default canonifier) is applied. (Christian Kreibich, Corelight with tiny additions by Johanna Amann, Corelight) 3.3.0-dev.611 | 2020-12-07 09:42:37 -0800 * Fix misassociation of Zeekygen-style comments within function scopes (Jon Siwek, Corelight) All Zeekygen-style comments relate to entities at global scope, so those found within functions are now ignored instead of misassociated. 3.3.0-dev.609 | 2020-12-04 19:20:39 -0800 * Remove an extraneous Zeekygen-style comment (Jon Siwek, Corelight) 3.3.0-dev.607 | 2020-12-04 11:16:09 -0800 * Fix the CMake 'dist' target of Zeek plugins to only run when outdated (Benjamin Bannier, Corelight) 3.3.0-dev.604 | 2020-12-04 18:40:03 +0000 * Sumstats: allow users to manage epoch manually This change allows users to specify an epoch length of 0, which means that the user manually has to finish the epochs. A new next_epoch function is introduced to allow users to manually end epochs. Addresses GH-348 (Johanna Amann, Corelight) * Sumstats: epoch_finished was not called under certain circumstances In non-clustered mode, epoch_finished was not called when there was no data during the epoch. This behavior does not fit the documentation, and also is different in cluster-mode, where epoch_finished is, indeed, called after every epoch. This small change fixes this behavior. (Johanna Amann, Corelight) 3.3.0-dev.600 | 2020-12-03 18:02:22 -0800 * Add a CI task for compiling with static broker/binpac (Johanna Amann, Corelight) 3.3.0-dev.596 | 2020-12-03 09:35:42 -0700 * Fix a couple of life-time issues when plugin loading fails. Reported by Coverity. Follow-up to #1179. (Robin Sommer, Corelight) 3.3.0-dev.593 | 2020-12-02 12:53:04 -0800 * Add `count_to_double` and `int_to_double` bif functions (Yacin Nadji, Corelight) 3.3.0-dev.590 | 2020-12-02 11:11:26 -0800 * Update minimum required CMake to 3.5 (Jon Siwek, Corelight) Also now uses CMake's ENABLE_EXPORTS target property for the zeek executable to ensure symbols are visible to plugins. Prior to CMake 3.4, the policy was to export symbols by default for certain platforms, but later versions need either the explicit target property or policy. 3.3.0-dev.587 | 2020-12-01 10:17:42 -0700 * GH-1184: Add 'source' field to weird log denoting where the weird was reported (Tim Wojtulewicz, Corelight) 3.3.0-dev.585 | 2020-12-01 14:42:54 +0000 * Retry loading plugins on failure to resolve to dependencies. Closes #1179. (Robin Sommer, Corelight) 3.3.0-dev.580 | 2020-11-30 14:07:39 -0700 * Find correct zeek namespace in debug logger macros. These macros forward to functionality in `zeek::detail::debug_logger` and are not intended for customization. This patch fixes the macros to always use `::zeek::detail::debug_logger` as without the leading `::` lookup could happen in any potentially local namespace `zeek` which does not need to provide this symbol. This closes zeek/spicy#597. (Benjamin Bannier, Corelight) 3.3.0-dev.576 | 2020-11-26 18:16:07 +0000 * Remove Python2 compatibility logic. We now require at least Python 3.5. This includes script changes, improves the cmake logic to find python3, makes scripts explicitly call python3 and documentation updates. (Jon Siwek, Corelight) * Remove Fedora 31 (EOL) from CI (Jon Siwek, Corelight) 3.3.0-dev.564 | 2020-11-24 15:23:50 -0800 * Improve support for custom libdir locations (Christian Kreibich, Corelight) - Remove hardwiring of $ZEEK_ROOT/lib throughout the three and defaults the name of Zeek's library directory to the default on the given platform (e.g. lib64), via GNUInstallDirs. - Consistently use that lib directory, instead of two lib folders resulting when using a custom libdir. - Remove the old lib directory in the installation prefix, if one exists - Add --lib_dir to zeek-config (and sort its options a bit). 3.3.0-dev.561 | 2020-11-23 21:50:19 -0800 * Move implementation of internal_{type,var,etc} methods back into global namespace. (Tim Wojtulewicz, Corelight) This fixes an unknown symbol error if using those methods. They're defined as extern in the global namespace in Var.h, but Var.cc had their implementations defined in the zeek::detail namespace. 3.3.0-dev.559 | 2020-11-23 21:39:29 -0800 * Simplify Debian/Ubuntu CI dependencies and setup (Dominik Charousset, Corelight) * Update .gitignore to ignore pyenv .python-version (Otto Fowler) 3.3.0-dev.554 | 2020-11-19 18:09:01 -0800 * Reverts the SMTP regex change in dead3226a545e264072ced40284f86ac41528ba8. (Tim Wojtulewicz, Corelight) The regex change broke some of the external tests. I added some more cases to the regular email btest to hopefully cover all of the cases better. 3.3.0-dev.551 | 2020-11-17 15:01:04 -0700 * Added unit tests for regex fix (christina23) * Improved regex for SMTP parsing (christina23) 3.3.0-dev.548 | 2020-11-17 13:32:57 -0800 * GH-1033: BrokerStore <-> Zeek Tables: support complex indices (Johanna Amann, Corelight) This change adds support for complex indexes for sets and tables. With this change, sets with indexes like: set[string, count, count] will function. Before this change, Zeek raised an error message in these cases. 3.3.0-dev.546 | 2020-11-17 11:50:47 +0000 * Fix memory leak in deprecated Analyzer::ConnectionEvent() (Jon Siwek, Corelight) 3.3.0-dev.543 | 2020-11-16 11:50:49 -0800 * GH-352: Improve HTTP::match_sql_injection_uri regex (Jon Siwek, Corelight) Changes \x00-\x37 ranges to \x00-\x1f with assumption that the former was attempting to match ASCII control characters, but mistook an octal range for hex. This change reduces some false positives. 3.3.0-dev.541 | 2020-11-16 11:22:00 -0800 * Simplify ssh/main.zeek by using "ssh_server_host_key" for fingerprinting (Jon Siwek, Corelight) * Deprecate "ssh1_server_host_key" parameters *e* and *p* (Jon Siwek, Corelight) They are named such that *e* is actually the modulus, not the exponent. The replacement parameters are named *exponent* and *modulus* for clarity. * GH-1264: Implement "ssh_server_host_key" event (Jon Siwek, Corelight) This event provides host key fingerprints for both SSH1 and SSH2. 3.3.0-dev.537 | 2020-11-16 11:03:05 +0000 * [SSH] Handle SSH version 1.99 SSH can set in its identification a version 1.99 (SSH-1.99-xxx). That means the client/server is compatible with SSHv1 and SSHv2. So the version choice depends of the both side. 1.99 : 1.99 => 2.0 1.99 : 1.x => 1.x 1.99 : 2.0 => 2.O (see "Compatibility With Old SSH Versions" in RFC 4253) (Brevet Vivien) 3.3.0-dev.534 | 2020-11-12 14:31:10 -0800 * Move UnknownProtocol options to init-bare.zeek (Jon Siwek, Corelight) Otherwise the `unknown_protocol` event cannot be used independently from `policy/mic/unknown-protocols.zeek`. * Coverity 1436183: Initialize packet_analysis::Manager fields (Jon Siwek, Corelight) * GH-1273: Change SizeExpr to yield "any" type when operating on "any" (Jon Siwek, Corelight) * Add enum_names() BIF to return names of an enum type's values (Jon Siwek, Corelight) * Add type_aliases() BIF for introspecting type-names of types/values (Jon Siwek, Corelight) * Change Type::type_aliases map to store IntrusivePtr (Jon Siwek, Corelight) And deprecate Type::GetAliases() and Type::AddAlias() since they took raw pointers. Now replaced with Type::Aliases() and Type::RegisterAlias(). * Fix lookup_ID() BIF to return enum values (Jon Siwek, Corelight) Looking up an enum value from a string equal to its name previously returned "". 3.3.0-dev.524 | 2020-11-12 12:16:00 -0700 * Move 'using namespace' declaration after other includes (Tim Wojtulewicz, Corelight) * Use binpac.h as if it was an external dependency (Tim Wojtulewicz, Corelight) * Fix includes of bif.h and _pac.h files to use full paths inside build directory (Tim Wojtulewicz, Corelight) * Remove unnecessary include of NetVar.h from packet analysis plugins (Tim Wojtulewicz, Corelight) * GH-1079: Use full paths starting with zeek/ when including files (Tim Wojtulewicz, Corelight) * Fix type clash fatal error with log filters that use $ext_func and $include/$exclude The logging manager's Manager::TraverseRecord(), called when adding a log filter to a stream, skipped any fields intoduced by a filter's $ext_func when such fields weren't mentioned in a $include restriction or mentioned in an $exclude restriction. This was inconsistent with Manager::RecordToFilterVals, used when actually writing log entries, which does include those values. The result was that the record indices descent in Manager::RecordToFilterVals expects to find only record values, when in fact only the record provided by ext_func is present. This leads to type mismatches and hard Zeek exits like this one: 1300475173.475401 fatal error in zeek/share/zeek//base/init-bare.zeek, line 4810: Val::CONVERTER (string/record) (zeek) The fix makes ext_func's field additions decisive, meaning the filter's include/exclude lists don't apply to it. If a user really wants to override this, they can reset the filter's ext_func back to our no-op default. The included btest produces the above error when the fix is not present. (Christian Kreibich, Corelight) 3.3.0-dev.516 | 2020-11-11 12:53:02 -0700 * Support for additional DNS RR Type: LOC[29], SSHFP[44], NSEC3PARAM[51], custom BIND9 signaling[65534] (Fatema BW) 3.3.0-dev.514 | 2020-11-10 14:05:51 -0700 * GH-1269: Fix LogAscii::enable_leftover_log_rotation crash in bad dirs Running with that option enabled inside a bad directory (e.g. lack of permissions) crashed due to not checking for failure of opendir(). (Jon Siwek, Corelight) 3.3.0-dev.512 | 2020-11-10 13:55:33 -0700 * Change Debian 10 CI config to use GCC (Jon Siwek, Corelight) * Update external testing repo hashes for canonification updates (Tim Wojtulewicz, Corelight) 3.3.0-dev.509 | 2020-11-10 08:41:54 -0700 * GH-1221: Add unknown_protocols.log for logging packet analyzer lookup failures (Tim Wojtulewicz, Corelight) * Remove default_analyzer for Ethernet packet analzyer (Tim Wojtulewicz, Corelight) 3.3.0-dev.506 | 2020-11-09 16:27:04 -0800 * Add CI support for Debian 10, Fedora 31 & 33, FreeBSD 11, Ubuntu 20.04 (Christian Kreibich, Corelight) This also adds two environment variables to the Dockerfiles for the Debian/Ubuntu flavors to avoid stalling on user input when configuring tzdata. 3.3.0-dev.504 | 2020-11-09 10:56:09 -0700 * Remove now-unused Packet::l2_valid field (Tim Wojtulewicz, Corelight) * GH-1186: Remove Packet::hdr_size and uses of it. This change also removes Packet::IP(), since Packet now contains an ip_hdr member that points at the IP header if it exists. (Tim Wojtulewicz, Corelight) 3.3.0-dev.501 | 2020-11-04 10:41:56 -0800 * GH-1252: rename files with colons for Windows compatibility (Jon Siwek, Corelight) A short-term measure so that the Zeek source tree can simply exist on a Windows filesystem. For true support, the logic that decides/generates the filename format will need to change. * Use std::function instead of a function pointer in packet_analysis::Component (Tim Wojtulewicz, Corelight) 3.3.0-dev.495 | 2020-11-02 12:03:52 -0700 * GH-1215: Remove dispatch_map from packet analysis, replace with BIF methods for registering dispatches (Tim Wojtulewicz, Corelight) 3.3.0-dev.493 | 2020-11-02 09:51:03 -0800 * GH-1256: Write out strerror when writing errno during safe_write (Tim Wojtulewicz, Corelight) 3.3.0-dev.490 | 2020-10-27 13:41:27 -0700 * updated to mention notice_alarm.log by name. That has taken the place of the file formerly named alarm.log (Duffy O'Craven) 3.3.0-dev.488 | 2020-10-26 11:03:51 -0700 * Improve how Zeekygen generated record/enum redefinition docs It now provides a summary of the new fields/enums added by any given redefinition along with associated commentary. (Jon Siwek, Corelight) 3.3.0-dev.486 | 2020-10-26 10:41:48 -0700 * GH-1245: require TLD of hostname literals to start with a letter (Jon Siwek, Corelight) 3.3.0-dev.484 | 2020-10-23 14:47:44 -0700 * Add coveralls badge to README.md (Tim Wojtulewicz) * Remove the entire auxil directory from the results (Tim Wojtulewicz) * Prevent non-master builds, add cirrus information to upload (Tim Wojtulewicz) * Add support for sending data to coveralls with lcov_html.sh via Cirrus (Tim Wojtulewicz, Corelight) 3.3.0-dev.479 | 2020-10-23 12:25:15 -0700 * GH-251 (revert): remove coercion-to-signed-integer for |x| expressions (Jon Siwek, Corelight) For `|x|`, where `x` is an expression with an integral result, an implicit coercion of that result into signed `int` type no longer takes place. This was actually the behavior before Zeek 3.0 as well, but the attempt to prevent mistakes that easily result from integer literals in Zeek being unsigned like `|5 - 9|` causing an overflow/wraparound and yielding a very large number is not generally consistent since overflows are still generally able to happen in other ways and also in other contexts besides just absolute-values. So the preference was to revert to a behavior that favors consistency. For reference, see https://github.com/zeek/zeek/pull/251#issuecomment-713956976 3.3.0-dev.476 | 2020-10-22 15:59:56 -0400 * Add an option to ignore packets sourced from particular subnets. It's implemented with a new set[subnet] option named ignore_checksums_nets. If you populate this set with subnets, any packet with a src address within that set of subnets will not have it's checksum validated. (Seth Hall, Corelight) * Update submodule(s) [nomail] (Jon Siwek, Corelight) 3.3.0-dev.467 | 2020-10-21 11:06:18 -0700 * Fix a couple of Coverity findings (1433618, 1433619) (Tim Wojtulewicz, Corelight) 3.3.0-dev.465 | 2020-10-20 13:34:20 +0000 * Bump BTest to 0.64. 3.3.0-dev.464 | 2020-10-20 08:43:40 +0000 * Update BTest to 0.63. New test baselines will now be store canonified. (Robin Sommer, Corelight) 3.3.0-dev.463 | 2020-10-19 18:57:00 -0700 * Add length checks for IP header values before parsing TCP/UDP (Tim Wojtulewicz, Corelight) 3.3.0-dev.461 | 2020-10-19 11:54:23 -0700 * Fix deprecation warning in POP3 fuzzer (Jon Siwek, Corelight) * Add header length check to GRE packet analyzer (Tim Wojtulewicz, Corelight) 3.3.0-dev.456 | 2020-10-16 12:41:58 -0700 * Avoid passing null pointers to memcmp in ZeekString comparison methods (Jon Siwek, Corelight) 3.3.0-dev.454 | 2020-10-16 10:34:53 -0700 * Change ICMP Neighbor Discovery option length storage to a uint16 (Vlad Grigorescu) This fixes an overflow in the calculation of option lengths in ICMP Neighbor Discovery messages. 3.3.0-dev.451 | 2020-10-16 07:09:43 +0000 * Make event ordering deterministic NetControl::init and filter_change_tracking could basically be raised in random order. (Johanna Amann, Corelight) * Introduce generate_all_events bif and add option to misc/dump-events generate_all_events causes all events to be raised internally; this makes it possible for dump_events to really capture all events (and not just those that were handled). Addresses GH-169 (Johanna Amann, Corelight) 3.3.0-dev.444 | 2020-10-15 13:25:12 -0700 * Rework Sessions::Weird (Tim Wojtulewicz, Corelight) * Prevent String::Set from crashing if passed a nullptr (Tim Wojtulewicz, Corelight) * Store packet's ip header as unique_ptr (Tim Wojtulewicz, Corelight) * Remove some unused includes from Packet.h (Tim Wojtulewicz, Corelight) * Use shared_ptr for encapsulation data instead of raw pointer (Tim Wojtulewicz, Corelight) * Review cleanup (Tim Wojtulewicz, Corelight) * Update external testing repo hashes (Tim Wojtulewicz, Corelight) * Reorder Packet member variables slightly for better packing (Tim Wojtulewicz, Corelight) * Store the ip header in the packet after processing, reuse other places (Tim Wojtulewicz, Corelight) * Change to store data in packet directly instead of keystore (Tim Wojtulewicz, Corelight) * Don't always insert data into keystore for tunnels (Tim Wojtulewicz, Corelight) * Add comment about packet header size and session analysis (Tim Wojtulewicz, Corelight) * Set data to ip header's payload instead of advancing the pointer (Tim Wojtulewicz, Corelight) * Move packet dumping to packet_mgr (Tim Wojtulewicz, Corelight) * Make Sessions::NextPacket call packet_mgr, fix fuzzer code to do the same (Tim Wojtulewicz, Corelight) * Move IP and IP tunnel code from Sessions into packet analyzers (Tim Wojtulewicz, Corelight) * Prep work for IP changes - Move all of the time handling code out of PktSrc into RunState - Call packet_mgr->ProcessPacket() from various places to setup layer 2 data in packets (Tim Wojtulewicz, Corelight) 3.3.0-dev.426 | 2020-10-14 10:47:37 -0700 * GH-1211: Improve error message for already-defined functions (Jon Siwek, Corelight) 3.3.0-dev.422 | 2020-10-13 16:26:24 -0700 * GH-1208: Use Dictionary validity assertions only during CI (Jon Siwek, Corelight) 3.3.0-dev.420 | 2020-10-13 15:34:02 -0700 * Change "Cluster::Node$p" field to use "&default=0/unknown" (Seth Hall, Corelight) This effectively makes defining the listening port optional in the cluster node layout config, with "0/unknown" meaning the node is not pre-configured to listen for incoming connections from other cluster nodes. 3.3.0-dev.414 | 2020-10-13 13:49:05 -0700 * Update cmake submodule to pull in fix for building libkqueue (Tim Wojtulewicz, Corelight) * GH-1063: Update libkqueue to fix pf_ring-zc failures (Tim Wojtulewicz, Corelight) 3.3.0-dev.411 | 2020-10-13 13:38:38 -0700 * Remove unused LoginConn type and variable in Conn.h (Tim Wojtulewicz, Corelight) 3.3.0-dev.409 | 2020-10-13 13:32:18 -0700 * Make it possible to pass command line options through to scripts. (Seth Hall, Corelight) A new ``zeek_script_args`` variable contains a list of arguments passed to a script. E.g. either when explicitly executing Zeek like ``zeek -- myscript.zeek -arg1 -arg2``, or when using Zeek to interpret executable scripts that contain a hashbang line at the top like:: #!/usr/local/zeek/bin/zeek -- 3.3.0-dev.403 | 2020-10-13 10:50:12 -0700 * Add new Pcap::findalldevs() BIF (Seth Hall, Corelight) * Remove superfluous RuleCondition destructors (Jon Siwek, Corelight) * Silence Clang's warning about ignoring GCC's maybe-uninitialized warning (Jon Siwek, Corelight) * Add reference to network_time_init from zeek_init docs (Jon Siwek, Corelight) 3.3.0-dev.390 | 2020-10-12 17:43:15 -0700 * Improve documentation for zeek_init event scheduling pitfalls (Jon Siwek, Corelight) 3.3.0-dev.388 | 2020-10-12 17:02:20 -0700 * Add CaptureLoss::Too_Little_Traffic notice (Vlad Grigorescu) * Add CaptureLoss::initial_watch_interval for a quick read on cluster health after startup. (Vlad Grigorescu) * Improve capture-loss.zeek documentation. (Vlad Grigorescu) * Fix whitespace in capture-loss.zek (Vlad Grigorescu) 3.3.0-dev.381 | 2020-10-12 11:15:29 -0700 * GH-779: Add "udp-state" signature condition (Jon Siwek, Corelight) It accepts "originator" or "responder" states as a way to enforce that the signature only matches packets in the associated direction. The "established" state is rejected as an error since it doesn't have a useful meaning like it does for the "tcp-state" condition. * Rename RuleConditionTCPState::TCPState enum values (Jon Siwek, Corelight) * Rename "tcp-state" signature parser tokens to not be TCP-specific (Jon Siwek, Corelight) 3.3.0-dev.377 | 2020-10-08 12:07:42 -0700 * GH-1200: ignore a maybe-uninitialized warning (Jon Siwek, Corelight) 3.3.0-dev.374 | 2020-10-07 10:44:11 -0700 * Fix multipart MIME leak of sub-part found after closing-boundary (Jon Siwek, Corelight) After detecting a closing-boundary for a given multipart MIME entity, it enters into an "end of data" state, however any subsequent boundary delimiter could still cause the allocation of a sub-entity object that is never released due to cleanup logic being bypassed upon finding the "end of data" state already reached. This change prevents allocation/processing of sub-entities after the "end of data" state is reached (e.g. from detecting a multipart closing-boundary). This new behavior still aligns with RFC 2046 expectations: "There appears to be room for additional information prior to the first boundary delimiter line and following the final boundary delimiter line. These areas should generally be left blank, and implementations must ignore anything that appears before the first boundary delimiter line or after the last one." Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26027 (Link to details becomes public 30 days after patch release) * Btests for log filter policy hooks (Christian Kreibich, Corelight) * Btest baseline updates to reflect new logging policy hooks (Christian Kreibich, Corelight) * Migrate existing use of filter predicates to policy hooks (Christian Kreibich, Corelight) * Support for log filter policy hooks (Christian Kreibich, Corelight) This adds a "policy" hook into the logging framework's streams and filters to replace the existing log filter predicates. The hook signature is as follows: hook(rec: any, id: Log::ID, filter: Log::Filter); The logging manager invokes hooks on each log record. Hooks can veto log records via a break, and modify them if necessary. Log filters inherit the stream-level hook, but can override or remove the hook as needed. The distribution's existing log streams now come with pre-defined hooks that users can add handlers to. Their name is standardized as "log_policy" by convention, with additional suffixes when a module provides multiple streams. The following adds a handler to the Conn module's default log policy hook: hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter) { if ( some_veto_reason(rec) ) break; } By default, this handler will get invoked for any log filter associated with the Conn::LOG stream. The existing predicates are deprecated for removal in 4.1 but continue to work. 3.3.0-dev.365 | 2020-10-06 12:19:49 -0700 * GH-425: Avoid temporary value while coercing records already of the right type. (Robin Sommer, Corelight) The combination of this commit with the previous one now lets the examples in GH-425 all execute with the same performance. * GH-425: Optimize record constructor expression. (Robin Sommer, Corelight) We remove the inheritance from UnaryExpr because we know the type of the operand precisely and can skip a temporary when evaluating the expression. * Unify type comparisions for records. (Robin Sommer, Corelight) For records, same_type(r1, r2) would not check if the fields' attributes match as well. That seems like an oversight, and some callers of same_type() did indeed add that check on their end. This commit moves the check into same_type() itself. That generally doesn't seem make any differences except for a couple of places validating code, which we update a bit. That in turn leans to slightly different (better?) error messages for a couple of test cases. 3.3.0-dev.361 | 2020-10-06 10:13:37 -0700 * logging/ascii: Support leftover log rotation in non-supervisor setups (Arne Welzel, Corelight) Allow enabling leftover log rotation through LogAscii::enable_leftover_log_rotation and redef this for the logger node in a supervisor setup individually. 3.3.0-dev.359 | 2020-10-02 10:49:35 -0700 * GH-1180: Add external dependency include paths to `zeek-config --include_dir` (Jon Siwek, Corelight) This helps ensure successful compilation of a plugin that includes a Zeek header which includes an external dependency header even if the plugin itself doesn't necessarily depend on that external library. 3.3.0-dev.357 | 2020-10-02 10:26:11 -0700 * Switch one's complement checksum implementation (Jon Siwek, Corelight) Borrows the `in_cksum` code from tcpdump, which borrowed from FreeBSD. It handles unaligned data better and also unrolls the inner loop to process 16 two-byte values at a time versus 2 one-byte values at a time in the previous version. Generally measured as ~1.5x faster in a release build. The new API should generally be more amenable to any future optimization explorations since all relevant data blocks are available within a single call rather than spread across multiple. 3.3.0-dev.355 | 2020-09-29 17:08:44 -0700 * Improve broker/ssl_auth_failure.zeek test (Jon Siwek, Corelight) There was a race that previously may cause it to not truly test whether the connecting-process exited due to SSL authentication failure or just because the listening-process wasn't available yet (e.g. due to process scheduling variability). 3.3.0-dev.352 | 2020-09-25 15:09:29 -0700 * GH-160: change find-filtered-trace to ignore pcaps with any non-TCP traffic (Robin Sommer, Corelight) 3.3.0-dev.350 | 2020-09-25 14:37:58 -0700 * Add dce_rpc_request_stub and dce_rpc_response_stub events (Yacin Nadji, Corelight) * Fix namespace warning in fuzzer code (Tim Wojtulewicz, Corelight) 3.3.0-dev.346 | 2020-09-24 16:03:28 -0700 * Fix a Sphinx warning about misformatted packet analyzer comment (Jon Siwek, Corelight) * Add Zeekygen documentation support for packet analyzers (Jon Siwek, Corelight) * Move packet_mgr to the zeek namespace (Tim Wojtulewicz, Corelight) 3.3.0-dev.341 | 2020-09-24 08:16:45 -0700 * Fix negative-value-left-shift undefined behavior in patricia trie (Jon Siwek, Corelight) * Improve negation of ConstExpr (Jon Siwek, Corelight) * Instead of creating a NegExpr for negation of a literal/constant, a ConstExpr is now created directly. * For negation of integer literals, there's now an additional check for whether the integer would be outside the range of possible 'int' values. This can also help prevent the undefined behavior due to overflow as a result of trying to represent the minimum 'int' value of -9223372036854775808 as a literal in a script -- the unsigned value is cast to signed yielding INT64_MIN, then INT64_MIN is negated. * Avoid signed integer overflow when combining SMB header PID bits (Jon Siwek, Corelight) Such an overflow invokes undefined behavior. * Avoid unary negation of INT64_MIN in modp_litoa10 (Jon Siwek, Corelight) Overlow can occur in that case, which is undefined behavior. * Avoid double-to-int conversion overflows in modp_dtoa functions (Jon Siwek, Corelight) Those methods already had a fallback to use sprintf() for large values except: * The check-for-large-value was unnecessarily done after many operations that aren't relevant to the check and those operations can result in a conversion overflow (undefined behavior). * The check-for-large-value was using the literal value for a 32-bit INT_MAX instead of just using INT_MAX. For a platform where `int` is less than 32-bits, the same conversion overflow from the previous point could still occur (undefined behavior). * The check-for-large-value was not inclusive of INT_MAX. In a case where the conversion of INT_MAX itself to a double can't be represented exactly, it's implementation-defined whether the closest higher or closest lower representable-value is selected. If the higher value is selected, then a `double` value comparing equal to INT_MAX-as-converted-to-double would cause an overflow of an `int` upon conversion (undefined behavior). * Fix divide-by-zero in Entropy analyzer (Jon Siwek, Corelight) * Fix divide-by-zero in stats/profiling memory usage calculation (Jon Siwek, Corelight) * Fix uninitialized field in POP3 fuzzer (Jon Siwek, Corelight) * Add framework for running UndefinedBehaviorSanitizer in CI (Jon Siwek, Corelight) 3.3.0-dev.329 | 2020-09-23 11:32:06 -0700 * Update NEWS (Tim Wojtulewicz, Corelight) * Fix build on FreeBSD, which was missing full definition of sockaddr for ARP (Tim Wojtulewicz, Corelight) * Update hashes for external testing repos (Tim Wojtulewicz, Corelight) * Fix handling of session analysis. This reverts commit 0e51be18ac59d7a176f18780f1f8df85561449b9. (Jan Grashoefer) * Fix Sessions::DoNextPacket to ignore ARP packets and not report weirds for them (Tim Wojtulewicz, Corelight) * Packet analysis cleanup. (Jan Grashoefer) * Minor cleanup (Tim Wojtulewicz, Corelight) * Simplify packet analyzer config. (Jan Grashoefer) * Make default packet analyzer definition explicit. (Jan Grashoefer) * Add explicit root analyzer for packet analysis. (Jan Grashoefer) * Extend packet analysis test. (Jan Grashoefer) * Simplify MPLS analysis. (Jan Grashoefer) * Improve packet analysis data flow. (Jan Grashoefer) * Improve packet analyzer API. (Jan Grashoefer) * Rename DefaultAnalyzer to IP. (Jan Grashoefer) * Move ARP analysis into packet analyzer. (Jan Grashoefer) * Small cleanup of packet analysis. (Jan Grashoefer) * Fix some build issues from the rebase (Tim Wojtulewicz, Corelight) * Remove encap_hdr_size (replaced by skip analyzer). (Jan Grashoefer) * Add skip analyzer test and update baselines. (Jan Grashoefer) * Allow to overwrite packet analysis mappings. (Jan Grashoefer) * Add SkipAnalyzer. This is WIP: The test case would require a new pcap or the possibility to overwrite analyzer mappings. The CustomEncapsulationSkip method and the corresponding options need to be removed. (Jan Grashoefer) * Suggested code improvements for packet analysis. (Jan Grashoefer) * Improve naming in packet analysis. (Jan Grashoefer) * Migrate all packet analyzers to new API. (Jan Grashoefer) * Remove magic identifiers from Ethernet analyzer. (Jan Grashoefer) * Bring back default packet analysis. Default analyzers can be configured per packet analyzer by omitting the identifier in the ConfigEntry. (Jan Grashoefer) * Further simplified the packet analysis API. This is still WIP and includes the following changes: * Dispatchers are now part of analyzers (moving dispatching logic from the manager to the analyzers) * All available analyzers are instantiated on start up * Removal of configuration class (Jan Grashoefer) * Move dispatching into packet analyzers. WIP that updates only the Ethernet analyzer. (Jan Grashoefer) * Move cur_pos from packet into packet manager loop. (Jan Grashoefer) * Remove packet_analysis/Defines.h - Replace uses of identifier_t with uint32_t - Replace repeated usage of tuple type for Analysis results with type alias (Tim Wojtulewicz, Corelight) * Move VectorDispatcher to be the only dispatcher (Tim Wojtulewicz, Corelight) * Remove Manager::Reset() method (Tim Wojtulewicz, Corelight) * Remove enabled state from Components, ability to enable/disable from Manager (Tim Wojtulewicz, Corelight) * Use shared_ptr instead of raw pointers in packet_analysis for analyzers and dispatchers (Tim Wojtulewicz, Corelight) * Whitespace fixes from review (Tim Wojtulewicz, Corelight) * Reorganize some pointer handling (Tim Wojtulewicz, Corelight) * Fix ordering of debug logger strings to match the enum (Tim Wojtulewicz, Corelight) * Fixes for various btest issues - Fix handling of truncated ethernet headers, fix core.truncation test output - Update commit hashes for external private test repo (Tim Wojtulewicz, Corelight) * Merge ProtocolAnalyzerSet into Manager, remove AnalyzerSet base class (Tim Wojtulewicz, Corelight) * Minor fixes for packet analyzer renaming. (Jan Grashoefer) * Renamed LL-Analyzers to Packet Analyzers. (Jan Grashoefer) * Initial implementation of Lower-Level analyzers (Peter Oettig) 3.3.0-dev.285 | 2020-09-22 16:01:51 -0700 * Improve fa_file/fa_metadata documentation (Duffy O'Craven, Corelight) 3.3.0-dev.284 | 2020-09-22 14:10:36 -0700 * Avoid race condition in control/id_value btest (Jon Siwek, Corelight) 3.3.0-dev.282 | 2020-09-22 11:08:56 -0700 * GH-927: Fix circumvention of evaluation order in 'when' conditions (Jon Siwek, Corelight) 3.3.0-dev.279 | 2020-09-21 15:22:45 -0700 * CI: Move to Fedora 32 (Christian Kreibich, Corelight) Fedora 30 is EOL. 3.3.0-dev.275 | 2020-09-17 13:39:46 -0700 * GH-1164: Fix incorrect RSTOS0 conn_state determinations The RSTOS0 `conn_state` label is documented as "Originator sent a SYN followed by a RST, never saw SYN-ACK from responder", but was previously applied to cases where no originator SYN exists, like a single RST-only packet. (Jon Siwek, Corelight) 3.3.0-dev.273 | 2020-09-17 13:11:44 -0700 * GH-1119: add base/protcols/conn/removal-hooks.zeek This adds two new functions: `Conn::register_removal_hook()` and `Conn::unregister_removal_hook()` for registering a hook function to be called back during `connection_state_remove`. The benefit of using hook callback approach is better scalability: the overhead of unrelated protocols having to dispatch no-op `connection_state_remove` handlers is avoided. (Jon Siwek, Corelight) * Avoid run-time cost of find-filtered-trace.zeek for live traffic (Jon Siwek, Corelight) * Remove connection_successful and successful_connection_remove events Related to https://github.com/zeek/zeek/issues/1119 (Jon Siwek, Corelight) 3.3.0-dev.269 | 2020-09-17 11:42:38 -0700 * GH-1155: Recursively check table index for unsupported types Previously, container types used within a table/set index were not deeply checked to ensure all constituents could be part of an index. (Jon Siwek, Corelight) * GH-1159: Fix vector-of-interval multiplication/division arithmetic Those operations done between a vector-of-interval and a vector-of-arithmetic-type previously threw a runtime expression error due to an incorrect coercion being used internally. (Jon Siwek, Corelight) 3.3.0-dev.265 | 2020-09-17 11:24:42 -0700 * Avoid passing address of member in packed struct #1074 This appeases -Waddress-of-packed-member warnings in some compiler/platform combinations via use of local variables. (Christian Kreibich, Corelight) 3.3.0-dev.262 | 2020-09-14 12:27:52 -0700 * Improve notice/suppression-cluster btest to avoid race potential (Jon Siwek, Corelight) 3.3.0-dev.260 | 2020-09-10 11:53:03 -0700 * Fix generate-docs Action to stage autogenerated script-reference files (Jon Siwek, Corelight) 3.3.0-dev.258 | 2020-09-09 10:46:37 -0700 * Improve Reporter weird-sampling-whitelist getters/setters (Jon Siwek, Corelight) - getter methods return const-ref - setter methods pass by value and std::move() - ranged-for loops over the whitelists access by const-ref * Fix memory leak caused by re-entering GTPv1 parsing (Jon Siwek, Corelight) If the inner packet of a decapsulated GTPv1 frame maps to the same Connection as the outer packet and also contains another GTPv1 encapsulation, the same GTPv1 analyzer attempts to parse the inner packet. In the process, the inner parsing method overwrites pointers to memory resources the outer parsing method has not yet released and results in a memory leak. * Fix memory leak caused by re-entering AYIYA parsing (Jon Siwek, Corelight) If the inner packet of a decapsulated AYIYA frame maps to the same Connection as the outer packet and also contains another AYIYA encapsulation, the same AYIYA analyzer attempts to parse the inner packet. In the process, the inner parsing method overwrites pointers to memory resources the outer parsing method has not yet released and results in a memory leak. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=25256 3.3.0-dev.254 | 2020-09-08 17:21:16 -0700 * GH-343: Change HTTP DPD signatures to trigger analyzer independent of peer state. (Robin Sommer, Corelight) This is to avoid missing large sessions where a single side exceeds the DPD buffer size. It comes with the trade-off that now the analyzer can be triggered by anybody controlling one of the endpoints (instead of both). 3.3.0-dev.252 | 2020-09-08 17:04:19 -0700 * GH-623: Extend weird sampling with option to track selected weirds globally. (Robin Sommer, Corelight) The new set "sampling_global_list" lists weirds to rate-limit globally instead of per connection/flow. 3.3.0-dev.249 | 2020-09-04 18:30:19 -0700 * Exclude installing "zeek -> ." include dir symlink (Jon Siwek, Corelight) The symlink only exists for use within the source-tree and isn't needed for the install-tree. 3.3.0-dev.247 | 2020-09-04 17:24:28 -0700 * GH-165: Fix global initializations that indirectly use builtin types (Jon Siwek, Corelight) * Improve how primary/top-level BIFs get initialized (Jon Siwek, Corelight) * GH-1122: Allow initializing globals with calls to subdir BIFs (Jon Siwek, Corelight) * GH-1122: Improve error for global record initialization exceptions (Jon Siwek, Corelight) 3.3.0-dev.241 | 2020-09-03 15:02:29 -0700 * Change a script comment to use "code-block" directive (Jon Siwek, Corelight) 3.3.0-dev.240 | 2020-09-02 20:51:13 -0700 * Improve CI benchmark script's error handling/messaging (Jon Siwek, Corelight) 3.3.0-dev.238 | 2020-09-02 11:39:25 -0700 * GH-955: Prohibit `fallthrough` in typecasting `switch` cases (Jon Siwek, Corelight) 3.3.0-dev.236 | 2020-09-01 22:33:05 -0700 * Fix/finalize the generate-docs Github Action (Jon Siwek, Corelight) 3.3.0-dev.230 | 2020-09-01 20:32:01 -0700 * Fix generate-docs action to not fail when there's no changes (Jon Siwek, Corelight) 3.3.0-dev.229 | 2020-09-01 19:49:36 -0700 * Update generate-docs action to use access token for submodule update (Jon Siwek, Corelight) 3.3.0-dev.228 | 2020-09-01 18:37:33 -0700 * Update generate-docs action to install Sphinx dependency (Jon Siwek, Corelight) 3.3.0-dev.227 | 2020-09-01 17:58:22 -0700 * Add missing configure step for doc-generation action (Jon Siwek, Corelight) 3.3.0-dev.226 | 2020-09-01 17:53:55 -0700 * Improve documentation for rfind_str() (Jon Siwek, Corelight) * GH-1149: Add GitHub Action to automate generation of zeek-docs (Jon Siwek, Corelight) 3.3.0-dev.224 | 2020-09-01 11:57:44 -0700 * Reformat strings.bif Zeekygen comments to fix Sphinx warnings (Jon Siwek, Corelight) 3.3.0-dev.221 | 2020-08-31 17:31:21 -0700 * GH-174: Treat ambiguous attribute duplication as an error (Tim Wojtulewicz, Corelight) For example, a &default=1 and a &default=2 attribute are not valid when used together, but two duplicate &log attributes together are acceptable although redundant. 3.3.0-dev.216 | 2020-08-31 14:57:57 -0700 * Simplify a broker btest (Jon Siwek, Corelight) 3.3.0-dev.215 | 2020-08-28 16:51:40 -0700 * GH-926: Improve type-checking for ternary conditional operator (Jon Siwek, Corelight) * Unbreak build on power architectures by fixing small cmake issue (Johanna Amann, Corelight) 3.3.0-dev.211 | 2020-08-27 18:00:55 -0700 * Potentially improve intel/remove-item-cluster btest (Jon Siwek, Corelight) 3.3.0-dev.210 | 2020-08-27 15:34:14 -0700 * GH-158: Ensure packet-filter framework is loaded if -f option is passed (Tim Wojtulewicz, Corelight) 3.3.0-dev.208 | 2020-08-27 11:12:22 -0700 * GH-1145: Fix a umask issue for processing-status file (Vlad Grigorescu) 3.3.0-dev.206 | 2020-08-26 11:20:58 -0700 * Rename bro-bif.h to zeek-bif.h (Tim Wojtulewicz, Corelight) * Fix uses of bro.pac to use zeek.pac (Tim Wojtulewicz, Corelight) * Rename the bro-related binpac files to zeek (Tim Wojtulewicz, Corelight) 3.3.0-dev.202 | 2020-08-26 10:59:59 -0700 * GH-1145: Have mkdir in ensure_dir respect umask. (Vlad Grigorescu) This also aligns with the mkdir BIF. 3.3.0-dev.200 | 2020-08-25 19:47:02 -0700 * Update plugin btests for namespace changes (Tim Wojtulewicz, Corelight) * Remove namespace qualifications where unnecessary. (Tim Wojtulewicz, Corelight) 3.3.0-dev.196 | 2020-08-25 17:04:12 -0700 * Fix closing timestamp of rotated log files in supervised-cluster mode (Jon Siwek, Corelight) 3.3.0-dev.195 | 2020-08-25 14:32:47 -0700 * Improve an Intel framework btest (Jon Siwek, Corelight) Relying on non-atomic `cp` operation may not cooperate well with the Input framework file re-reading logic. 3.3.0-dev.194 | 2020-08-24 19:20:29 -0700 * Mark Continuation.h and PacketDumper.h as deprecated (Tim Wojtulewicz, Corelight) * Remove vector iterator type aliases from SmithWaterman code, fix uses of them (Tim Wojtulewicz, Corelight) * Fix warning with usage of fmt() (Tim Wojtulewicz, Corelight) 3.3.0-dev.190 | 2020-08-24 14:43:06 -0700 * Make set_processing_status() signal-safe. Closes #574. (Robin Sommer, Corelight) 3.3.0-dev.188 | 2020-08-24 14:30:43 -0700 * GH-710: Check that &expire_func is a function and not hook/event (Jon Siwek, Corelight) 3.3.0-dev.186 | 2020-08-24 14:28:25 -0700 * Fix a case where PktSrc gets processed twice in one runloop iteration For a non-live PktSrc, it had a special-case to be considered "ready" every iteration, but additionally every 1 in 100 iterations (the polling frequency), if there were no other "ready" IOSources, it would get added to the "ready" set a 2nd time. This commit completely excludes PktSrc from being processed during the 1/100 runloop iteration where a Poll() happens. That exclusion is desirable for a second reason: if reading a pcap happens to do its final Process() during that 1/100 polling-iteration and there's other IOSources ready to process like EventMgr/TimerMgr, those sources have logic to advance network-time to current-time if a PktSrc is no longer open. So in such a case, PktSrc::Process() closes, then EventMgr::Process() sees there's no longer an active PktSrc and advances to current-time, then EventMgr::Drain() happens and may dispatch various events that were previous scheduled, with those events now unexpectedly seeing a network_time() returning current-time. (Jon Siwek, Corelight) 3.3.0-dev.184 | 2020-08-24 14:27:31 -0700 * GH-594: Improve table initialization type-check error messages (Jon Siwek, Corelight) 3.3.0-dev.182 | 2020-08-21 13:46:16 -0700 * Fix `zeek -NN` not printing canonical names for file analyzers (Tim Wojtulewicz, Corelight) 3.3.0-dev.180 | 2020-08-21 13:28:23 -0700 * Remove definition-less ThreadVal<->BrokerData conversion functions (Jon Siwek, Corelight) Their presence/deprecation may cause "undefined reference" build failures for some compilers (GCC 9.3.0 in my case). 3.3.0-dev.177 | 2020-08-21 08:36:48 -0700 * Add missing namespace to usage of get_exe_path in fuzzer (Tim Wojtulewicz, Corelight) * Rename methods in RunState to remove 'net' from their names (Tim Wojtulewicz, Corelight) * Move zeek::net namespace to zeek::run_state namespace. This also moves all of the code from Net.{h,cc} to RunState.{h,cc} and marks Net.h as deprecated (Tim Wojtulewicz, Corelight) * Move ScannedFile class and associated globals into ScannedFile.h and out of Net.h and scan.l (Tim Wojtulewicz, Corelight) * Rename types in ZeekList.h to be consistent with the style guide (Tim Wojtulewicz, Corelight) * Move NetVar from zeek to zeek::detail namespace (Tim Wojtulewicz, Corelight) * Remove PRI_PTR_COMPAT macros (Tim Wojtulewicz, Corelight) * Fix indentation of namespaced aliases (Tim Wojtulewicz, Corelight) * Move zeek-setup code into namespaces (Tim Wojtulewicz, Corelight) * Move ZeekList types to zeek namespace (Tim Wojtulewicz, Corelight) * Move __RegisterBif from zeek::detail::plugin to zeek::plugin::detail (Tim Wojtulewicz, Corelight) * Remove unimplemented zeek_magic_path/bro_magic_path method (Tim Wojtulewicz, Corelight) * Move all plugin classes into zeek::plugin::detail namespaces (Tim Wojtulewicz, Corelight) * Rename BroList.h to ZeekList.h (Tim Wojtulewicz, Corelight) * Move a few smaller files to zeek namespaces (Tim Wojtulewicz, Corelight) * Tag the end of some namespaces for consistency (Tim Wojtulewicz, Corelight) * Move a few of the zeek::util methods and variables to zeek::util::detail (Tim Wojtulewicz, Corelight) * Move zeekygen code to zeek::zeekygen::detail namespace (Tim Wojtulewicz, Corelight) * Mark zeek::util::pad_size as constexpr, which provides a small performance improvement (Tim Wojtulewicz, Corelight) * Move everything in util.h to zeek::util namespace. This commit includes renaming a number of methods prefixed with bro_ to be prefixed with zeek_. (Tim Wojtulewicz, Corelight) * Deprecate ptr_compat_uint and ptr_compat_int in util.h (Tim Wojtulewicz, Corelight) * Move NetVar.h to zeek namespace (Tim Wojtulewicz, Corelight) * Move the functions and variables in Net.h to the zeek::net namespace. This includes moving network_time out of util.h. (Tim Wojtulewicz, Corelight) * Add namespaced version of trigger_mgr, fix a couple uses of event_registry (Tim Wojtulewicz, Corelight) * Move the remainder of the analyzers to zeek namespaces (Tim Wojtulewicz, Corelight) * Move arp, tcp, udp, pia, and stepping stone analyzers (Tim Wojtulewicz, Corelight) * Move bro_broker code to zeek::Broker namespace (Tim Wojtulewicz, Corelight) * Move notifier code to zeek::notifier::detail namespace (Tim Wojtulewicz, Corelight) * Move threading classes to zeek namespaces (Tim Wojtulewicz, Corelight) * Move probabilistic code into zeek namespaces (Tim Wojtulewicz, Corelight) * Move iosource code to zeek namespaces (Tim Wojtulewicz, Corelight) * Move logging code to zeek namespaces (Tim Wojtulewicz, Corelight) * Move input code to zeek namespaces (Tim Wojtulewicz, Corelight) * Move file_analysis code to zeek namespaces (Tim Wojtulewicz, Corelight) 3.3.0-dev.142 | 2020-08-20 16:26:06 -0700 * Improve stability of config/cluster_resend btest (Jon Siwek, Corelight) 3.3.0-dev.141 | 2020-08-20 14:56:15 -0700 * Mark FileAssembler destructor as default, remove implementation (Tim Wojtulewicz, Corelight) * Remove a few protected or private no-op constructors (Tim Wojtulewicz, Corelight) * Properly initialize Timer::type. Fixes Coverity 1431144. (Tim Wojtulewicz, Corelight) * Properly initialize members of Reassembler. Fixes Coverity 1413348. (Tim Wojtulewicz, Corelight) * Fix a number of Coverity issues in Dict (Tim Wojtulewicz, Corelight) 1431186: Asserting that an unsigned value is >= 0 is pointless 1431188/1431189/1431191: Side effect of using an assignment operator in a call to ASSERT() 1431193: Dereference after null check could lead to null being used 1431195: Use of rand() replaced with random() 3.3.0-dev.135 | 2020-08-20 10:23:29 -0700 * remove variables in netvar, use tabs in DNS.h and polish comments in dns events. (FlyingWithJerome) * add new line at the end of the test baseline (FlyingWithJerome) * add data+=option_len to skip unknown bytes (FlyingWithJerome) * better explanation to server cookie (FlyingWithJerome) * add edns-cookie testcase (FlyingWithJerome) * remove data+=option_len error in cookie and keepalive (FlyingWithJerome) * add EDNS cookie parsing (FlyingWithJerome) * add testcases (FlyingWithJerome) * add units (100ms) to comments (FlyingWithJerome) * initial adding EDNS TCP keepalive (FlyingWithJerome) 3.3.0-dev.119 | 2020-08-19 14:35:51 -0700 * Misc strings.bif adjustments * Declare rfind_str() with correct return type * Fix compiler warnings for signed/unsigned comparisons (Jon Siwek, Corelight) 3.3.0-dev.117 | 2020-08-19 14:12:06 -0700 * Use constexpr for IPAddr::v4_mapped_prefix declaration The inconsistent use of constexpr in the definition but not declaration causes an unrecoverable error for Coverity as seen in issue 1431151 (Jon Siwek, Corelight) 3.3.0-dev.115 | 2020-08-19 11:49:24 -0700 * Update docs for configuring CI email notifications (Jon Siwek, Corelight) Secrets are now stored organization-wide rather than per-repository. 3.3.0-dev.113 | 2020-08-18 12:10:26 -0700 * GH-467: Catch use of &add_func/&delete_func on IDs without values. (Robin Sommer, Corelight) * Fix compiler warning from missing namepsace qualification (Tim Wojtulewicz, Corelight) 3.3.0-dev.109 | 2020-08-14 10:00:07 -0700 * GH-178: Add new string bif methods based on python string utilities (Tim Wojtulewicz, Corelight) 3.3.0-dev.107 | 2020-08-12 20:01:37 +0000 * Accept bracketed ipv6 addresses (Justin Azoff, Corelight) * Fix minimize_info in ftp/main not returning a value. Fixes GH-1120 (Johanna Amann, Corelight) * Fix config reader regular expression for MUSL (Seth Hall, Corelight) * Moved verb ACTION_DROP to base/frameworks/notice/main.zeek. ACTION_DROP is not only part of catch-n-release subsystem and can be useful in other cases (Aashish Sharma) * Added new acld verbs of filter and nofilter (Aashish Sharma) * Fix some printf warnings with size_t values (Tim Wojtulewicz, Corelight) 3.3.0-dev.97 | 2020-08-11 11:34:17 -0700 * Move Dict constants to detail namespace (Tim Wojtulewicz, Corelight) * Add a few missing deprecation fixes (Tim Wojtulewicz, Corelight) * Adjust Dict whitespace/style (Jon Siwek, Corelight) * Adjust more btest timings (Jon Siwek, Corelight) * Improve termination reliability/speed for brokerstore btests (Jon Siwek, Corelight) * General btest cleanup - Use `-b` most everywhere, it will save time. - Start some intel tests upon the input file being fully read instead of at an arbitrary time. - Improve termination condition for some sumstats/cluster tests. - Filter uninteresting output from some supervisor tests. - Test for `notice_policy.log` is no longer needed. (Jon Siwek, Corelight) * Update NEWS about change in Dictionary implementation (Jon Siwek, Corelight) * Improve Intel expire-item btest to be less time-sensitive (Jon Siwek, Corelight) * Improve btests with unstable table/set output ordering Particularly, the final output order of a table/set is sensitive to order of input/insertions and some tests were converting std::unordered_{set,map} to Zeek table/set and iteration over those standard containers may not always loop through elements in the same order across all platforms. (Jon Siwek, Corelight) * Update doc submodule (Jon Siwek, Corelight) * Adjust a few btests that were unstable due to time-sensitivity (Jon Siwek, Corelight) * Fix DNS script deleting a table element while iterating Doesn't typically cause any problems since the loop breaks after deleting, except there's now an assert in debug builds catching potential problems like this. (Jon Siwek, Corelight) * Improve a brokerstore btest to filter out Broker connection messages (Jon Siwek, Corelight) * Sort output of a few SumStats cluster tests The order of $epoch_result() function calls among all keys within a single epoch isn't consequential. (Jon Siwek, Corelight) * Fix extract_first_email_addr() to really return the first email The use of find_all() in extract_email_addrs_vec() extracted occurrences to an intermediate set and thus lost any sense of ordering. This changes extract_email_addrs_vec() to use find_all_ordered() and return all occurrences of email addresses found in the argument, included duplicates, with their order of occurrence preserved. (Jon Siwek, Corelight) * Add find_all_ordered() BIF Operates similar to find_all(), except returns a vector instead of set to allow preservation of order/duplicates. (Jon Siwek, Corelight) * Extend external test suite canonifier with set-sorting logic Two new canonifiers: one to sort the contents of conn.log "service" field and another to sort the contents of any field of type "set". (Jon Siwek, Corelight) * Update btests/baselines for OpenDict compat Haven't checked different build configurations yet, but all except a few SumStats tests are stable for me now. The external tests are also completely failing, but haven't looked at those yet. (Jon Siwek, Corelight) * Fix new/malloc/delete/free mismatches in Dictionary code (Jon Siwek, Corelight) * Add explanation for a Dict TODO item (Jon Siwek, Corelight) * Fix compiler warning in Dictionary debug/dump printf format string (Jon Siwek, Corelight) * Overwrite old Dict.h/cc with new ones (Tim Wojtulewicz, Corelight) * Fix issue with sumstats script and fix baselines that were crashing previously (Tim Wojtulewicz, Corelight) * Extend the timeouts on a few intel tests, update baselines (Tim Wojtulewicz, Corelight) * Updating test baselines for new dictionary code due to changes in ordering of fields in the dictionary (Tim Wojtulewicz, Corelight) * Add namespaces for OpenDict files (Tim Wojtulewicz, Corelight) * Add unit tests from Dict into OpenDict files (Tim Wojtulewicz, Corelight) * Build fixups - Fix some clang-tidy warnings - Add std:: namespaces for ofstream and ios - Remove some unused methods (Tim Wojtulewicz, Corelight) * Massive formatting cleanup (jasonlue) * Remove key reference after insertion because the key may become invalid afterwards. (jasonlue) * add Clustered Hashing based Open Addressing Dict. To replace the existing dict, #define USE_OPEN_DICT (jasonlue) 3.3.0-dev.55 | 2020-08-10 09:57:36 -0700 * GH-1100: Fix reported body-length of HTTP messages w/ sub-entities (Jon Siwek, Corelight) The body-lengths of sub-entities, like multipart messages, got counted twice by mistake: once upon the end of the sub-entity and then again upon the end of the top-level entity that contains all sub-entities. The size of just the top-level entity is the correct one to use. 3.3.0-dev.53 | 2020-08-05 12:35:50 -0700 * Fix deprecation warnings in fuzz targets (Jon Siwek, Corelight) * Remove dbg_read_internal_state() (Jon Siwek, Corelight) It's never had a definition anyway and the namespacing/deprecation process was now causing a compile error in debug/fuzzing builds w/ GCC 3.3.0-dev.50 | 2020-08-04 20:22:48 +0000 * Fix incorrect conflict detection of namespaced-enum-names E.g. defining a `Foo::RED` enum name when a `GLOBAL::RED` identifier already exists would previously be treated as an error, even though the names don't truly conflict. (Jon Siwek, Corelight) * GH-1080: Treat enum name re-use across different enum types as an error (Jon Siwek, Corelight) * GH-1080: Rename conflicting NetControl::DROP enum definitions ``NetControl::DROP`` had 3 conflicting definitions that could potentially be used incorrectly without any warnings or type-checking errors. Such enum redefinition conflicts are now caught and treated as errors, so the ``NetControl::DROP`` enums had to be renamed: * The use as enum of type ``Log::ID`` is renamed to ``NetControl::DROP_LOG`` * The use as enum of type ``NetControl::CatchReleaseInfo`` is renamed to ``NetControl::DROP_REQUESTED`` * The use as enum of type ``NetControl::RuleType`` is unchanged and still named ``NetControl::DROP`` (Jon Siwek, Corelight) 3.3.0-dev.44 | 2020-08-04 08:47:08 -0700 * Silence compiler warnings related to deprecated TYPE_COUNTER (Jon Siwek, Corelight) 3.3.0-dev.42 | 2020-08-03 10:37:38 -0700 * Remove counter type, only leaving compatibility enum. The type was unused/non-functional. See GH-1095 (Johanna Amann, Corelight). 3.3.0-dev.38 | 2020-08-01 09:21:17 -0700 * Properly forward-declare zeek::plugin::plugin in DebugLogger (Tim Wojtulewicz, Corelight) * Rename Brofiler to ScriptCoverageManager (Tim Wojtulewicz, Corelight) * Move Conn and related types to zeek namespace (Tim Wojtulewicz, Corelight) * Move reassembler code to namespaces (Tim Wojtulewicz, Corelight) * Move Session code to namespaces (Tim Wojtulewicz, Corelight) * Move ODesc to zeek namespace (Tim Wojtulewicz, Corelight) * Move Stats and related classes to zeek::detail (Tim Wojtulewicz, Corelight) * Move a few low-use classes to namespaces (Tim Wojtulewicz, Corelight) * Move DebugLogger to zeek namespaces (Tim Wojtulewicz, Corelight) * Move all of the hashing classes/functions to zeek::detail namespace (Tim Wojtulewicz, Corelight) * Move all of the Packet-related classes to namespaces (Tim Wojtulewicz, Corelight) * Move EventMgr, EventHandler, and EventRegistry code to zeek namespace. Rename mgr to event_mgr. (Tim Wojtulewicz, Corelight) * Move Timer and PriorityQueue classes to namespaces (Tim Wojtulewicz, Corelight) * Move BroFile to zeek namespace, rename to File (Tim Wojtulewicz, Corelight) * Move Reporter to zeek namespace (Tim Wojtulewicz, Corelight) * Move DNS_Mgr to zeek::detail namespace (Tim Wojtulewicz, Corelight) * Move BroSubstring to zeek::detail, rename to Substring (Tim Wojtulewicz) * Move regex matching code to zeek namespaces (Tim Wojtulewicz, Corelight) * Move all of the rule matching code to zeek::detail (Tim Wojtulewicz, Corelight) * Move IP Addr/Prefix/Header classes into namespaces (Tim Wojtulewicz, Corelight) * Move all debugger code into the zeek::detail namespace (Tim Wojtulewicz, Corelight) * Restore globally-namespaced plugin_mgr variable, move zeek::plugin::plugin_mgr to zeek::plugin_mgr (Tim Wojtulewicz, Corelight) * Move all base analyzer classes to zeek::analyzer namespace (Tim Wojtulewicz, Corelight) * Move base Tag class to zeek namespace (Tim Wojtulewicz, Corelight) * Update submodule [nomail] (Johanna Amann, Corelight) * Update submodule(s) [nomail] (Jon Siwek, Corelight) 3.3.0-dev.11 | 2020-07-30 17:47:44 +0000 * Use .empty() instead of checking size against zero (Tim Wojtulewicz, Corelight) * Use properly-sized loop variables or convert to ranged-for (bugprone-too-small-loop-variable) (Tim Wojtulewicz, Corelight) * Fix potential heap-use-after-free access (Dominik Charousset, Corelight) 3.3.0-dev.4 | 2020-07-28 19:56:35 +0000 * GH-1083: Fix Input Framework 'change' events for 'set' destinations (Jon Siwek, Corelight) 3.3.0-dev.1 | 2020-07-27 12:04:48 -0700 * Update site/local.zeek compatibility test (Jon Siwek, Corelight) 3.3.0-dev | 2020-07-27 11:49:28 -0700 * Reserve spot in NEWS for 4.0.0 (Jon Siwek, Corelight) 3.2.0-dev.992 | 2020-07-27 11:36:53 -0700 * Fix potential NetbiosSSN analyzer stack overflow (Jon Siwek, Corelight) The Contents_NetbiosSSN analyzer used a recursive message parsing function that determined the size of the next message from the input packet-data itself. A packet containing a sequence of many small messages could cause a stack overflow since a recursion happened after processing each message. * Fix potential DNS analyzer stack overflow (Jon Siwek, Corelight) The Contents_DNS analyzer used a recursive message parsing function that determined the size of the next message from the input packet-data itself. A packet containing a sequence of many small messages could cause a stack overflow since a recursion happened after processing each message. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=24272 3.2.0-dev.988 | 2020-07-27 11:19:20 -0700 * Update submodules to release versions in prep for release/3.2 branch (Jon Siwek, Corelight) 3.2.0-dev.986 | 2020-07-24 13:57:19 -0700 * Misc/minor cleanup after reviewing new Broker table store code (Jon Siwek, Corelight) Just some naming/style + minor memory management tweaks (avoid some copies; use move semantics). 3.2.0-dev.985 | 2020-07-24 11:40:22 -0700 * MySQL: Fix EOF parsing logic bug. (Vlad Grigorescu) We were correctly NOT expecting an EOF, but because we were parsing the header and then not parsing the rest, we would get out of sync 3.2.0-dev.979 | 2020-07-24 09:03:06 -0700 * Implement EDNS Client Subnet Option (Ron Wellman) 3.2.0-dev.974 | 2020-07-23 13:02:13 -0700 * origin/topic/jsiwek/gh-1076-fix-random: Deprecate bro_srandom(), replace with zeek::seed_random(). Add zeek::max_random() & fix misuse of RAND_MAX w/ zeek::random_number() Deprecate bro_random(), replace with zeek::random_number() Deprecate bro_prng(), replace with zeek::prng() GH-1076: Fix bro_srandom() to replace 0 seeds with 1 GH-1076: Fix bro_prng() implementation GH-1076: Fix use of getrandom() (Tim Wojtulewicz, Corelight) * Deprecate bro_srandom(), replace with zeek::seed_random(). Avoiding zeek::srandom() to avoid potential for confusion with srandom() (Jon Siwek, Corelight) * Add zeek::max_random() & fix misuse of RAND_MAX w/ zeek::random_number() In deterministic mode, RAND_MAX is not related to the result of zeek::random_number() (formerly bro_random()), but some logic was using RAND_MAX as indication of the possible range of values. The new zeek::max_random() will give the correct upper-bound regardless of whether deterministic-mode is used. (Jon Siwek, Corelight) * Deprecate bro_random(), replace with zeek::random_number() Avoiding the use of zeek::random() due to potential for confusion with random(). (Jon Siwek, Corelight) * Deprecate bro_prng(), replace with zeek::prng() The type used for storing the state of the RNG is changed from `unsigned int` to `long int` since the former has a minimal range of [0, 65,535] while the RNG function itself has a range of [1, 2147483646]. A `long int` must be capable of [−2147483647, +2147483647] and is also the return type of `random()`, which is what zeek::prng() aims to roughly parity. (Jon Siwek, Corelight) * GH-1076: Fix bro_srandom() to replace 0 seeds with 1 The bro_prng() implementation cannot generate 0 as a result since it causes every subsequent number from the PRNG to also be 0, so use the number 1 instead of 0. (Jon Siwek, Corelight) * GH-1076: Fix bro_prng() implementation The intermediate result of the PRNG used unsigned storage, preventing the ( result < 0 ) branch from ever being evaluated. This could cause return values to exceed the modulus as well as RAND_MAX. One interesting effect of this is potential for the rand() BIF to return values outside the requested maximum limit. Another interesting effect of this is that a PacketFilter may start randomly dropping packets even if it was not configured for random-packet-drops. (Jon Siwek, Corelight) * GH-1076: Fix use of getrandom() The availability and use of getrandom() actually caused unrandom and deterministic results in terms of Zeek's random number generation. (Jon Siwek, Corelight) 3.2.0-dev.965 | 2020-07-23 08:31:41 -0700 * GH-1068: Add zeek symlink to allow "zeek/Foo.h" include style Also changes `zeek-config --include_dir` to now output `$prefix/include:$prefix/include/zeek` instead of only `$prefix/include/zeek` (Jon Siwek, Corelight) 3.2.0-dev.962 | 2020-07-21 12:39:34 -0700 * Add zeek-archiver tool as submodule This is added to the default Zeek build as a convenience since it's the new suggested method for archiving the logs produced by a Supervised Zeek Cluster. (Jon Siwek, Corelight) 3.2.0-dev.959 | 2020-07-21 15:34:59 +0000 * Broker Store table synchronizatio, (Johanna Amann, Corelight) Zeek now supports synchronizing tables/sets across clusters using a backing Broker store. The same feature also allows persistent storage of data in tables/sets over Zeek restarts. This feature is implemented using the new ``&backend`` attribute. To synchronize a table over a cluster, you can, e.g., use: global t: table[string] of count &backend=Broker::MEMORY; This feature is documented in detail here: https://docs.zeek.org/en/current/frameworks/broker.html#broker-store-backed-zeek-tables-for-data-synchronization-and-persistence This feature is experimental and the syntax/specifics can change in the future. 3.2.0-dev.919 | 2020-07-17 16:37:11 -0700 * Use namespaced version of Location to silence warnings (Tim Wojtulewicz, Corelight) 3.2.0-dev.915 | 2020-07-17 16:10:46 -0700 * Various deprecation fixes, reported by failed Spicy builds (Tim Wojtulewicz, Corelight) - Add deprecated version of ID::SetType() that takes Type* - Add deprecated versions of zeek::set_location in the global namespace - Fix global namespace version of lookup_ID to return ID* * Add deprecated version of EnumType::GetVal() to return EnumVal* (Tim Wojtulewicz, Corelight) Rename IntrusivePtr version to GetEnumVal 3.2.0-dev.911 | 2020-07-17 22:32:42 +0000 * Fix race condition in ensure_dir() If something else created the dir between the stat() and mkdir(), it previously reported that as a failure. (Jon Siwek, Corelight) * Fix tokenize_string() to work with delimiters of length > 1. (Jon Siwek, Corelight) 3.2.0-dev.908 | 2020-07-17 10:19:07 +0000 * GH-734: Improve handling of lambdas that escape enclosing frame. (Jon Siwek, Corelight) 3.2.0-dev.906 | 2020-07-17 10:08:31 +0000 * GH-837: Add test cases for mismatched Broker SSL configs. (Jon Siwek, Corelight) * GH-837: Emit reporter errors for Broker errors. (Jon Siwek, Corelight) * Fix incorrect/missing Broker error status code numbers. (Jon Siwek, Corelight) 3.2.0-dev.902 | 2020-07-14 19:10:36 -0700 * GH-1062: fix integer conversion errors related to Tag subtypes (Jon Siwek, Corelight) 3.2.0-dev.899 | 2020-07-14 00:02:05 +0000 * Improve Broker store API's handling of invalid arguments * Some methods mistakenly returned a bool instead of QueryResult when passed an invalid `opaque of Broker::Store` handle. * Now generates a runtime exception for store_name() and is_closed() calls that pass an invalid `opaque of Broker::Store` handle as any returned value can't be reasonably used in any subsequent logic. * Descriptions of any invalid arguments are now given in the error message. (Jon Siwek, Corelight) * Add zeek::detail::emit_builtin_exception() functions These work like zeek::emit_builtin_error(), but also throw an InterpreterException (Jon Siwek, Corelight) * GH-1024: fix crash on passing wrong types to Broker store API (Jon Siwek, Corelight) 3.2.0-dev.894 | 2020-07-13 12:12:17 -0700 * GH-1019: deprecate icmp_conn params for ICMP events Previously, a single `icmp_conn` record was built per ICMP "connection" and re-used for all events generated from it. This may have been a historical attempt at performance optimization, but: * By default, Zeek does not load any scripts that handle ICMP events. * The one script Zeek ships with that does handle ICMP events, "detect-traceroute", is already noted as being disabled due to potential performance problems of doing that kind of analysis. * Re-use of the original `icmp_conn` record tends to misreport TTL and length values since they come from original packet instead of the current one. * Even if we chose to still re-use `icmp_conn` records and just fill in a new TTL and length value each packet, a user script could have stored a reference to the record and not be expecting those values to be changed out from underneath them. Now, a new `icmp_info` record is created/populated in all ICMP events and should be used instead of `icmp_conn`. It also removes the orig_h/resp_h fields as those are redundant with what's already available in the connection record. (Jon Siwek, Corelight) 3.2.0-dev.892 | 2020-07-13 12:10:22 -0700 * Fix wrong frame offsets for locals of alternate event/hook prototypes Local frame offsets were being assigned based on number of the alternate prototype's parameters, which may end up having less total parameters than the canonical prototype, causing the local value to incorrectly overwrite an event/hook argument value. (Jon Siwek, Corelight) * Add deprecation expression to deprecated prototype/parameter messages (Jon Siwek, Corelight) * Improve "use of deprecated prototype" warning message The location information now points out the place of the deprecated prototype instead of the location where the ID was initially declared (which may not itself be a deprecated prototype). (Jon Siwek, Corelight) * Emit deprecation warning for use of &deprecated function parameters Particularly, this is meant for using &deprecated on canonical event/hook prototype parameters to encourage users to create handlers to another, non-deprecated prototype. i.e. for canonical prototypes, we may not always want to put &deprecated directly on the prototype itself since that signals deprecation of the ID entirely. (Jon Siwek, Corelight) 3.2.0-dev.885 | 2020-07-10 11:20:41 -0700 * Add more error checks to shadow log parsing (Jon Siwek, Corelight) i.e. Coverity warns about possible use of ftell() negative return value 3.2.0-dev.884 | 2020-07-09 14:09:58 -0700 * Add Supervisor::{stdout,stderr}_hook (Jon Siwek, Corelight) These allow capturing/handling the stdout/stderr of child processes via Zeek scripts. * Add Log::rotation_format_func and Log::default_rotation_dir options (Jon Siwek, Corelight) These may be redefined to customize log rotation path prefixes, including use of a directory. File extensions are still up to individual log writers to add themselves during the actual rotation. These new also allow for some simplication to the default ASCII postprocessor function: it eliminates the need for it doing an extra/awkward rename() operation that only changes the timestamp format. This also teaches the supervisor framework to use these new options to rotate ascii logs into a log-queue/ directory with a specific file name format (intended for an external archiver process to monitor separately). * Allow Zeek function equality comparisons (Jon Siwek, Corelight) Equality between two functions acts like a pointer comparison. * Implement leftover log rotation/archival for supervised nodes (Jon Siwek, Corelight) This helps prevent a node from being killed/crashing in the middle of writing a log, restarting, and eventually clobbering that log file that never underwent the rotation/archival process. The old `archive-log` and `post-terminate` scripts as used by ZeekControl previously implemented this behavior, but the new logic is entirely in the ASCII writer. It uses ".shadow" log files stored alongside the real log to help detect such scenarios and rotate them correctly upon the next startup of the Zeek process. * Move some supervisor structures to detail namespace (Jon Siwek, Corelight) * StemHandle -> detail::SupervisorStemHandle * SupervisedNode -> detail::SupervisedNode * Supervisor::Node -> detail::SupervisorNode * Improve default supervisor stdout/stderr handling (Jon Siwek, Corelight) The stdout/stderr of child processes is now redirected over a pipe back to the supervisor process so that it can prefix the output with the name of the emitting node. * Factor Supervisor initialization data to separate struct from Stem (Jon Siwek, Corelight) * Move ParentProcessCheckTimer to detail namespace (Jon Siwek, Corelight) 3.2.0-dev.874 | 2020-07-08 17:46:20 -0700 * scan.l: Remove "constant" did_module_restore logic / dead code (Arne Welzel) 3.2.0-dev.872 | 2020-07-08 12:25:11 -0700 * Fix FreeBSD CI script to install right SWIG package (Jon Siwek, Corelight) 3.2.0-dev.870 | 2020-07-08 15:41:15 +0000 * Fix WhileStmt to call Stmt(Tag) constructor. (Jon Siwek, Corelight) 3.2.0-dev.868 | 2020-07-08 15:39:46 +0000 * GH-928: use realpath() instead of inode to de-duplicate scripts. (Jon Siwek, Corelight) 3.2.0-dev.864 | 2020-07-06 21:06:41 -0700 * GH-1040: Add zero-indexed version of str_split (Tim Wojtulewicz, Corelight) 3.2.0-dev.862 | 2020-07-06 20:40:44 -0700 * GH-1041: Move compress_path to a bif that uses normalize_path (Tim Wojtulewicz, Corelight) 3.2.0-dev.859 | 2020-07-06 14:55:00 +0000 * Add backtrace() and print_backtrace() BIF functions. (Jon Siwek, Corelight) 3.2.0-dev.855 | 2020-07-03 14:29:55 -0700 * Fix --enable-mobile-ipv6 build (Jon Siwek, Corelight) 3.2.0-dev.854 | 2020-07-03 14:17:42 -0700 * Fix namespace of GetCurrentLocation() to zeek::detail (Jon Siwek, Corelight) 3.2.0-dev.853 | 2020-07-02 19:24:09 -0700 * Rename BroString files to ZeekString (Tim Wojtulewicz, Corelight) * Rename BroObj to Obj (Tim Wojtulewicz, Corelight) * Rename BroString to zeek::String (Tim Wojtulewicz, Corelight) * Move Func up to zeek namespace, rename BroFunc to ScriptFunc (Tim Wojtulewicz, Corelight) * Mark global val_mgr as deprecated and fix uses of it to use namespaced version (Tim Wojtulewicz, Corelight) * Update binpac and bifcl submodules with review changes (Tim Wojtulewicz, Corelight) * Move Location to zeek::detail and BroObj to zeek (Tim Wojtulewicz, Corelight) * Move BroString to zeek namespace (Tim Wojtulewicz, Corelight) * Move Dictionary/PDict, List/PList, and Queue/PQueue to zeek namespace (Tim Wojtulewicz, Corelight) * Remove typedef that should have been removed in 3.1 (Tim Wojtulewicz, Corelight) * Move Func and associated classes into zeek::detail namespace (Tim Wojtulewicz, Corelight) * Move Frame and Scope to zeek::detail namespace (Tim Wojtulewicz) * Move all Val classes to the zeek namespaces (Tim Wojtulewicz, Corelight) * Use type aliases for IntrusivePtr definitions (Tim Wojtulewicz, Corelight) * Move deprecation macro to zeek-config.h.in to avoid having to over-include util.h (Tim Wojtulewicz, Corelight) * Move IntrusivePtr and utility methods to the zeek namespace (Tim Wojtulewicz, Corelight) 3.2.0-dev.834 | 2020-07-02 15:57:09 -0700 * GH-786: fix Val::As methods casting incomplete types (Jon Siwek, Corelight) * GH-786: fix Stmt::As methods casting incomplete types (Jon Siwek, Corelight) * GH-786: fix Expr::As methods casting incomplete types (Jon Siwek, Corelight) 3.2.0-dev.830 | 2020-07-02 11:36:28 -0700 * GH-786: Move Type::As methods to cc file so they have type info for casting safely (Tim Wojtulewicz, Corelight) 3.2.0-dev.828 | 2020-07-01 09:57:23 -0700 * Fix a deprecation warning compiling with GCC (Jon Siwek, Corelight) 3.2.0-dev.827 | 2020-06-30 16:54:22 -0700 * Deprecate plugin::HookType and plugin::component::Type in a different way (Tim Wojtulewicz, Corelight) * Deprecate init_class and IDScope in another way. (Tim Wojtulewicz, Corelight) This also renamed init_class to InitClass for consistency. * Deprecate TypeTag and friends in a different way (Tim Wojtulewicz, Corelight) * Deprecate attr_tag in a different way, rename to AttrTag (Tim Wojtulewicz, Corelight) 3.2.0-dev.822 | 2020-06-30 15:54:38 -0700 * Revert Attributes::Attrs back to return an attr_list and mark it deprecated (Tim Wojtulewicz, Corelight) * GH-1034: Revert TypeList::Types() back to return a type_list* and mark it deprecated (Tim Wojtulewicz, Corelight) 3.2.0-dev.817 | 2020-06-29 19:23:21 +0000 * GH-1036: change print.log to log network time instead of current (Jon Siwek, Corelight) 3.2.0-dev.815 | 2020-06-29 10:25:05 -0700 * reduce memory usage of ConnPolling (Justin Azoff) Instead of scheduling the event with the full 'connection' record, schedule it with the smaller 'conn_id' record. 3.2.0-dev.813 | 2020-06-26 16:25:34 +0000 * Fix several issues with command-line option redefs * Variables of `string` type can now be set to an empty string * Trying to set a variable with non-`string` type to an empty value now emits an error instead of silently doing nothing * Providing an invalid identifier now emits an "unknown identifier" error instead of silently doing nothing (Jon Siwek, Corelight) * Fix "possibly-truncated" compiler warning in BuildJSON snprintf() (Jon Siwek, Corelight) * GH-1025: allow copying/cloning of `opaque of Broker::Store` Implemented simply as a reference count increment of the data store handle. (Jon Siwek, Corelight) * Fix shadowed variable that breaks lookup_hostname(). (Jon Siwek, Corelight) 3.2.0-dev.804 | 2020-06-25 23:58:59 -0700 * Extend dns_request, dns_reject, and dns_query_reply events with original_query param (Ryan Victory) Where the "query" parameter is downcased, the "original_query" is passed unmodified to these events. Added a policy/protocols/dns/log-original-query-case.zeek script that adds the original_query to the dns.log file as well. * Convert to using permissions to check for access to cirrus variables in benchmark script (Tim Wojtulewicz, Corelight) 3.2.0-dev.798 | 2020-06-19 13:25:02 -0700 * Remove last_access_time from TableEntryVal. (Johanna Amann, Corelight) Turns out - this was no longer used. And it takes up at least 8 bytes of space for every single table/set entry. 3.2.0-dev.796 | 2020-06-18 20:40:59 +0000 * cmake: Make musl support more distro agnostic (Andrew Benson) 3.2.0-dev.794 | 2020-06-18 20:04:06 +0000 * Minimize data published for expected FTP data channel analysis Previously, more data than could effectively be utilized by any remote Zeek was published (e.g. full list of pending commands or other transient state that may add up to non-trivial amount of bytes). (Jon Siwek, Corelight) * GH-998: Fix Reporter::conn_weird() to handle expired connections This introduces a new sampling state-map for expired connections to fix segfaults that previously occured when passing in a `connection` record to `Reporter::conn_weird()` for which the internal `Connection` object had already been expired and deleted. This also introduces a new event called `expired_conn_weird`, which is similar to `conn_weird`, except the full `connection` record is no longer available, just the `conn_id` and UID string. (Jon Siwek, Corelight) * Place build file in explicit location for benchmarking to work correctly (Tim Wojtulewicz, Corelight) * Update highwayhash submodule to upstream. This includes a patch by us, which fixes the compile on AARCH64. Fixes GH-1006. (Johanna Amann, Corelight) 3.2.0-dev.788 | 2020-06-15 11:04:20 -0700 * Fix location where CI places build.tgz (Jon Siwek, Corelight) 3.2.0-dev.787 | 2020-06-15 10:27:48 -0700 * Add new CI task for running benchmarks on a remote host (Tim Wojtulewicz, Corelight) 3.2.0-dev.783 | 2020-06-11 23:21:41 -0700 * Compare pcap_next_ex() result to PCAP_ERROR/PCAP_ERROR_BREAK (Jon Siwek, Corelight) * GH-977: Improve pcap error handling (Jon Siwek, Corelight) Switches from pcap_next() to pcap_next_ex() to better handle all error conditions. This allows, for example, to have a non-zero exit code for a Zeek process that fails to fully process all packets in a pcap file. * Remove not-useful code in iosource::Manager::OpenPktSrc (Jon Siwek, Corelight) 3.2.0-dev.779 | 2020-06-11 23:17:46 -0700 * Rename BroType to zeek::Type (Tim Wojtulewicz, Corelight) * Move Type types to zeek namespace (Tim Wojtulewicz, Corelight) * Move Flare/Pipe from the bro namespace to zeek::detail (Tim Wojtulewicz, Corelight) * Move Attr to the zeek::detail namespace (Tim Wojtulewicz, Corelight) * Move Trigger into the zeek::detail namespace (Tim Wojtulewicz, Corelight) * Move ID to the zeek::detail namespace (Tim Wojtulewicz, Corelight) * Move Anon.h into zeek::detail namespace (Tim Wojtulewicz, Corelight) * Mark all of the aliased classes in plugin/Plugin.h deprecated (Tim Wojtulewicz, Corelight) And fix all of the plugins that were using them * Move all of the base plugin classes into the zeek::plugin namespace (Tim Wojtulewicz, Corelight) * Expr: move all classes into zeek::detail (Tim Wojtulewicz, Corelight) * Stmt: move Stmt classes into zeek::detail namespace (Tim Wojtulewicz, Corelight) * Add utility macro for creating namespaced aliases for classes (Tim Wojtulewicz, Corelight) 3.2.0-dev.763 | 2020-06-10 16:34:31 -0700 * Optimize Connection::RemovalEvent() for bare-mode usage (Jon Siwek, Corelight) It was creating RecordVals even if they wouldn't be used by any event handler and that situation is common/expected for `zeek -b` mode. Normally, there's at least the tunnel scripts with a `new_connection` handler causing the connection RecordVals to be built. 3.2.0-dev.762 | 2020-06-09 15:19:15 -0700 * Restrict Cirrus CI to only zeek repo's branches (Jon Siwek, Corelight) 3.2.0-dev.761 | 2020-06-09 09:39:31 -0700 * Prevent IP fragment reassembly on packets without minimal IP header (Jon Siwek, Corelight) The IP fragment reassembly process assumes a packet contains at least the minimum IP header, but such a check did not previously occur, resulting in a heap buffer over-read. For example, a self-reported IPv4 IHL field with a value less than minimum IPv4 header length of 20 bytes. Such packets likely aren't routable on their own, but one can create an artifical pcap like that or possibly encapsulate it within another protocol to trigger this bug. * Fix potential leak of Analyzers added to tree during Analyzer::Done (Jon Siwek, Corelight) Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22630 * Fix multiple content-transfer-encoding headers causing a memory leak (Jon Siwek, Corelight) The MIME analyzer may also leak if it sees many multipart boundary parameters. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22871 * Fix potential stack overflow in NVT analyzer (Jon Siwek, Corelight) The NVT_Analyzer (e.g. as instantiated to support the FTP analyzer) uses a recursive parsing function that may only advance one byte at a time and can easily cause a stack overflow as a result. This change replaces the recursive calls with equivalent iterative logic. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22898 https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22972 * Fix NVT analyzer memory leak from multiple telnet authn name options (Jon Siwek, Corelight) Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=23069 3.2.0-dev.751 | 2020-06-08 22:30:18 +0000 * GH-999: Stop formatting DHCP Client ID Hardware Type 0 as MAC For `DHCP::ClientID$hwtype` fields equal to 0, the `hwaddr` field is no longer misformatted as a MAC and instead just contains the raw bytes seen in the DHCP Client ID Option. (Jon Siwek, Corelight) 3.2.0-dev.749 | 2020-06-08 11:22:45 -0700 * Remove inline from some static KeyedHash members (Jon Siwek, Corelight) Coverity Scan builds currently encounter catastrophic error, claiming alignas requires use on both declaration and definition, so appears to actually not understand "static inline" in combo with alignas. * Improve Func.h inclusion (Jon Siwek, Corelight) Now forward declares some Broker types since Broker/CAF headers generally slow things down and also Coverity Scan currently has a catastrophic error on some CAF headers. Also a few other changes to EventHandler/BifReturnVal to reduce number of places that depend on Func.h. * Rename aux/ to auxil/ (Jon Siwek, Corelight) Since "aux" is not an allowed file/dir name on Windows. 3.2.0-dev.744 | 2020-06-04 15:11:56 -0700 * Fix use-after-move of proc_status_file breaking -U flag (Jon Siwek, Corelight) * Change Cirrus CI config to use org-level secured variable (Jon Siwek, Corelight) 3.2.0-dev.742 | 2020-06-03 13:29:12 -0700 * GH-973: Fix the return type of topk_get_top() e.g. when using `string` elements it should return `vector of string`, not `vector of list of string` like it did before. (Jon Siwek, Corelight) 3.2.0-dev.740 | 2020-06-03 13:17:26 -0700 * Deprecate Val(double, TypeTag) ctor, add TimeVal/DoubleVal subclasses This also updates all usages of the deprecated Val ctor to use either IntervalVal, TimeVal, or DoubleVal ctors. The reason for doing away with the old constructor is that using it with TYPE_INTERVAL isn't strictly correct since there exists a more specific subclass, IntervalVal, with overriden ValDescribe() method that ought to be used to print such values in a more descriptive way. (Jon Siwek, Corelight) 3.2.0-dev.738 | 2020-06-02 18:13:50 -0700 * GH-985: Fix descriptions of double_to_interval() return values (Jon Siwek, Corelight) The BIF was not returning an IntervalVal which has an overriden ValDescribe() method that allows for prettier printing like "6.0 secs" instead of just "6.0". 3.2.0-dev.736 | 2020-06-02 12:37:56 -0700 * Decrease number of CPUs/memory for Cirrus CI tasks (Jon Siwek, Corelight) 3.2.0-dev.735 | 2020-06-02 10:55:43 -0700 * Change CI script to compile from build/Makefile (Jon Siwek, Corelight) 3.2.0-dev.734 | 2020-06-01 20:44:29 -0700 * Add missing include in util.cc (Jon Siwek, Corelight) 3.2.0-dev.733 | 2020-06-01 19:25:37 -0700 * Remove Analyzer.h from bro-bif.h (Tim Wojtulewicz, Corelight) * Remove IPAddr.h from Reporter.h (Tim Wojtulewicz, Corelight) * Remove the inclusion of Func.h from NetVar.h (Tim Wojtulewicz, Corelight) 3.2.0-dev.727 | 2020-06-01 10:57:16 -0700 * Integrate review feedback * Add deprecation for MIME_Entity::ContentType(), use GetContentType() * Add deprecation for MIME_Entity::ContentSubType(), use GetContentSubType() * Add deprecation for MIME_Message::BuildHeaderVal(), use ToHeaderVal() * Add deprecation for MIME_Message::BuildHeaderTable(), use ToHeaderTable() * Add deprecation for mime::new_string_val(), use mime::to_stringval() * Add deprecation for ARP_Analyzer::ConstructAddrVal(), use ToAddrVal() * Add deprecation for ARP_Analyzer::EthAddrToStr(), use ToEthAddrStr() * Change the Func::Call() replacement to be named Func::Invoke() (Jon Siwek, Corelight) * Switch Broker Val converter visitor to return IntrusivePtr (Jon Siwek, Corelight) * Change BroFunc ctor to take const-ref IntrusivePtr (Jon Siwek, Corelight) * Add version of Frame::SetElement() taking IntrusivePtr Expect the version using raw ID* could go away eventually, but this is convenience for the meantime. (Jon Siwek, Corelight) * Change Scope/Func inits from id_list* to vector> (Jon Siwek, Corelight) * Change Scope::GenerateTemporary() to return IntrusivePtr (Jon Siwek, Corelight) * Deprecate Scope::ReturnType(), replace with GetReturnType() (Jon Siwek, Corelight) * Deprecate Scope::ScopeID(), replace with GetID() (Jon Siwek, Corelight) * Switch parsing to use vector> from attr_list This allows improved passing/storing of Attr references to Exprs, TypeDecl, Scope, etc. (Jon Siwek, Corelight) * Deprecate TableVal::FindAttr(), replace with GetAttr() (Jon Siwek, Corelight) * Deprecate TypeDecl::FindAttr(), replace with GetAttr() (Jon Siwek, Corelight) * Deprecate ID::FindAttr(), replace with GetAttr() (Jon Siwek, Corelight) * Deprecate Attributes::FindAttr(), replace with Find() (Jon Siwek, Corelight) * Deprecate Attributes::AddAttrs(Attributes*) Replaced with version taking an IntrusivePtr parameter (Jon Siwek, Corelight) * Add Attributes ctor that takes IntrusivePtrs (Jon Siwek, Corelight) * Change Attributes to store std:vector> This also changes the return type of Attributes::Attrs() from attr_list* (Jon Siwek, Corelight) * Change Attr::SetAttrExpr() to non-template (Jon Siwek, Corelight) * Deprecate Attr::AttrExpr(), replace with GetExpr() (Jon Siwek, Corelight) * Deprecate ID::Attrs(), replace with GetAttrs() (Jon Siwek, Corelight) * Remove weak_ref param from ID::SetVal() It was not used anywhere. (Jon Siwek, Corelight) * Store weak ref boolean along with Frame element Vals (Jon Siwek, Corelight) * Deprecate Frame::GetElement(ID*), replace with GetElementByID() (Jon Siwek, Corelight) * Deprecate Frame::NthElement(), replace with GetElement() (Jon Siwek, Corelight) * Switch Frame::SetElement() to use IntrusivePtr (Jon Siwek, Corelight) * Store IntrusivePtrs in Frame (Jon Siwek, Corelight) * Deprecate Plugin::HookCallFunction(), replace with HookFunctionCall() This also changes the argument type of Func::operator() to zeek::Args* to allow plugins to be able to alter function arguments in place as was previously documented. (Jon Siwek, Corelight) * Switch plugin::Manager::HookCallFunction() to return IntrusivePtr The plugin::Plugin side of things is not (yet) changed. (Jon Siwek, Corelight) * Port remaining file analysis API to use IntrusivePtr (Jon Siwek, Corelight) * Deprecate file analyzer construction methods taking raw RecordVal* Replaced with versions that instead take IntrusivePtr (Jon Siwek, Corelight) * Deprecate file_analysis::Analyzer::Args(), replace with GetArgs() (Jon Siwek, Corelight) * Deprecate file_analysis::File::GetVal(), replace with ToVal() (Jon Siwek, Corelight) * Change file_analysis::File::GetID() to return const-ref (Jon Siwek, Corelight) * Fix build: some don't like IntrusivePtr default arg w/ incomplete type (Jon Siwek, Corelight) * Change bro_broker::DataVal::ScriptDataType() to IntrusivePtr (Jon Siwek, Corelight) * Change find_nested_record_types() to take IntrusivePtr (Jon Siwek, Corelight) * Deprecate TypeType::Type(), replace with GetType() (Jon Siwek, Corelight) * Add is_assignable() overload taking TypeTag (Jon Siwek, Corelight) * Add is_atomic_type() overloads for IntrusivePtr (Jon Siwek, Corelight) * Add same_type() overloads for IntrusivePtr args (Jon Siwek, Corelight) * Change merge_types() to take IntrusivePtrs (Jon Siwek, Corelight) * Deprecate IndexType::Indices(), replace with GetIndices() (Jon Siwek, Corelight) * Add is_vector() methods taking const-ref IntrusivePtr (Jon Siwek, Corelight) * Factor static-local nil IntrusivePtrs to global locations Minor optimization to remove any run-time impact. (Jon Siwek, Corelight) * Deprecate VectorVal::Lookup(), replace with At() (Jon Siwek, Corelight) * Switch VectorVal BroValUnion to store std::vector> This changes the return type of AsVector() from std::vector* (Jon Siwek, Corelight) * Deprecate VectorVal::Insert() taking raw Val*, use IntrusivePtr (Jon Siwek, Corelight) * Deprecate VectorVal::Assign methods taking raw Val* And adapt usages to pass in to alternate method taking IntrusivePtr (Jon Siwek, Corelight) * Merge remote-tracking branch 'origin/master' into topic/jsiwek/gh-893-intrusive-ptr-migration (Jon Siwek, Corelight) * Change EventHandler to store IntrusivePtr Also deprecates the LocalHandler() and SetLocalHandler() methods, replaced with GetFunc() and SetFunc(). (Jon Siwek, Corelight) * Add Val::AsFuncPtr() convenience method Since it's not trivial to store IntrusivePtr in BroValUnion and also not satisfying to store IntrusivePtr*. (Jon Siwek, Corelight) * Deprecate StringVal::Substitute(), replace with Replace() (Jon Siwek, Corelight) * Switch a TableVal::CallChangeFunc param to IntrusivePtr (Jon Siwek, Corelight) * Fix ambiguous ODesc::Add() call (Jon Siwek, Corelight) * Minor TableVal::Assign() ref-counting optimization (Jon Siwek, Corelight) * Improve TableVal HashKey management * Deprecated ComputeHash() methods and replaced with MakeHashKey() which returns std::unique_ptr * Deprecated RecoverIndex() and replaced with RecreateIndex() which takes HashKey& and returns IntrusivePtr. * Updated the new TableVal Assign()/Remove() methods to take either std::unique_ptr or HashKey& as appropriate for clarity of ownership expectations. (Jon Siwek, Corelight) * Add back in a deprecated TableVal ctor taking raw pointers (Jon Siwek, Corelight) * Deprecate TableVal::Attrs(), replace with GetAttrs() (Jon Siwek, Corelight) * Deprecate TableVal::Delete(), replace with Remove() (Jon Siwek, Corelight) * Switch Func::Call(val_list*) back to returning Val* And renamed the method returning IntrusivePtr to operator(). This corrects the deprecation process for Func::Call(val_list*). (Jon Siwek, Corelight) * Deprecate TableVal::Lookup(), replace with Find()/FindOrDefault() (Jon Siwek, Corelight) * Deprecate TableVal IsSubsetOf and EqualTo taking Val*, use Val& (Jon Siwek, Corelight) * Deprecate ComputeHash(Val*) methods, replace with ComputeHash(Val&) (Jon Siwek, Corelight) * Deprecate TableVal::Intersect(), replace with Intersection() (Jon Siwek, Corelight) * Deprecate TableVal::Assign methods with Val*, add IntrusivePtr overloads (Jon Siwek, Corelight) * Deprecate RecordVal::Lookup(const char*, bool) Replace with GetField(const char*) and GetFieldOrDefault(const char*). (Jon Siwek, Corelight) * Deprecate RecordVal::LookupWithDefault() replace with GetFieldOrDefault() (The former was previously changed during this release cycle to return Intrusive pointer, but this just changes it back to return Val* and deprecates it). (Jon Siwek, Corelight) * Deprecate RecordVal::Lookup(int), replace with GetField(int) (Jon Siwek, Corelight) * Change BroValUnion to use IntrusivePtr for record field storage This also changes the AsRecord() and AsNonConstRecord() accessors to return std::vector>* instead of val_list* (Jon Siwek, Corelight) * Switch ASN1 Val conversion functions to return IntrusivePtr (Jon Siwek, Corelight) * Deprecate RecordVal::Assign(int, Val*) And adapt all usages to the existing overload taking IntrusivePtr. (Jon Siwek, Corelight) * Switch RPC analyzers to use IntrusivePtr (Jon Siwek, Corelight) * Switch RecordVal::CoerceTo() to use IntrusivePtr (Jon Siwek, Corelight) * Deprecate TableEntryVal::Value(), replace with GetVal() (Jon Siwek, Corelight) * Switch OpaqueVal::SerializeType() to IntrusivePtr (Jon Siwek, Corelight) * Switch BlommFilterVal/CardinalityVal to use IntrusivePtr (Jon Siwek, Corelight) * Switch some TopkVal methods to use IntrusivePtr (Jon Siwek, Corelight) * Switch TopkVal to store IntrusivePtr (Jon Siwek, Corelight) * Switch OpaqueVal::UnserializeType() to return IntrusivePtr (Jon Siwek, Corelight) * Deprecate FuncType::ArgTypes(), replace with ParamList() (Jon Siwek, Corelight) * Deprecate FuncType::Args(), replace with Params() (Jon Siwek, Corelight) * Deprecate EventHandler::FType(), replace with GetType() (Jon Siwek, Corelight) * Deprecate BroFile::FType(), replace with GetType() (Jon Siwek, Corelight) * Deprecate Func::FType(), replace with Func::GetType() (Jon Siwek, Corelight) * Change {Get,Set}ContentsFile() to use IntrusivePtr (Jon Siwek, Corelight) * Deprecate BroFile::GetFile(), replace with BroFile::Get() (Jon Siwek, Corelight) * Deprecate Val(BroFile*) ctor, replace with one using IntrusivePtr (Jon Siwek, Corelight) * Deprecate Val(Func*) ctor, replace with one using IntrusivePtr (Jon Siwek, Corelight) * Deprecate OpaqueVal/HashVal ctors that take OpaqueType* Replaced with ones that take IntrusivePtr (Jon Siwek, Corelight) * fixup! Deprecate Scope::Lookup(), replace with Scope::Find() (Jon Siwek, Corelight) * Switch EnumType::GetVal() to return const-ref (Jon Siwek, Corelight) * Change protected EnumVal ctor to use IntrusivePtr (Jon Siwek, Corelight) * Deprecate RecordVal(RecordType*) ctor Replaced with one that takes IntrusivePtr (Jon Siwek, Corelight) * Switch zeek::id::lookup to zeek::id::find For parity with Scope since it now uses Find instead of Lookup (Jon Siwek, Corelight) * Use zeek::BifEvent:: for enqueue_ functions instead of BifEvent:: (Jon Siwek, Corelight) * Replace deprecated usage of BifFunc:: with zeek::BifFunc:: Names of functions also changed slightly, like bro_fmt -> fmt_bif. Should generally be unusual/unexpected to see somone calling these directly from C++ in their plugin, but since technically possible in previous versions, I also removed the "private" restriction on accessing the BifReturnVal member. (Jon Siwek, Corelight) * Deprecate names in BifConst, replace with zeek::BifConst Some Val* types are also replaced with IntrusivePtr at the new location (Jon Siwek, Corelight) * Deprecate all BroType* in BifType:: namespace Replaced with equivalently named IntrusivePtr in zeek::BifType:: (Jon Siwek, Corelight) * Change EventRegistry/EventHandler methods to use std::string{_view} (Jon Siwek, Corelight) * Deprecate internal_handler(), replace with EventRegistry::Register() Added a couple explicit event declarations that were missing: "net_done" and "dns_mapping_name_changed". (Jon Siwek, Corelight) * Deprecate remaining "opt_internal" functions in Var.h (Jon Siwek, Corelight) * Remove signal_val declaration from Var.h (Jon Siwek, Corelight) * Change zeek::id::lookup functions to use std::string_view (Jon Siwek, Corelight) * Change Scope::Find() and Scope::Remove() to use std::string_view (Jon Siwek, Corelight) * Change lookup_ID() to return a const-reference (Jon Siwek, Corelight) * Deprecate Scope::Lookup(), replace with Scope::Find() (Jon Siwek, Corelight) * Move various elements into ID.h and zeek::id namespace * A handful of generic/useful/common global type pointers that used to be in NetVar.h * Lookup functions that used to be Var.h (Jon Siwek, Corelight) * Trim the list of "global type pointers" from NetVar.h further Most of them are deprecated now, with usage sites now doing the lookup themselves. (Jon Siwek, Corelight) * Deprecate global Val pointers in NetVar.h All of these have fairly niche uses, so better maintained as lookup/static closer to the usage site. (Jon Siwek, Corelight) * Deprecate global type pointers in NetVar.h There's analogous IntrusivePtrs in zeek::vars (Jon Siwek, Corelight) * Add RecordVal ctor that takes IntrusivePtr (Jon Siwek, Corelight) * Remove unused FlattenExpr (Jon Siwek, Corelight) * Add missing "vector_coerce" to expr_name() (Jon Siwek, Corelight) * Deprecate Expr::Type(), replace with GetType() (Jon Siwek, Corelight) * Deprecate Val::Type(), replace with GetType() (Jon Siwek, Corelight) * Change Val to store IntrusivePtr (Jon Siwek, Corelight) * Remove VectorVal::vector_type member (Jon Siwek, Corelight) * Deprecate VectorVal(VectorType*) ctora Adds a new one taking an IntrusivePtr. (Jon Siwek, Corelight) * Deprecate internal_list_val() (Jon Siwek, Corelight) * Deprecate opt_internal_val() (Jon Siwek, Corelight) * Deprecate internal_func(), replace with zeek::lookup_func() (Jon Siwek, Corelight) * Deprecate internal_val() and internal_const_val() Replaced with zeek::lookup_val() and zeek::lookup_const() (Jon Siwek, Corelight) * Deprecate internal_type(), replace with zeek::lookup_type() (Jon Siwek, Corelight) * Deprecate ID::ID_Val(), replace with ID::GetVal() (Jon Siwek, Corelight) * Use std::move in some zeekygen::Manager methods (Jon Siwek, Corelight) * Add Val TypeType constructor taking an IntrusivePtr (Jon Siwek, Corelight) * Deprecate ID::Type(), replace with GetType() (Jon Siwek, Corelight) * Deprecate ID::AsType(), add ID::IsType() and ID::GetType() (Jon Siwek, Corelight) * Deprecate BroType::YieldType(), replace with Yield() (Jon Siwek, Corelight) * Change base_type() to return const-ref, deprecate base_type_no_ref() (Jon Siwek, Corelight) * Deprecate BroType::GetField() and BroType::HasField() (Jon Siwek, Corelight) * Deprecate RecordType::FieldType(), replace with GetFieldType() (Jon Siwek, Corelight) * Migrate TypeList to store IntrusivePtrs This changes return types of TypeList::Types() and IndexType::IndexTypes() to return std::vector instead of type_list* (Jon Siwek, Corelight) * Deprecate TypeList::PureType(), replace with TypeList::GetPureType() (Jon Siwek, Corelight) * Deprecate SetType::SetElements(), replace with SetType::Elements() (Jon Siwek, Corelight) * Remove unused TableType::ExpandRecordIndex() (Jon Siwek, Corelight) * Add cast_intrusive() and make use of it in two spots (Jon Siwek, Corelight) * Give make_intrusive() access to protected EnumVal ctor (Jon Siwek, Corelight) * Remove two superfluous IntrusivePtr NewRefs in Type.cc (Jon Siwek, Corelight) * Change BroType::ShallowClone() to return IntrusivePtr (Jon Siwek, Corelight) * Migrate IP.cc to use IntrusivePtr (Jon Siwek, Corelight) * Migrate SMB analyzer to use IntrusivePtr Deprecates the utf16_bytestring_to_utf8_val() function with replacement being utf16_to_utf8_val(). (Jon Siwek, Corelight) * Migrate ARP analyzer to use IntrusivePtr (Jon Siwek, Corelight) * Migrate HTTP/MIME analyzers to use IntrusivePtr (Jon Siwek, Corelight) * Migrate ICMP analyzer to use IntrusivePtr (Jon Siwek, Corelight) * Migrate DNS analyzer to use IntrusivePtr (Jon Siwek, Corelight) * Migrate Tag classes to use IntrusivePtr Deprecates various methods that previously took raw pointers (Jon Siwek, Corelight) * Deprecate TableVal::ConvertToList() and TableVal::ConvertToPureList() Replaced with ToListVal() and ToPureListVal() that return IntrusivePtr (Jon Siwek, Corelight) * Switch ListVal to store IntrusivePtrs * Deprecates ListVal::Index() methods and replaces with ListVal::Idx() * Replaces ListVal::Vals() method with one that returns std::vector> rather than val_list (Jon Siwek, Corelight) * Deprecant ListVal::Append(Val*) and add IntrusivePtr version (Jon Siwek, Corelight) * Deprecate ListVal::ConvertToSet(), add ListVal::ToSetVal() (Jon Siwek, Corelight) * Deprecate TunnelEncapsulation BuildRecordVal/BuildVectorVal methods Replaced with ToVal methods that return IntrusivePtr (Jon Siwek, Corelight) * Deprecate various IP/packet header Val-building methods And supply new alternatives that use IntrusivePtr (Jon Siwek, Corelight) * Update submodule(s) [nomail] (Jon Siwek, Corelight) 3.2.0-dev.573 | 2020-05-29 17:13:36 -0700 * Add "Known::service_udp_requires_response" option (Jon Siwek, Corelight) Determines whether to require UDP server response before considering an active service to log in known_services.log. * Update known-services.zeek logic for "active" TCP services (Jon Siwek, Corelight) To better check for only endpoints that either have observed a TCP handshake or else assumed to have done one in the past (partial connections, missed the handshake, but now see data/acks from server) * Log services with unknown protocol names (Michael Dopheide) * GH-989: Fix crash on using some deprecated environment variables (Jon Siwek, Corelight) 3.2.0-dev.570 | 2020-05-28 14:49:10 -0700 * Add negotiation flags parameter to some RDP events (Anthony Kasza, Corelight) Namely rdp_connect_request, rdp_negotiation_response, and rdp_negotiation_failure * Update test baselines for new Broker connection status/error strings (Jon Siwek, Corelight) 3.2.0-dev.565 | 2020-05-26 21:55:54 +0000 * Add DCE-RPC constants from BZAR project (V) 3.2.0-dev.562 | 2020-05-26 11:06:34 -0700 * GH-979: Update libkqueue to fix use on CentOS 6 (Jon Siwek, Corelight) 3.2.0-dev.561 | 2020-05-21 13:05:39 -0700 * GH-983: Fix opaque Broker types lacking a Type after (de)serialization (Jon Siwek, Corelight) 3.2.0-dev.559 | 2020-05-21 13:04:19 -0700 * Make SendEvent callable from all threads This commit refactors the SendEvent call and moves it from the Input ReaderBackend to to MsgThread. This allows all other types of threads to access this functionality. This necessitated a few more changes. Most importantly, one of the ValueToVal methods was moved over to SerialTypes. Whereit arguably belongs - there was nothing that was input-framework specific in that method - and the functionality could come in useful in a number of cases. (Johanna Amann, Corelight) 3.2.0-dev.557 | 2020-05-21 11:41:12 -0700 * Speed up FuzzBuffer ChunkCount validity check (Justin Azoff, Corelight) 3.2.0-dev.555 | 2020-05-20 11:19:08 -0700 * Disable output of Reporter messages to stderr in fuzz targets (Jon Siwek, Corelight) 3.2.0-dev.554 | 2020-05-20 10:56:46 -0700 * Improve standalone fuzz driver timing output (Jon Siwek, Corelight) * Skip fuzz inputs that have more than 64 chunks (Justin Azoff, Corelight) 3.2.0-dev.550 | 2020-05-19 10:50:42 -0700 * Upgrade to latest Broker changes for CAF 0.18 (Dominik Charousset, Corelight) * Include pcap.h instead of pcap/dlt.h in packet-fuzzer (Tim Wojtulewicz, Corelight) 3.2.0-dev.547 | 2020-05-18 10:47:54 -0700 * add packet fuzzer (Justin Azoff) * Fix building fuzz targets on macOS (Jon Siwek, Corelight) * Highwayhash: small build fix (Johanna Amann, Corelight) Turns out that hh_neon should not be compiled on generic arm CPUs. That one is only for aarch64. 3.2.0-dev.539 | 2020-05-15 19:47:55 +0000 * Replace bzero() with memset() (Noah Treuhaft) * Change Coverity action to output build log (Jon Siwek, Corelight) 3.2.0-dev.530 | 2020-05-13 15:05:31 -0700 * GH-906: Fix the regex in url.zeek to better match for find_all_urls (Tim Wojtulewicz, Corelight) 3.2.0-dev.526 | 2020-05-13 13:49:29 -0700 * Hash unification: address PR feedback (Johanna Amann, Corelight) * Update architecture checks for highwayhash Now we use cmake to determine which compiler flags are set to determine which implementation of highwayhash to compile. This should be much more robust. Also - fix missing include on old ubuntus. (Johanna Amann, Corelight) * Highwayhas: unbreak 32 bit compiles (Johanna Amann, Corelight) * Update external test suites for hash changes. (Johanna Amann, Corelight) * Hashing: Remove unnecessary include (Johanna Amann, Corelight) * Merge remote-tracking branch 'origin/master' into topic/johanna/hash-unification (Johanna Amann, Corelight) * Add highwayhash in a more cmake-y way (Johanna Amann, Corelight) * Remove highwayhash submodule. It is moving to 3rdparty... (Johanna Amann, Corelight) * Merge remote-tracking branch 'origin/master' into topic/johanna/hash-unification (Johanna Amann, Corelight) * Switch file UID hashing from md5 to highwayhash. This commit switches UID hashing from md5 to a highway hash. It also moves the salt value out of the file plugin - and makes it installation-specific instead - it is moved to the global namespace. There now are digest hash functions to make "static" installation-specific hashes that are stable over workers available to everyone; hashes can be 64, 128 or 256 bits in size. Due to the fact that we switch the file hashing algorithm, all file hashes change. The underlyigng algorithm that is used for hashing is highwayhash-128, which is significantly faster than md5. (Johanna Amann, Corelight) * Switch most internal md5 calls to digest calls. The places that used md5 basically already used it as a digest algorithm. Switching to a digest just means that the internal values used to not change between runs - which is actually wanted in these cases. This commit also removes our special cmake subdirectory. We don't expose highwayhash in headers anymore - so we can just treat it as an internal implementation choice that is not directly exposed to plugins. (Johanna Amann, Corelight) * Start refactoring hashing. This commit moves some of the hash datastructures and code from util.cc into Hash.cc - where it seems more appropriate. It also starts to make more Keyed hash functions available - still using siphash as the default 64 bit keyed hash, but also making 128 and 256 bit highway hashes available. There already are a few other functions that are defined but not yet implemented - these will be "static" keyed hashes - which use an installation specific key. These will be used to, e.g., get rid of md5 hashing for the generation of file UIDs. (Johanna Amann, Corelight) * Introduce calculate_digest convenience function This function just calculates the chosen digest and returns the result in either the passed buffer, or in a static buffer. Basically a superset to the surprisingly popular internal_md5. (Johanna Amann, Corelight) * Remove the siphash->hmac-md5 switch after 36 bytes. Currently, siphash is used for strings up to 36 bytes. hmac-md5 is used for longer strings. This switch-over is a remnant of the previous hash-function that was used, which apparently was slower with longer input strings. This change serves no purpose anymore. I performed a few performance tests on strings of varying sizes: For a 40 byte string with 10 million iterations: siphash: 0.31 seconds hmac-md5: 3.8 seconds For a 1080 byte string with 10 million iterations: siphash: 4.2 seconds hmac-md5: 17 seconds For a 18360 byte string with 10 million iterations: siphash: 69 seconds hmac-md5: 240 seconds Hence, this commit removes the use of hmac-md5. This change causes reordering of lines in a few logs. This commit also changes the datastructure for the seed in probabilistic/Hasher to get rid of a type-punning warning. (Johanna Amann, Corelight) * Fix plugin compile problems with highwayhash The way this is included is still hacky as hell - but now all tests pass. (Johanna Amann, Corelight) * Replace siphash with Google implementation This adds the entirety of the highwayhash implementation of Google. This includes siphash as well as severl highwayhash variants - which are faster. This first commit only switches out the siphash implementation. All hashes that are generated are exactly the same as before. However, this does make all other hashes available to be used by us. I did some performance tests vs the previous siphash implementation by running the 2009-M57-day11-18 trace 100x through both cases. The average runtime was virtually the same (within 0.014 seconds of each other). Note that the way that I included the highwayhash implementation in our cmake setup is... well, let's say hacky. This definitely needs to be changed a bit before including this in a real build. (Johanna Amann, Corelight) 3.2.0-dev.508 | 2020-05-13 11:16:05 -0700 * Disable broker tests from building on the CI (Tim Wojtulewicz, Corelight) This results in a ~10% reduction in build time. * Disable broker documentation examples if disabling broker tests (Tim Wojtulewicz, Corelight) 3.2.0-dev.505 | 2020-05-12 18:52:19 -0700 * Use zeek::detail namespace for fuzzer utils (Jon Siwek, Corelight) * Set terminating flag during fuzzer cleanup (Jon Siwek, Corelight) * Add missing include to standalone fuzzer driver (Jon Siwek, Corelight) * Improve standalone fuzzer driver error messages (Jon Siwek, Corelight) * Merge branch 'master' into topic/jsiwek/fuzzing (Jon Siwek, Corelight) * Test fuzzers against seed corpus under CI ASan build (Jon Siwek, Corelight) * Update fuzzing README with OSS-Fuzz integration notes (Jon Siwek, Corelight) * Link fuzzers against shared library to reduce executable sizes (Jon Siwek, Corelight) * Improve FuzzBuffer chunking Now allocates a new buffer for each chunk to better detect over-reads (Jon Siwek, Corelight) * Fix compiler warning in standalone fuzzer driver (Jon Siwek, Corelight) * Adjust minor fuzzing documentation (Jon Siwek, Corelight) * Exit immediately after running unit tests (Jon Siwek, Corelight) * Add OSS-Fuzz Zeek script search path to fuzzers (Jon Siwek, Corelight) * Assume libFuzzer when LIB_FUZZING_ENGINE file doesn't exist i.e. environment variable may be set, but not point to existing file (Jon Siwek, Corelight) * Change handling of LIB_FUZZING_ENGINE Should better support OSS-Fuzz, which may set it to either be path to library to link against or linker flag like "-fsanitize=fuzzer" (Jon Siwek, Corelight) * Change --enable-fuzzing to --enable-fuzzers Since it controls whether to build the fuzzer targets, not whether those fuzzer targets actually use a fuzzing engine. (Jon Siwek, Corelight) * Add standalone driver for fuzz targets Useful for cases that don't need to use a fuzzing engine, but just run the fuzz targets over some set of inputs, like for regression/CI tests. Also added a POP3 fuzzer dictionary, seed corpus, and README with examples. (Jon Siwek, Corelight) * Add basic structure for fuzzing targets General changes: * Add -D/--deterministic command line option as convenience/alternative to -G/--load-seeds (i.e. no file needed, it just uses zero-initialized random seeds). It also changes Broker data stores over to using deterministic timing rather than real time. * Add option to make Reporter abort on runtime scripting errors (Jon Siwek, Corelight) 3.2.0-dev.486 | 2020-05-11 11:11:51 -0700 * Limit rate at which MMDB error/status messages are emitted If there's some bad state we can be in where MMDB lookup/open operations consistently fail, then the volume of associated reporter messages can get overwhelmingly large especially if a lookup operation is being done for each network connection. This adds a limit of an arbitrary 20 messages every 5 minutes, which should be enough information to understand the overall open/close/lookup-failure pattern. (Jon Siwek, Corelight) 3.2.0-dev.484 | 2020-05-08 11:50:54 -0700 * Change timer_list in BroList to be an unordered list. (Tim Wojtulewicz, Corelight) This type is used by Conn and Analyzer to hold onto timers being added and removed. We don't expect the elements in those lists to maintain an order as the list is being modified. * Add ability for List to be ordered/unordered (Tim Wojtulewicz, Corelight) This fixes a "bug" with List where remove_nth() can be an O(n) operation when it doesn't need to be. remove_nth for lists that don't necessarily need to keep an order can be an O(1) operation instead. * Add unit testing for List (Tim Wojtulewicz, Corelight) * Fix bug with List where replace() doesn't work with non-pointer types (Tim Wojtulewicz, Corelight) 3.2.0-dev.478 | 2020-05-08 11:47:38 -0700 * Added examples to set_to_regex comments (James Lagermann, Corelight) * Unbreak build on Fedora 32 (gcc 10.0.1) (Johanna Amann, Corelight) It requires cstdint in a few more headers. 3.2.0-dev.475 | 2020-05-07 17:15:23 -0700 * GH-958: Fix crash when trying to redef non-existing enum (Johanna Amann, Corelight) 3.2.0-dev.473 | 2020-05-06 10:40:09 -0700 * Revert addition of final modifier to JSON formatter (Tim Wojtulewicz, Corelight) 3.2.0-dev.471 | 2020-05-06 10:00:58 -0700 * Fix global buffer over-read in POP3 analyzer (Justin Azoff, Corelight) * Fix SSL scripting error leading to access of unitialized field (Jon Siwek, Corelight) Reported by Justin Azoff * Remove outdated comment on set_to_regex. (Johanna Amann, Corelight) We can add patterns at runtime since 2.6. 3.2.0-dev.467 | 2020-05-04 18:00:35 -0700 * GH-952: Correct spelling of DCE/RPC operation string NetrLogonSameLogonWithFlags (Jon Siwek, Corelight) 3.2.0-dev.466 | 2020-05-04 17:50:14 -0700 * Add network_time_init() event. (Jan Grashoefer) This event is generated upon first initialization of network_time. 3.2.0-dev.461 | 2020-05-04 17:08:46 -0700 * Avoid scheduling multiple inactivity timers (Justin Azoff and Jon Siwek, Corelight) Also updated language.expire_subnet btest which is unduly sensitive to timer-related changes 3.2.0-dev.459 | 2020-05-01 17:46:20 -0700 * Extend CI config to cover building with libmaxminddb support (Jon Siwek, Corelight) * Ensure time continues moving forward if a pcap source is suspended (Tim Wojtulewicz, Corelight) 3.2.0-dev.455 | 2020-05-01 09:44:30 -0700 * GH-938: fix IO loop iterations sometimes skipping offline pcap sources (Jon Siwek, Corelight) 3.2.0-dev.451 | 2020-04-29 16:28:34 -0700 * Organized and added to the shipped file identification signatures. (Seth Hall, Corelight) - Added ISO 9660 disk image - Created new files for categorizing signatures better. - executable.sig - Executable (and bytecode) files. - java.sig - Java related files (class/jar, etc). - programming.sig - Mostly scripting language identification 3.2.0-dev.447 | 2020-04-29 15:55:03 -0700 * GH-713: Fixed misc/stats.zeek skipping a log entry on termination (Brittany Donowho) 3.2.0-dev.445 | 2020-04-29 15:25:03 -0700 * Add warning message for unknown Broker statuses (Jon Siwek, Corelight) There's now a couple placeholder/unimplemented status values in Broker related to upcoming routing features that we don't want to handle explicitly for compatibility reasons, but also don't want the compiler warning about unhandled values in the switch. 3.2.0-dev.443 | 2020-04-28 17:10:38 -0700 * GH-941: Fix build when configured to use libmaxminddb (Jon Siwek, Corelight) 3.2.0-dev.441 | 2020-04-27 13:34:22 -0700 * Fix a few more IntrusivePtr deprecation warnings (Tim Wojtulewicz, Corelight) * Fix cloning of TypeType values (Vern Paxson, Corelight) 3.2.0-dev.437 | 2020-04-27 19:30:24 +0000 * GH-854: provide access to original HTTP/MIME header names The "http_header" event now has an "original_name" parameter that allows access to the original header name (the "name" parameter reamins the same as before: it's the uppercased header name). The "mime_header_rec" record type now also includes an "original_name" field to similarly provide access to original header name in the following events: "http_all_headers", "mime_one_header", and "mime_all_headers". (Jon Siwek, Corelight) * Remove error message from empty bloomfilter lookups If a bloomfilter doesn't have a type, that just means no bloomfilter_add() has been called yet, so seems undesirable to emit an error for a lookup against something that's known to be empty. (Jon Siwek, Corelight) * unused variables found via use-def analysis (plus an indentation micro-nit) (Vern Paxson, Corelight) 3.2.0-dev.431 | 2020-04-27 12:09:30 -0700 * Update various BIFs to return IntrusivePtr (Jon Siwek, Corelight) 3.2.0-dev.428 | 2020-04-24 16:19:45 -0700 * Deprecate returning Val* from BIFs (Jon Siwek, Corelight) * Deprecate binpac::string_to_val (Jon Siwek, Corelight) * Deprecate binpac::bytestring_to_val, replace with binpac::to_stringval (Jon Siwek, Corelight) * Update deprecated BifEvent::generate_* usages (Jon Siwek, Corelight) * Deprecate Connection::Event and Analyzer::Event methods And update usages to the "EnqueueEvent" methods. (Jon Siwek, Corelight) * Deprecate BuildConnVal() methods and update usages to ConnVal() The later being a new method that returns IntrusivePtr (Jon Siwek, Corelight) * Update all BIFs to return IntrusivePtr instead of Val* (Jon Siwek, Corelight) * Update deprecated ValManager::GetPort usages (Jon Siwek, Corelight) * Update deprecated ValManager::GetEmptyString usages (Jon Siwek, Corelight) * Update deprecated ValManager::GetCount usages (Jon Siwek, Corelight) * Update deprecated ValManager::GetInt usages (Jon Siwek, Corelight) * Update deprecated ValManager::GetBool usages (Jon Siwek, Corelight) * Update deprecated ValManager GetTrue/GetFalse usages (Jon Siwek, Corelight) * Deprecate all ValManager "Get" methods Alternate methods that return IntrusivePtr are available in similarly named methods that omit the "Get" prefix. (Jon Siwek, Corelight) * Change BIFs to return a wrapper object That allows returning either Val* or IntrusivePtr. The former could eventually be deprecated, but it's used extensively at the moment. (Jon Siwek, Corelight) 3.2.0-dev.412 | 2020-04-22 10:43:39 -0700 * Fix buffer over-read in Ident analyzer (Max Kellermann) 3.2.0-dev.410 | 2020-04-21 10:25:29 -0700 * Remove "connection_external" event (Johanna Amann, Corelight) This event was forgotten in our broccoli cleanup. It cannot be raised by anything anymore 3.2.0-dev.405 | 2020-04-16 15:38:46 -0700 * Remove ineffective &default in netcontrol cluster event handler args (Jon Siwek, Corelight) 3.2.0-dev.404 | 2020-04-16 14:12:20 -0700 * GH-895: Remove use of Variable-Length-Arrays (Jon Siwek, Corelight) 3.2.0-dev.402 | 2020-04-16 13:00:59 -0700 * Add default function for Kerberos constant-lookup-tables (Jon Siwek, Corelight) * check for the existance of f?$conns in file_sniff event in policy/protocols/ssl/log-hostcerts-only.zeek (SG) * Update README.md to fix the logo and one of the links (Tim Wojtulewicz, Corelight) 3.2.0-dev.396 | 2020-04-14 15:45:54 -0700 * A few minor cleanups in Dict (Tim Wojtulewicz, Corelight) 3.2.0-dev.394 | 2020-04-14 15:29:00 -0700 * Fix a confusing variable name shadowing (Jon Siwek, Corelight) 3.2.0-dev.392 | 2020-04-14 11:06:04 -0700 * Fix Stack Overflow in POP3_Analyzer::ProcessRequest. (Johanna Amann, Corelight) The VLA can overflow given a large enough string. As a small fix, this commit gets rid of the VLA and assigns the password directly to the target string. This was reported by Matteo Rizzo (Google). 3.2.0-dev.391 | 2020-04-13 14:59:50 -0700 * Add warning for ineffective &default arguments in handlers For event/hook handlers that had a previous declaration, any &default arguments are ineffective. Only &default uses in the initial prototype's arguments have an effect (that includes if the handler is actually the site at which the declaration occurs). (Jon Siwek, Corelight) * Fix frame size allocation of alternate event/hook handlers When the only existing handler doesn't use the full argument list, still have to ensure the full frame is allocated because all arguments get set regardless. (Jon Siwek, Corelight) * Emit error for alternate event/hook prototype args with attributes Argument attributes are only allowed in the canonical prototype. (Jon Siwek, Corelight) * Improve alternate event/hook prototype matching This fixes it to again allow the old behavior of matching a handler against the canonical prototype as long as all argument types, but not necessarily names, match. (Jon Siwek, Corelight) * Allow alternate event/hook prototype declarations The alternates must be some subset of the canonical prototype (the one that's first declared) and allows users to define handlers for any such prototype. Example: # Prototype declarations global my_event: event(s: string, c: count); global my_event: event(c: count); global my_event: event(); # Handler definitions event my_event(s: string, c: count) { print s, c; } event my_event(c: count) { print c; } event my_event() { } This allows handlers to consume a subset of the arguments or even re-order them. This makes it easier to either extend an existing event/hook's arguments and/or deprecate usages of certain prototypes. (Jon Siwek, Corelight) * Fix Broker topics used to uniquely identify cluster nodes Node-specific topic prefix subscriptions/publications now add a trailing slash like "zeek/cluster/node//". Without the trailing slash, messages attempting to target "proxy-10" may also be sent to "proxy-1" since subscription matching is prefix-based. (Jon Siwek, Corelight) * Update submodule(s) [nomail] (Jon Siwek, Corelight) * Fix include in X509Common.h Give full path to Analyzer.h, so that X509Common.h is include-able by plugins. (Johanna Amann, Corelight) 3.2.0-dev.382 | 2020-04-09 13:17:03 -0700 * Remove 'using namespace std' as well as other using statements from headers. This unfortunately cuases a ton of flow-down changes because a lot of other code was depending on that definition existing. This has a fairly large chance to break builds of external plugins, considering how many internal ones it broke. (Tim Wojtulewicz, Corelight) 3.2.0-dev.378 | 2020-04-09 08:47:44 -0700 * Replace most of the uses of 0 or NULL to indicate null pointers with nullptr. This change does not change any calls to syscalls, in the interest of passing what the API for those calls says to pass. (Tim Wojtulewicz, Corelight) 3.2.0-dev.372 | 2020-04-08 14:00:28 -0700 * Lazy-initalize some of the fields in Frame to reduce size (Tim Wojtulewicz, Corelight) * Set InternalHashTag to a uint16_t so CompositeHash doesn't have a gap in it. Resizes CompositeHash from 32 bytes to 24 bytes. (Tim Wojtulewicz, Corelight) * Mark constants in List constexpr so they don't actually take up space in created objects This resizes List from 24 bytes to 16 bytes. (Tim Wojtulewicz, Corelight) * Reorder some class variables to fill in gaps in structure packing (Tim Wojtulewicz, Corelight) * Mark classes as final, where appropriate. (Tim Wojtulewicz, Corelight) These classes were recommended by gcc's -Wsugggest-final-types as places where large numbers of virtual functions could be optimized away. 3.2.0-dev.360 | 2020-04-07 14:12:27 -0700 * Add "udp_content_ports" option Any port in that set found as either source or destination port of a UDP packet will cause the "udp_contents" event to be raised. (Jon Siwek, Corelight) * Add new "udp_content_delivery_ports_use_resp" option This controls whether ports given by "udp_content_delivery_ports_orig" and "udp_content_delivery_ports_orig" are in terms of the UDP packet's destination port or by the Connection's "responder" port (the former is the unchanged default behavior). (Jon Siwek, Corelight) 3.2.0-dev.357 | 2020-04-07 13:43:38 -0700 * Fix connection duration thresholding Connection durations were being compared "less than" instead of "greater than" the desired duration threshold as provided to set_current_conn_bytes_threshold() or ConnThreshold::set_duration_threshold() (Jon Siwek, Corelight) 3.2.0-dev.355 | 2020-04-07 12:05:17 -0700 * Fix using patterns stored as table/set indices Pattern values used as table/set indices are stored in serialized form using just the pattern text, but re-creating the value from that didn't fully initialize/compile the regex matcher after (re-)creating it from that pattern text. (Jon Siwek, Corelight) 3.2.0-dev.353 | 2020-04-06 17:27:35 -0700 * Improve parsing of RDPEUDP/RDPEUDP2 (Anthony Kasza, Corelight) 3.2.0-dev.338 | 2020-04-02 18:29:23 -0700 * Add RDP over UDP analyzer (Anthony Kasza, Corelight) 3.2.0-dev.336 | 2020-04-02 15:03:04 -0700 * Fix uses of GetBool in bifs to use GetTrue/GetFalse (Tim Wojtulewicz, Corelight) * Change Sessions::dump_this_packet to a bool (Tim Wojtulewicz, Corelight) 3.2.0-dev.332 | 2020-03-31 15:34:52 -0700 * GH-725: fix logic for finding a lambda's usage of outer IDs (Jon Siwek, Corelight) * Change record field anonymous functions to use lambda expressions There was an alternate syntax to assign anonymous functions to record fields that was never migrated to use the new lambda expression machinery (and so didn't allow referencing variables in outer scope): type myrec: record { foo: function(a: string); }; local o = "o"; local mr = myrec($foo(a: string) = { print a + o; }); (Jon Siwek, Corelight) 3.2.0-dev.329 | 2020-03-31 08:48:10 -0700 * Mark input/output message classes as final, since nothing should be inheriting from them (Tim Wojtulewicz, Corelight) * Add missing override specifier to a number of methods, remove virtual from some as well (Tim Wojtulewicz, Corelight) * Add override specifier to Configure() method in almost all of the internal plugins (Tim Wojtulewicz, Corelight) 3.2.0-dev.325 | 2020-03-31 07:04:59 +0000 * Fix bug in intel framework letting deletes of mixed-cased entries fail. (Antonio Nappa, Corelight) 3.2.0-dev.322 | 2020-03-31 06:44:57 +0000 * A large number of functions had return values and/or arguments changed to use ``bool`` types instead of ``int``. (Tim Wojtulewicz, Corelight) 3.2.0-dev.320 | 2020-03-30 14:09:23 -0700 * GH-887: improve GRE/ERSPAN parsing of non-IPv4/IPv6 inner payload This changes the decapsulation logic for GRE/ERSPAN payloads to re-use existing Layer 2 parsing logic that already handles things like 802.1Q tags correctly before going on to process the inner IPv4/IPv6 payload. (Jon Siwek, Corelight) * Fix ERSPAN III optional sub-header parsing (Jon Siwek, Corelight) * Many small cleanups of Expr.cc (Tim Wojtulewicz, Corelight) * Remove unused cq queue code (Tim Wojtulewicz, Corelight) * Add a "reserved" field to tcp_hdr record This record is available by handling either raw_packet or new_packet events. (Jon Siwek, Corelight) * GH-878: Make RocksDB usage opt-in and add --enable-rocksdb configure flag (Jon Siwek, Corelight) 3.2.0-dev.300 | 2020-03-27 11:48:46 -0700 * Func::DescribeDebug: move a NumFields() call out of loop (Jon Siwek, Corelight) * Use const-ref parameter for zeek::val_list_to_args() It ended up being used a bit more than initially expected and this is closer to the style we're generally aiming for. (Jon Siwek, Corelight) * Fix missing IntrusivePtr.h include and ambiguous ODesc::Add call (Jon Siwek, Corelight) * Remove TimerMgr arg from event queuing/scheduling methods It's not useful for anything since there's only ever a single TimerMgr. (Jon Siwek, Corelight) * Deprecate Analyzer::ConnectionEvent() And update usages to Analyzer::EnqueueConnEvent() (Jon Siwek, Corelight) * Deprecate file_analysis::File::FileEvent methods using val_list args And update usages to the overload that takes a zeek::Args instead. (Jon Siwek, Corelight) * Deprecate Connection::ConnectionEvent methods And update usages to Connection::EnqueueEvent (Jon Siwek, Corelight) * Deprecate EventMgr::QueueEventFast() and update usages to Enqueue() (Jon Siwek, Corelight) * Deprecate EventMgr::QueueEvent() and update usages to Enqueue() (Jon Siwek, Corelight) * Deprecate Func::Call(val_list*, ...) The version taking a vector of intrusive pointers should be used instead. A variadic version is also added that forwards all arguments. (Jon Siwek, Corelight) * Use vector> for Func::Call and Event queuing args This change may break BIFs that use @ARGS@, @ARG@, or @ARGC@ since their types have changed. (Jon Siwek, Corelight) * Fix memory leak in Zeek when-statement bodies with runtime errors (Jon Siwek, Corelight) * Change TableVal::RecoverIndex() to return IntrusivePtr (Jon Siwek, Corelight) * Use IntrusivePtr in TableVal::CallExpireFunc (Jon Siwek, Corelight) * Fix memory leak when runtime error occurs in a Zeek for-loop (Jon Siwek, Corelight) * Enable leak checks for btests that produce runtime exceptions These were previously reporting leaks due to various allocations not getting cleaned up during the stack unwind, but at the current state of the transition toward IntrusivePtr usage, theses tests no longer leak. (Jon Siwek, Corelight) 3.2.0-dev.280 | 2020-03-24 13:49:43 -0700 * Add test case for binpac flowbuffer frame length parsing bug (Jon Siwek, Corelight) 3.2.0-dev.277 | 2020-03-20 11:29:36 -0700 * Initialize RecordVal default fields when redef'd (Jon Siwek, Corelight) If a RecordVal had been created, but later its RecordType redef'd to contain fields with &default, those fields were incorrectly left uninitialized. * GH-857: fix redefining record types used to index tables (Jon Siwek, Corelight) This change tracks all TableVals created at parse-time whose index depends on a given RecordType. Should that RecordType be redef'd, those TableVals are immediately rebuilt such that they are valid to subsequently use in either parse-time initializations or eventually in any arbitrary run-time expression. * Change RecordVals to get resized at time of RecordType redef (Jon Siwek, Corelight) Opposed to unconditionally checking all RecordVals whether they need to be resized after parsing ends. 3.2.0-dev.273 | 2020-03-20 10:05:21 -0700 * GH-865: fix parsing of SMB NegotiateContextList * The compression capability was incorrectly set to 0x0004 instead of 0x0003 * The padding was 4-byte instead of 8-byte aligned and also the spec. does not strictly require the padding for the last item in the list. * Add a default case to handle parsing of unknown context types. (Jon Siwek, Corelight) 3.2.0-dev.271 | 2020-03-17 22:52:40 -0700 * Scope: store IntrusivePtr in `local` (Max Kellermann) * Scope: pass IntrusivePtr to AddInit() (Max Kellermann) * DNS_Mgr: use class IntrusivePtr (Max Kellermann) * Scope: use class IntrusivePtr (Max Kellermann) * Attr: use class IntrusivePtr (Max Kellermann) * Expr: check_and_promote_expr() returns IntrusivePtr (Max Kellermann) Instead of returning a pseudo-boolean integer, it now returns a referenced object or nullptr on error. The old API was very error prone because of its obscure reference counting semantics. * Frame: use class IntrusivePtr (Max Kellermann) * Val: RecordVal::LookupWithDefault() returns IntrusivePtr (Max Kellermann) * Type: RecordType::FieldDefault() returns IntrusivePtr (Max Kellermann) Fixes memory leak in EventHandler::NewEvent(). * Val: TableVal::Delete() returns IntrusivePtr (Max Kellermann) * Type: base_type() returns IntrusivePtr (Max Kellermann) * Type: init_type() returns IntrusivePtr (Max Kellermann) * Type: merge_types() returns IntrusivePtr (Max Kellermann) * Type: use class IntrusivePtr in VectorType (Max Kellermann) * Type: use class IntrusivePtr in EnumType (Max Kellermann) * Type: use class IntrusivePtr in FileType (Max Kellermann) * Type: use class IntrusivePtr in TypeDecl (Max Kellermann) * Type: make TypeDecl `final` and the dtor non-`virtual` (Max Kellermann) * Type: use class IntrusivePtr in TypeType (Max Kellermann) * Type: use class IntrusivePtr in FuncType (Max Kellermann) * Type: use class IntrusivePtr in TypeList (Max Kellermann) * Type: use class IntrusivePtr in IndexType (Max Kellermann) * Val: use class IntrusivePtr in class TableVal (Max Kellermann) * Val: use class IntrusivePtr in class TableEntryVal (Max Kellermann) * CompHash: return IntrusivePtr (Max Kellermann) * Type: return IntrusivePtr (Max Kellermann) * Val: add TableVal::Assign() overload with IntrusivePtr (Max Kellermann) * zeekygen/IdentifierInfo: use class IntrusivePtr more (Max Kellermann) * Func: use class IntrusivePtr (Max Kellermann) * GH-845: reference `id_list` before passing to new `BroFunc` (Max Kellermann) Passing the `id_list` pointer to `BroFunc` transfers ownership of the contained `ID` instances, because `~BroFunc()` unreferences them. Therefore, we need to increase the reference counters for each `BroFunc` instance to fix the use-after-free bug. * input/Manager: fix three use-after-free bugs (Max Kellermann) 3.2.0-dev.237 | 2020-03-17 16:54:35 -0700 * Cleanup in iosource/Packet (Tim Wojtulewicz, Corelight) * Add ability for packet sources to flag a packet's l2 or l3 checksum as valid. This lets packet source plugins implement handling of hardware checksum offloading, if available. Setting the flags will skip the internal checksumming for either layer 2 and/or layer 3. (Tim Wojtulewicz, Corelight) 3.2.0-dev.234 | 2020-03-16 12:37:17 -0700 * Remove an unused smb2_negotiate_response event handler. (M.Shirk) 3.2.0-dev.231 | 2020-03-13 13:41:23 -0700 * Add branch whitelist for CI notifications (Jon Siwek, Corelight) 3.2.0-dev.230 | 2020-03-12 13:33:09 -0700 * Introduce X509 certificate caching. This change introduces caching of X509 certificates. If a certificate is encountered more than 10 times in a minute, it will be cached for the future (until it is not seen for more than a minute). The caching means that the base scripts cache the certificate information and use cached information to raise events. This means certificates do not have to be repeatedly parsed; furthermore the events are passed "recycled" Vals (so data is not repeatedly copied around). This new feature is highly configureable - see the x509 main.zeek for full information. You can both change the timeouts, disable the feature, or even stop X509 events from being raised (by hooking X509::x509_certificate_cache_replay). (Johanna Amann, Corelight) 3.2.0-dev.223 | 2020-03-12 12:03:48 -0700 * Update pointer to external test repo (Jon Siwek, Corelight) 3.2.0-dev.222 | 2020-03-11 16:39:53 -0700 * Fix whitespace in connection record comments (Jon Siwek, Corelight) 3.2.0-dev.221 | 2020-03-11 11:21:20 -0700 * Made additional MySQL fixes. (Vlad Grigorescu) 1) There are a couple more places where the new protocol uses and OK packet instead of the deprecated EOF. 2) With > 255 results, we could end up in an situation where the uint8 sequence number would wrap, and we'd naively think it'd be a new handshake. Now, we track the previous sequence number, and assume overflow if it was 255 previously and 0 now. We also reset the previous sequence number to 0 in various packets that we'd expect at the end of other commands. * Add support to MySQL for deprecation of EOF packets. (Vlad Grigorescu) From the docs: "As of MySQL 5.7.5, OK packes are also used to indicate EOF, and EOF packets are deprecated." The client sets a capability flag (CLIENT_DEPRECATE_EOF) to indicate that it expects an OK instead of an EOF after the resultset rows. * MySQL analyzer whitespace cleanup (Vlad Grigorescu) * Fix EOF detection in the MySQL protocol analyzer. (Vlad Grigorescu) The MySQL documentation (https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_eof_packet.html) warns us that "You must check whether the packet length is less than 9 to make sure that it is a EOF_Packet packet." While we were doing this in two places, we were comparing the total packet length, which includes the 4-byte header. Changed to compare to 13 instead. 3.2.0-dev.214 | 2020-03-09 13:35:26 -0700 * Stop running GitHub Actions in forked repos (Jon Siwek, Corelight) * Remove Travis CI configuration (Jon Siwek, Corelight) * Add GitHub Action for Coverity Scan (Jon Siwek, Corelight) 3.2.0-dev.209 | 2020-03-06 16:27:29 -0800 * Add urgent flag to tcp_flags documentation (Tim Wojtulewicz, Corelight) 3.2.0-dev.205 | 2020-03-06 07:59:08 -0800 * Move libkqueue submodule to the aux directory (Jon Siwek, Corelight) * Move rapidjson submodule to aux directory This also now installs the full rapidjson include/ tree in order to allow external plugins to properly build if they include threading/formatters/JSON.h (Jon Siwek, Corelight) * GH-831: Fix the "bro" symlink for binary packaging mode. (Jon Siwek, Corelight) 3.2.0-dev.198 | 2020-03-05 14:18:10 -0800 * Fix a btest to order zeek_init() events via &priority (Jon Siwek, Corelight) 3.2.0-dev.192 | 2020-02-28 11:36:12 -0800 * threading::Queue::WakeUp(): lock mutex before notifying condition_variable (Max Kellermann) Not locking the associated mutex can create race conditions and lockups. 3.2.0-dev.190 | 2020-02-28 00:42:17 -0800 * Stmt: use class IntrusivePtr (Max Kellermann) * Stmt: remove unused default constructors and `friend` declarations (Max Kellermann) * Val: remove unimplemented prototype recover_val() (Max Kellermann) * Val: cast_value_to_type() returns IntrusivePtr (Max Kellermann) * Val: use IntrusivePtr in check_and_promote() (Max Kellermann) * Val: use nullptr instead of 0 (Max Kellermann) * zeekygen: use class IntrusivePtr (Max Kellermann) * ID: use class IntrusivePtr (Max Kellermann) * Expr: use class IntrusivePtr (Max Kellermann) * Var: copy Location to stack, to fix use-after-free crash bug (Max Kellermann) * Scope: lookup_ID() and install_ID() return IntrusivePtr (Max Kellermann) * Scope: delete duplicate locals (Max Kellermann) * EventRegistry: automatically delete EventHandlers (Max Kellermann) * main: destroy event_registry after iosource_mgr (Max Kellermann) Fixes use-after-free bugs because PcapSource::Close() queues an event. * zeekygen/IdentifierInfo: delete duplicate fields (Max Kellermann) * main: free the global scope in terminate_bro() (Max Kellermann) Make valgrind a bit happier. * Scope: pop_scope() returns IntrusivePtr<> (Max Kellermann) * Scope: unref all inits in destructor (Max Kellermann) * Var: pass IntrusivePtr to add_global(), add_local() etc. (Max Kellermann) * plugin/ComponentManager: hold a reference to the EnumType (Max Kellermann) Use class IntrusivePtr<>. * Type: fix use-after-free bug in VectorType::ShallowClone() (Max Kellermann) The new VectorType instance needs a new reference to the `yield_type`. * Var: fix crash when redeclaring a function with different args (Max Kellermann) 3.2.0-dev.167 | 2020-02-27 14:24:55 -0800 * Expr: use fmt instead of sprintf (Tim Wojtulewicz, Corelight) * Expr: other minor initialization cleanup (Tim Wojtulewicz, Corelight) * Expr: use List::empty() (Tim Wojtulewicz, Corelight) * Expr: Convert a bunch of methods returning ints to return bools (Tim Wojtulewicz, Corelight) * IPAddr: minor cleanup (Tim Wojtulewicz, Corelight) - Mark empty constructors/destructors as default - Initialization cleanup - Remove unnecessary elses from before returns * PriorityQueue: initialization cleanup (Tim Wojtulewicz, Corelight) * IP: Cleanup initialization, make a few functions consistent (Tim Wojtulewicz, Corelight) 3.2.0-dev.159 | 2020-02-26 19:51:24 -0800 * Pop global frame stack on exception. (Johanna Amann, Corelight) When changing this code for 3.0.0, the code-flow changed significantly. One of the changes lead to exception being re-thrown from Call - which leads to the function cleanup not always being run. While the Unref of the framepointer was (correctly) moved into the catch block, the global frame stack was not cleaned up - and contains an invalid pointer after encountering an interpreter exception. This small commit pops the stack and removes the framepointer that was pushed onto it before re-throwing the exception. I also think that this change (re-throwing the exception) breaks the Zeek debugger in significant ways (you cannot debug a script that causes a Runtime error anymore). 3.2.0-dev.157 | 2020-02-26 10:04:32 -0800 * IntrusivePtr: overload std::swap() (Max Kellermann) * IntrusivePtr: eliminate setPtr() (Max Kellermann) There are only two call sites, and those hard-code the `add_ref` parameter. * IntrusivePtr: optimize release() using std::exchange() (Max Kellermann) * Expr: remove unused default constructors and `friend` declarations (Max Kellermann) * Remove useless override: RecordAssignExpr::Eval() (Max Kellermann) * Type: don't pass reference to pointer to MatchesIndex() (Max Kellermann) No implementation modifies the pointer value. To guard against this, this commit changes `&` to `const`. * Expr: don't pass reference to pointer to check_and_promote*() (Max Kellermann) The function never modifies the pointer value. check_and_promote_expr() is left untouched because it really does modify the pointer. * ID: remove unused default constructor (Max Kellermann) * Scope: simplify PList access, use pop_back() and back() (Max Kellermann) * List: add method empty() (Max Kellermann) 3.2.0-dev.146 | 2020-02-26 09:40:18 -0800 * Type: fix double free bug in SetType::ShallowClone() (Max Kellermann) 3.2.0-dev.144 | 2020-02-25 19:52:57 -0800 * Add a faster siphash24 implementation than the reference one (Tim Wojtulewicz, Corelight) - Average of 10 runs of 2009-M57-day11-18.trace (release build at -O3): - Master: 6.027s 93650 bytes max RSS - Commit: 5.950s 93271 bytes max RSS - Hashing a fixed 32-byte payload 10 million times with a fixed key: - Master: 1.397411s - Commit: 0.998211s 3.2.0-dev.142 | 2020-02-25 19:27:28 -0800 * Updates to Broker to build CAF as sub-project (Dominik Charousset, Corelight) The embedded CAF now builds directly as a CMake subdirectory rather than an external project. 3.2.0-dev.137 | 2020-02-24 20:41:43 -0800 * parse.y: fix memory leak in FieldAssignExpr call (Max Kellermann) * parse.y: fix use-after-free bug in open-ended index_slice (Max Kellermann) * Type: fix use-after-free bug in init_type() (Max Kellermann) * Expr: fix potential memory leak in RecordCoerceExpr::Fold() (Max Kellermann) * Expr: fix memory leak in RecordCoerceExpr::InitVal() (Max Kellermann) * zeekygen/IdentifierInfo: fix memory leak in operator=() (Max Kellermann) * Func: fix memory leaks in get_func_priority() (Max Kellermann) * parse.y: fix several memory leaks after lookup_ID() (Max Kellermann) * Func: fix memory leaks in check_built_in_call() (Max Kellermann) * Var: fix memory leaks in add_global() and add_local() (Max Kellermann) * Var: add missing references to `init` in add{,_and_assign}_local() (Max Kellermann) * parse.y: hold reference on init_expr for zeekygen::Manager::Redef() (Max Kellermann) * Expr: fix two memory leaks in AssignExpr::InitVal() (Max Kellermann) * parse.y: fix memory leak after "&deprecated" without string (Max Kellermann) * RuleMatcher: delete PatternSet instances in destructor (Max Kellermann) * Fix reference counting in Option::set_change_handler() (Max Kellermann) 3.2.0-dev.120 | 2020-02-24 18:13:04 -0800 * Update zeek-testing commit (Jon Siwek, Corelight) 3.2.0-dev.119 | 2020-02-24 18:02:10 -0800 * Use IntrusivePtr in TableVal::ExpandAndInit() to clarify ownership (Max Kellermann) 3.2.0-dev.117 | 2020-02-24 18:53:03 -0700 * Switch to using [[deprecated]] attribute (Jon Siwek, Corelight) 3.2.0-dev.115 | 2020-02-24 18:48:50 -0700 * Improve formatting of doubles that are close to integers Now checks for approximate floating point equality so that more doubles get properly disambiguated from integers (Jon Siwek, Corelight) * Improve HTTP version number comparisons Previous use of floating point comparisons was not always stable. (Jon Siwek, Corelight) * Add a 32-bit task to Cirrus CI config (Jon Siwek, Corelight) * Replace va_list fmt() overload with vfmt() Using an overload that takes a va_list argument potentially causes accidental misuse on platforms (e.g. 32-bit) where va_list is implemented as a type that may collide with commonly-used argument types. For example: char* c = copy_string("hi"); fmt("%s", (const char*)c); fmt("%s", c); The first fmt() call correctly goes through fmt(const char*, ...) first, but the second mistakenly goes through fmt(const char*, va_list) first because variadic function overloads have lower priority during overload resolution and va_list on a 32-bit system happens to be defined as a pointer type that can match with "char*" but not "const char*". (Jon Siwek, Corelight) * Format tables indexed by patterns consistently across 32-bit/64-bit Uses a full 64 bit integer for length values regardless of actual size_t to get consistent results between either 32-bit and 64-bit platforms. (Jon Siwek, Corelight) * Format interval values consistently across 32-bit/64-bit platforms (Jon Siwek, Corelight) 3.2.0-dev.108 | 2020-02-24 17:24:07 -0800 * Change OpaqueVal/HashVal APIs to use IntrusivePtr (Max Kellermann) 3.2.0-dev.106 | 2020-02-24 17:07:10 -0800 * ScannedFile: Identify already scanned files by device and inode (Arne Welzel) Zeek scripts located on separate filesystems, but sharing the same inode number leads to scripts not being loaded. The reason is that a `ScannedFile` is only identified by `st_ino` which is not enough to uniquely identify a file in a system. * GH-808: Add ZEEK_VERSION_NUMBER definition to zeek-config.h (Jon Siwek, Corelight) This is the result of (major * 10000 + minor * 100 + patch), for example 3.1.2 becomes 30102. This definition may be helpful for external code that requires conditional compilation to support multiple Zeek versions with differing APIs. * Fix CI to checkout right commit of zeek-testing-private (Jon Siwek, Corelight) 3.2.0-dev.99 | 2020-02-21 21:23:52 -0800 * Update a URL in CI README (Jon Siwek, Corelight) 3.2.0-dev.98 | 2020-02-21 20:04:30 -0800 * Val: use C++ initializers (Max Kellermann) * Val: add BroValUnion constructors (Max Kellermann) * Val: reduce duplicate code by using delegating constructors (Max Kellermann) * Val: remove unused default constructors and `friend` declarations (Max Kellermann) * Val: remove the unnecessary BroValUnion typedef (Max Kellermann) * Type: remove unnecessary enum typedefs (Max Kellermann) * Type: use C++ initializers (Max Kellermann) * Type: move code from BroType::BroType() to constexpr functions (Max Kellermann) Prepare to inline the constructor, which will one day be `constexpr` (requires moving the `std::string name` field somewhere else). * Type: remove useless BroType destructor (Max Kellermann) * Obj: disallow copying BroObj (Max Kellermann) Copying a BroObj is dangerous, and should only be done with dedicated (virtual) methods which are implemented by all derived classes. This commit avoids unintentional copies. * Obj: use C++ initializers (Max Kellermann) * Obj: make `no_location` constexpr (Max Kellermann) This ensures that the variable is initialized at compile time and may allow the compiler to apply more initializations. 3.2.0-dev.85 | 2020-02-21 15:29:45 -0800 * threading/MsgThread: add [[noreturn]] to InternalError() (Max Kellermann) * Flare: add [[noreturn]] to bad_pipe_op() (Max Kellermann) * Obj: add [[noreturn]] attributes to Internal(), bad_ref() (Max Kellermann) * Reporter: add [[noreturn]] attribute to several methods (Max Kellermann) 3.2.0-dev.80 | 2020-02-21 10:14:05 -0800 * OpaqueVal: remove misplaced `virtual` keywords (Max Kellermann) These methods are not meant to be overridden. * CompHash: use class IntrusivePtr for the `type` field (Max Kellermann) * IntrusivePtr: replace the "add_ref" parameter with tag structs (Max Kellermann) Using a runtime parameter is obscure and error-prone. Avoiding error-prone code and getting reference counting right is the whole point of this class. * IntrusivePtr: remove reset(), nobody uses it (Max Kellermann) This method mimicks std::unique_ptr::reset(), but adds an obscure "add_ref" parameter which is error prone. Since nobody uses this method, and this method is all about dealing with raw pointers which we shouldn't be doing, let's remove it. * IntrusivePtr: remove ordering operators (Max Kellermann) These violate the C++ standard because comparing pointers to unrelated objects is undefined behavior. * IntrusivePtr: rename detach() to release() (Max Kellermann) Follow the C++ standard library conventions (here: `std::unique_ptr`). * IntrusivePtr: move nullptr initializer to field declaration (Max Kellermann) This allows "defaulting" the default constructor, and guarantees that all constructors really initialize the field to a legal value. 3.2.0-dev.71 | 2020-02-20 14:57:58 -0800 * Fix missing reference count incrment in AssignExpr::InitVal() (Max Kellermann) 3.2.0-dev.69 | 2020-02-19 18:40:58 -0800 * Fix various reference counting issues in Assign() implementations/callers (Jon Siwek, Corelight) * Fix memory leak in Input stream creation when using &type_column (Max Kellermann) * Fix two memory leaks in BloomFilterVal::Merge() (Max Kellermann) * Fix memory leaks in script debugger (Max Kellermann) * scan.l: fix missing Unref() in do_atif() (Max Kellermann) * Fix parse-time memory leak in RecordConstructorExpr::InitVal() (Max Kellermann) * Fix memory leaks in AssignExpr::EvalIntoAggregate() error conditions (Max Kellermann) * Fix memory leaks in CondExpr::Eval() error conditions (Max Kellermann) Also fixes reference counting issue for vector-based conditionals. * Fix memory leaks in BoolExpr::Eval() error conditions (Max Kellermann) * Fix various memory leaks in Assign() error conditions (Max Kellermann) * Fix memory leaks in BinaryExpr::Eval() error conditions (Max Kellermann) * Fix potential future memory leak in HTTP analyzer (Max Kellermann) This isn't really a memory leak because ParseRequest() never fails, but if it one day "learns" to fail, the `request_method` allocation will leak. 3.2.0-dev.53 | 2020-02-18 12:12:28 -0800 * Make DNS NSEC3 parsing more resilient to introducing a memory leak when no dns_NSEC3 handler exists (Max Kellermann) * analyzer/protocol/dns: change runtime check to assert() (Max Kellermann) If it were legal to call SendReplyOrRejectEvent() without an EventHandlerPtr, then this would leak the `question_name` object. But this method has just one caller, and it verifies the EventHandlerPtr. * Fix memory leak when no dns_TSIG_addl event handler exists (Max Kellermann) 3.2.0-dev.46 | 2020-02-14 22:02:50 -0800 * Fix code format of various reporter btests (Jon Siwek, Corelight) 3.2.0-dev.45 | 2020-02-14 21:14:49 -0800 * Update libkqueue for a linux bugfix (Jon Siwek, Corelight) 3.2.0-dev.44 | 2020-02-14 16:16:54 -0700 * RE: make the RE_Matcher destructor non-virtual (Max Kellermann) * analyzer/protocol/smtp: remove unnecessary nullptr check (Max Kellermann) * analyzer/Manager: remove unnecessary clear() calls from destructor (Max Kellermann) * analyzer/protocol/http: remove unnecessary empty destructor (Max Kellermann) * iosource/pcap/Source: remove unused fields `last_hdr`, `last_data` (Max Kellermann) * iosource/Manager: eliminate two std::string copies (Max Kellermann) 3.2.0-dev.37 | 2020-02-14 11:09:50 -0800 * Fix various format specifiers (Jon Siwek, Corelight) 3.2.0-dev.36 | 2020-02-14 10:29:36 -0700 * broker: include cleanup (Max Kellermann) * file_analysis: include cleanup (Max Kellermann) * file_analysis/Analyzer: eliminate duplicate constructor (Max Kellermann) * probabilistic/Topk: include cleanup (Max Kellermann) * digest: eliminate the "Reporter.h" include (Max Kellermann) * Val: eliminate the "RE.h" include (Max Kellermann) * Val: eliminate the "BroString.h" include (Max Kellermann) * Val: eliminate the "CompHash.h" include (Max Kellermann) * Val: forward-declare class PDict, reduce includes (Max Kellermann) * Val: eliminate the "Scope.h" include (Max Kellermann) 3.2.0-dev.25 | 2020-02-13 19:05:56 -0800 * Reset the number of entries in a dict when calling Clear() (Tim Wojtulewicz, Corelight) * Code cleanup in Dict.h (Tim Wojtulewicz, Corelight) * Add unit testing for the public Dictionary API (Tim Wojtulewicz, Corelight) 3.2.0-dev.21 | 2020-02-13 17:14:26 -0800 * Check for failure when registering event manager with iosource manager (Tim Wojtulewicz, Corelight) 3.2.0-dev.18 | 2020-02-12 00:04:40 -0800 * Fix SSH analyzer assertion/leak after duplicate KEX packet (Max Kellermann) 3.2.0-dev.16 | 2020-02-11 23:44:06 -0800 * Add missing #includes (clang-diagnostic-error) (Tim Wojtulewicz, Corelight) * Remove unnecessary const from a return value (readability-const-return-type) (Tim Wojtulewicz, Corelight) * Create SegmentProfiler objects with names (bugprone-unused-raii) (Tim Wojtulewicz, Corelight) The reason behind this one is that without a real variable name, the profile objects are immediately desctructed and the profiling only happens for the small window when they were valid. If the intention is to profile the method where they were defined, this doesn't actually happen. * Mark a few clang-tidy findings as false-positive (Tim Wojtulewicz, Corelight) * Use std::move in a few places instead of copying argument (performance-unnecessary-value-param) (Tim Wojtulewicz, Corelight) * Use const-references in lots of places (preformance-unnecessary-value-param) (Tim Wojtulewicz, Corelight) * Use string_view for a couple of Dbg methods (Tim Wojtulewicz, Corelight) * Use const references over copying variables (performance-unnecessary-copy-initialization, performance-for-range-copy) (Tim Wojtulewicz, Corelight) * Use single-character version of string find() (performance-faster-string-find) (Tim Wojtulewicz, Corelight) * Move constructors/operators should be marked noexcept to avoid the compiler picking the copy constructor instead (performance-noexcept-move-constructor) (Tim Wojtulewicz, Corelight) 3.2.0-dev.1 | 2020-02-08 16:20:20 -0800 * Update Broker submodule (Jon Siwek, Corelight) 3.2.0-dev | 2020-02-08 16:01:42 -0800 * Update to start of 3.2.0 development (Jon Siwek, Corelight) 3.1.0-dev.603 | 2020-02-08 15:47:48 -0800 * Fix broken links in documentation (Jon Siwek, Corelight) 3.1.0-dev.602 | 2020-02-08 12:44:40 -0800 * Update submodules to release versions (Jon Siwek, Corelight) 3.1.0-dev.601 | 2020-02-08 09:40:15 -0800 * Update embedded CAF to 0.17.4 (Jon Siwek, Corelight) 3.1.0-dev.600 | 2020-02-07 21:10:00 -0800 * Note new CMake 3.0+ and C++17 requirements in NEWS (Jon Siwek, Corelight) 3.1.0-dev.599 | 2020-02-07 19:58:35 -0800 * Improve an input framework test (Jon Siwek, Corelight) 3.1.0-dev.598 | 2020-02-07 15:06:56 -0800 * Improve stability of a &expire_func btest (Jon Siwek, Corelight) 3.1.0-dev.597 | 2020-02-07 15:25:56 -0700 * plugin/Manager: migrate to std::string_view (Max Kellermann) * util: optimize the normal_path() common case Speeds up Zeek startup by 2%. (Max Kellermann) * util: pass string_view to without_bropath_component() (Max Kellermann) * module_util: make GLOBAL_MODULE_NAME constexpr This allows the compiler to hard-code pointers to the string without looking up a global variable at runtime. (Max Kellermann) * Scope: convert Scope::Lookup() and others to template Allows passing rvalue references which eliminates unnecessary std::string copies. This speeds up Zeek startup by 1-2%. (Max Kellermann) * Scope: Vars() returns const reference No caller wants to modify the container. (Max Kellermann) * Anon: remove unnecessary {map,vector}::clear() calls The destructor will do this automatically. (Max Kellermann) * Dict: make the destructor non-virtual (Max Kellermann) * Obj: make the Location constructors `constexpr` (Max Kellermann) * Obj: remove unused fields Location::{timestamp,text} (Max Kellermann) * Obj: remove Location::delete_data, nobody ever sets it This allows removing the destructor completely, making the class trivially-destructible. (Max Kellermann) * Obj: make the Location destructor non-virtual Nobody ever derives from this class. This removes an indirect call and allows inlining the destructor. This speeds up Zeek startup by 1-2%. (Max Kellermann) 3.1.0-dev.584 | 2020-02-07 14:12:17 -0800 * Improve kerberos analyzer address and event handling Adds a weird, "invalid_kerberos_addr_len", for invalid kerberos host address lengths and also fixes a memory leak when processing KRB_KDC_REQ and KRB_KDC_REP messages for message types that do not match a known/expected type. (Jon Siwek, Corelight) 3.1.0-dev.582 | 2020-02-07 09:46:00 -0800 * Add environment variable to disable supervisor SIGKILL For use in btests since SIGKILL can otherwise bypass the usual btest-bg-wait timeout mechanism and make reason for test failures less obvious. (Jon Siwek, Corelight) * Fix a btest's race condition by handling Pcap::file_done (Jon Siwek, Corelight) * Add Pcap::file_done event It signals when a pcap file is done being processed. (Jon Siwek, Corelight) * Improve btest timeouts * Generally increase timeouts for tests that have recent transient failures * Change any test that relied on `btest-bg-wait -k` since that's never going to play with with CI systems. Instead, we always need to have a well-defined termination condition in the test itself (and most already did, so didn't really need the `-k` flag anyway). (Jon Siwek, Corelight) 3.1.0-dev.577 | 2020-02-07 09:43:48 -0800 * Use 2 btest retries for CI (Jon Siwek, Corelight) 3.1.0-dev.575 | 2020-02-07 09:31:50 -0800 * GH-780: Prevent log batches from indefinite buffering Logs that got sent sparsely or burstily would get buffered for long periods of time since the logic to flush them only does so on the next log write. In the worst case, a subsequent log write could never happen and cause a log entry to be indefinitely buffered. This fix introduces a recurring event/timer to simply flush all pending logs at frequency of Broker::log_batch_interval. (Jon Siwek, Corelight) 3.1.0-dev.571 | 2020-02-07 10:33:16 +0000 * Update NEWS to point out Broker wireformat change. (Robin Sommer, Corelight) 3.1.0-dev.570 | 2020-02-06 14:58:34 -0800 * add &on_change attribute for sets/tables The &on_change attribute allows users to specify a function that is called when a set or table is changed via a direct insertion/delete/change or element expiry. Example: function set_change(t: set[string], tpe: TableChange, idx: string) { print "set_change", idx, tpe; } global s: set[string] &on_change=set_change; (Johanna Amann, Corelight) 3.1.0-dev.554 | 2020-02-06 13:05:39 -0800 * Switch GitHub Action for CI emails to use zeek organization (Jon Siwek, Corelight) * Restore accidentally reverted change to fix FreeBSD 11 build (Tim Wojtulewicz, Corelight) 3.1.0-dev.552 | 2020-02-05 21:47:08 -0800 * Update CAF to current 0.17.4 release branch (Jon Siwek, Corelight) 3.1.0-dev.551 | 2020-02-05 17:50:56 -0800 * Remove presumed ownership of HeartbeatTimer (Jon Siwek, Corelight) threading::Manager is currently never deleted, but if that ever changes, deleting the stored HeartbeatTimer pointer would be a double-free since TimerMgr owns it. * Set threading::Manager terminating state (Jon Siwek, Corelight) Coverity CID 1417429 * Remove unused HeartbeatTimer member: do_expire (Jon Siwek, Corelight) Coverity CID 1417434 3.1.0-dev.548 | 2020-02-05 14:57:08 -0800 * GH-773: Make EventMgr an IOSource so that events interrupt kevent waits (Tim Wojtulewicz, Corelight) 3.1.0-dev.546 | 2020-02-05 10:51:35 -0800 * Change threading formatter errors to warnings. (Johanna Amann, Corelight) This change standardizes threading formatter error handling and moves the remaining error calls to be warnings instead. This is in line with already existing code - in most cases warnings were raised, only a few cases raised errors. These cases do not differ significantly from other cases in which warnings are raised. This also fixes GH-692, in which misformatted lines prevent future file parsing. This commit also moves the FailWarn method that is used by both the config and the ascii reader up to the ReaderBackend. Furthermore it makes the Warning method of ReaderBackend respect the warning suppression that is introduced by the FailWarn method. * Update NEWS about plugin breakage from the include-file changes (Tim Wojtulewicz, Corelight) 3.1.0-dev.542 | 2020-02-04 14:05:19 -0700 * UID, ..: un-inline methods to reduce header dependencies Only 1% build time speedup, but still, it declutters the headers a bit. Before this patch: 2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k 72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps After this patch: 2537.19user 142.94system 2:26.90elapsed 1824%CPU (0avgtext+0avgdata 1434268maxresident)k 16240inputs+8887152outputs (1931major+48728888minor)pagefaults 0swaps (Max Kellermann) * include cleanup The Zeek code base has very inconsistent #includes. Many sources included a few headers, and those headers included other headers, and in the end, nearly everything is included everywhere, so missing #includes were never noticed. Another side effect was a lot of header bloat which slows down the build. First step to fix it: in each source file, its own header should be included first to verify that each header's includes are correct, and none is missing. After adding the missing #includes, I replaced lots of #includes inside headers with class forward declarations. In most headers, object pointers are never referenced, so declaring the function prototypes with forward-declared classes is just fine. This patch speeds up the build by 19%, because each compilation unit gets smaller. Here are the "time" numbers for a fresh build (with a warm page cache but without ccache): Before this patch: 3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k 760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps After this patch: 2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k 72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps (Max Kellermann) * Updating submodule(s). [nomail] (Jon Siwek, Corelight) 3.1.0-dev.538 | 2020-02-04 11:57:35 +0000 * Updating submodules. 3.1.0-dev.536 | 2020-02-03 15:44:10 -0700 * Change various functions to by-value std::string_view args (Jon Siwek, Corelight) 3.1.0-dev.534 | 2020-02-03 15:42:24 -0700 * Change packet source fields of Options to std::optional (Jon Siwek, Corelight) 3.1.0-dev.531 | 2020-02-03 13:23:22 -0800 * Update sqlite to 3.31.1 (Johanna Amann, Corelight) * Update certificate list to state of NSS 3.49.2. (Johanna Amann, Corelight) 3.1.0-dev.530 | 2020-02-03 13:21:16 -0800 * Disable Travis leak test (Jon Siwek, Corelight) Travis environment doesn't offer enough memory for running ASan very well, but Cirrus has it covered anyway. 3.1.0-dev.529 | 2020-02-03 13:04:30 -0800 * CMakeLists.txt: fix cross-compilation with binpac (Fabrice Fontaine) 3.1.0-dev.527 | 2020-02-03 10:11:00 -0800 * Change requested memory in Cirrus CI config (Jon Siwek, Corelight) The ASan build does require quite a bit more, but also the default release builds should do ok with slightly less. 3.1.0-dev.526 | 2020-02-03 10:04:00 -0800 * Increase timeout value for live interfaces without file descriptors (Tim Wojtulewicz, Corelight) * Use ranged-for loops in a few places in iosource Manager (Tim Wojtulewicz, Corelight) * Change order of includes in iosource Manager, fixes build on FreeBSD 11 (Tim Wojtulewicz, Corelight) 3.1.0-dev.521 | 2020-01-31 14:18:17 -0800 * util: optimize expand_escape() by avoiding sscanf() (Max Kellermann) sscanf() is notoriously slow, and the default scripts have lots of hex escapes. This patch can reduce Zeek's startup time by 3-9%. 3.1.0-dev.519 | 2020-01-31 13:19:09 -0800 * util: optimize tokenize_string() and normalize_path() (Max Kellermann) This patch can speed up Zeek startup by 10-25%. Adds a new tokenize_string() overload which returns string_views and changes existing tokenize_string() and normalize_path() to use string_view arguments. 3.1.0-dev.510 | 2020-01-31 11:20:28 -0800 * Remove extra fmt() in a reporter->Error() call (Jon Siwek, Corelight) * parse.y: Properly set location info for functions (Arne Welzel) When defining a function, remember the location where the function header was and restore it before calling `end_func()`. Inside `end_func()`, a `BroFunc` object is created using the current global location information. This came up while experimenting with zeek script profiling and wondering why the locations set for `BroFunc` were "somewhere" in the middle of functions instead of spanning them. * Desc: move realloc() call out of the loop (Max Kellermann) * SerializationFormat: move realloc() call out of the loop (Max Kellermann) * PacketDumper: remove unused types (Max Kellermann) 3.1.0-dev.503 | 2020-01-31 11:00:32 -0800 * Show percentage of packets dropped in the final process output (Tim Wojtulewicz, Corelight) * Remove concept of multiple timer managers (Tim Wojtulewicz, Corelight) - All timers are now handled by a single global timer manager, which simplifies how they handled by the IOSource manager. - This change flows down a number of changes to other parts of the code. The timer manager tag field is removed, which means that matching connections to a timer manager is also removed. This removes the ability to tag a connection as internal or external, since that's how the connections where differentiated. This in turn removes the `current_conns_extern` field from the `ConnStats` record type in the script layer. * Have terminate_processing() raise SIGTERM instead of calling the signal handler directly (Tim Wojtulewicz, Corelight) * PktSrc iosource changes to match the new IOSource API (Tim Wojtulewicz, Corelight) * Broker manager changes to match the new IOSource API and loop architecture (Tim Wojtulewicz, Corelight) * DNS_Mgr changes to match the new IOSource API and loop architecture (Tim Wojtulewicz, Corelight) * Threading changes for the new loop architecture (Tim Wojtulewicz, Corelight) - threading::Manager is no longer an IOSource. - threading::MsgThread is now an IOSource. This allows threads themselves to signal when they have data to process instead of continually checking each of the threads on every loop pass. - Make the thread heartbeat timer an actual timer and let it fire as necessary instead of checking to see if it should fire * Add Trigger manager for managing triggers created by things like 'when' statements - Adds new trigger namespace (Tim Wojtulewicz, Corelight) - Adds trigger::Manager class as a new IOSource for keeping track of triggers and integrating them into the loop. Previously the loop relied on the event manager Drain() method to process all triggers on every loop, but now that the loop actively waits for events to occur, triggers would not fire when they needed to. Adding them as part of the loop ensures they're checked. * Make TimerMgr an IOSource (Tim Wojtulewicz, Corelight) - This allows the loop to check what the next timeout is and use that as the basis for the timeout of the poll - This commit also removes the TimerMgr::Tag type, since it causes a name clash with other code in IOSource * Modify IOSource Manager to implement new loop architecture (Tim Wojtulewicz, Corelight) - Removes entire FindSoonest method that includes all of the calls to select() for checking for ready sources - Removes FD_Set checking against IOSources - Adds system for registering and unregistering file descriptors from IOSources. This allows individual sources to mark themselves as ready to be checked by the loop as they become available. - Adds entirely new loop architecture based on checking the IOSources for when their next timeout is, and then waiting for either that timeout or when the next source is ready. This also implements the polling based on what the OS supports, instead of just calling select() on all platforms. Currently it supports kqueue, epoll, and plain poll. - Adds system for pinging the loop to force it to wake up * Minor amount of code cleanup in Pcap IO source (Tim Wojtulewicz, Corelight) * IOSource API changes for new loop architecture (Tim Wojtulewicz, Corelight) - Removed GetFD and NextTimestamp methods - Removed concept of idle sources - Renamed Init to InitSource to avoid a name clash with the thread code - Added GetNextTimeout method This commit also removes the FD_Set file since it's no longer used * Type and variable usage cleanup in Net.h (Tim Wojtulewicz, Corelight) * Remove #include of some iosource files from Net.h (Tim Wojtulewicz, Corelight) * Only allow a single trace file (-r) or interface (-i) option on the command-line (Tim Wojtulewicz, Corelight) * Remove CQ_TimerMgr in favor of PQ_TimerMgr (Tim Wojtulewicz, Corelight) 3.1.0-dev.482 | 2020-01-30 19:16:03 -0800 * Update NEWS for naming changes. (Robin Sommer, Corelight) * Remove test for legacy plugin. (Robin Sommer, Corelight) We no longer support creating new plugins using the old naming scheme. * Remove legancy symlinks in aux/. (Robin Sommer, Corelight) That's aux/bro-aux and aux/broctl. * Add warnings when loading scripts ending in ".bro", or using legacy environment variables. (Robin Sommer, Corelight) * Fix missing rename to .zeek-dns-cache. (Robin Sommer, Corelight) * No longer symlink local.zeek to local.bro. (Robin Sommer, Corelight) * Update notice user agent. (Robin Sommer, Corelight) * Remove old_comm_usage_is_ok. (Robin Sommer, Corelight) * Remove bro-config.h.in and bro-path-dev.in. (Robin Sommer, Corelight) * Change Bro wrapper script to now abort when old executable names are still used. (Robin Sommer, Corelight) * Remove APIs that were explicitly deprecated to be removed in 3.1. (Robin Sommer, Corelight) Special handling for bro_{init,done,script_loaded} events: if still used, they cause Zeek to abort at startup. 3.1.0-dev.469 | 2020-01-30 08:44:15 -0700 * Fix leaked FDs in redirecting supervised node stdout/stderr (Jon Siwek, Corelight) * Add checks for failed fcntl calls (Jon Siwek, Corelight) * Set Pipe file descriptor flags correctly (Jon Siwek, Corelight) 3.1.0-dev.465 | 2020-01-29 11:45:15 -0800 * Scope: fix memory leak by removing duplicate copy_string() call (Max Kellermann) Note this is a parse-time memory leak, not a run-time leak. * util, nb_dns: fix off-by-one bugs in strncpy() calls (Max Kellermann) Fortunately, these bugs had no effect because the following lines overwrote the last character with a null byte. * Type, util: add `constexpr` to static variables (Max Kellermann) * Net: remove unused variable (Max Kellermann) 3.1.0-dev.460 | 2020-01-29 11:16:15 -0800 * Skip file analysis for zero-length SSL/TLS data (Jon Siwek, Corelight) 3.1.0-dev.458 | 2020-01-29 12:53:32 +0000 * Add a new supervisor framework that enables Zeek to operate clusters of processes itself without any external help. (Jon Siwek, Corelight) The Supervisor framework provides an entirely new deployment mode for Zeek, one that supervises a set of Zeek processes that are meant to be persistent. A Supervisor automatically revives any process that dies or exits prematurely and also arranges for an ordered shutdown of the entire process tree upon its own termination. This Supervisor mode for Zeek provides the basic foundation for process configuration/management that could be used to deploy a Zeek cluster similar to what ZeekControl does, but is also simpler to integrate as a standard system service. This mode is still experimental and will evolve over time. The command-line argument of ``-j`` toggles Zeek to run in "Supervisor mode" to allow for creation and management of child processes. If you're going to test this, please note that you will need some custom script code to configure the processes you want Zeek to run. See the documentation for more information: https://docs.zeek.org/en/stable/frameworks/supervisor.html 3.1.0-dev.408 | 2020-01-28 17:56:02 -0800 * Update Cirrus CI config to use macOS Catalina (Jon Siwek, Corelight) 3.1.0-dev.406 | 2020-01-28 12:42:25 -0800 * util: use getrandom() on Linux if available (Max Kellermann) Unlike /dev/urandom, getrandom() doesn't need a file descriptor and works when there is no /dev. It requires Linux 3.17 and glibc 2.25, but there is a fallback to the old code. 3.1.0-dev.403 | 2020-01-24 15:15:04 -0800 * Fixed decompose_uri() errors on URIs with empty port component (Frerich Raabe) 3.1.0-dev.398 | 2020-01-23 18:08:43 -0800 * Skip check for outdated docs in Cirrus CI for PRs (Jon Siwek, Corelight) 3.1.0-dev.397 | 2020-01-23 15:30:33 -0800 * Use consistent zeek_init priority for Log::create_stream calls Typically in base scripts, Log::create_stream() is called in zeek_init() handler with &priority=5 such that it will have already been created in the default zeek_init() &priority=0. (Jon Siwek, Corelight) 3.1.0-dev.395 | 2020-01-23 13:50:42 +0000 * Update Broker submodule. 3.1.0-dev.394 | 2020-01-23 13:18:54 +0000 * Improve creation of SMB transaction data strings. (Jon Siwek, Corelight) 3.1.0-dev.392 | 2020-01-23 13:14:28 +0000 * Improve FTP word/whitespace handling. (Jon Siwek, Corelight) 3.1.0-dev.389 | 2020-01-18 10:49:15 +0000 * GHI-595: Convert from nlohmann/json to RapidJSON for performance reasons. (Tim Wojtulewicz, Corelight) * Optimize json_escape_utf8() and expand its unit tests. (Tim Wojtulewicz, Corelight) * Convert type-checking macros to actual functions. (Tim Wojtulewicz, Corelight) * Use the list of files from clang-tidy when searching for unit tests. (Tim Wojtulewicz, Corelight) 3.1.0-dev.383 | 2020-01-17 11:51:01 +0000 * Various code modernization cleanup. (Tim Wojtulewicz, Corelight) 3.1.0-dev.378 | 2020-01-16 13:18:13 +0000 * Handle invalid Base64 encodings in FTP ADAT analyzer (Jon Siwek, Corelight) 3.1.0-dev.376 | 2020-01-14 09:45:45 -0800 * Fix warning when reading files from non-network sources (Seth Hall, Corelight) If files are being read from non-network sources, there was a warning in the SSL base scripts about missing the f$conns field. 3.1.0-dev.372 | 2020-01-13 12:10:42 +0000 * Fix method returning a reference to a temporary. Found by cppcheck, via by Arthur Wong. (Robin Sommer, Corelight) 3.1.0-dev.370 | 2020-01-09 19:04:59 -0800 * Update timeouts for a btest (Jon Siwek, Corelight) 3.1.0-dev.368 | 2020-01-09 13:39:38 -0800 * Extend ASAN_OPTIONS in btests instead of overwrite (Jon Siwek, Corelight) 3.1.0-dev.366 | 2020-01-09 12:28:32 -0800 * Update external test repo commit hash (Jon Siwek, Corelight) 3.1.0-dev.365 | 2020-01-09 11:38:06 -0800 * Enable LeakSanitizer for unit tests run via doctest (Jon Siwek, Corelight) 3.1.0-dev.363 | 2020-01-09 10:25:24 -0800 * fix NTLM field value access The fields being checked for existence were not the same as the fields being accessed. (Justin Azoff, Corelight) 3.1.0-dev.361 | 2020-01-08 10:29:59 -0800 * Increase timeout for a few btests (Jon Siwek, Corelight) 3.1.0-dev.360 | 2020-01-07 20:58:19 -0800 * Preserve optimization setting for sanitizer build (Jon Siwek, Corelight) 3.1.0-dev.359 | 2020-01-06 16:40:55 -0800 * Fix build warning in X509 plugin (Tim Wojtulewicz, Corelight) 3.1.0-dev.357 | 2020-01-06 14:10:54 -0800 * GH-412: Enable Patterns as Table index (Dev Bali, Corelight) 3.1.0-dev.354 | 2020-01-06 14:16:29 -0700 * Fix snprintf compiler warning in hexdump BIF (Jon Siwek, Corelight) * Updating submodule(s). [nomail] (Jon Siwek, Corelight) 3.1.0-dev.351 | 2020-01-06 11:20:09 -0800 * Increase a timeout for a btest (Jon Siwek, Corelight) 3.1.0-dev.350 | 2020-01-06 10:51:07 -0800 * Fix Cirrus CI FreeBSD python dependencies installation (Jon Siwek, Corelight) 3.1.0-dev.348 | 2020-01-06 11:19:36 -0700 * Increase timeout for a btest May be more timing-sensitive with slower ASAN/LSAN configuration in addition to already-slower CI environment. (Jon Siwek, Corelight) * Switch CI scripts to use ASAN/LSAN instead of gperftools Using AddressSanitizer/LeakSanitizer is better. It covers the full unit/baseline test suite by default without requiring one to write specific memory leak tests. It also covers other types of memory errors besides just leaks. (Jon Siwek, Corelight) * Remove redundant memory leak btests Or otherwise convert into a regular btest if it didn't already seem to be covered. There's no need for a separate memory leak test group since compiling with LeakSanitizer now covers leak checking for the full btest suite. (Jon Siwek, Corelight) * Fix reference counting issues related to lambdas/closures For example, circular references between a lambda function the frame it's stored within and/or its closure could cause memory leaks. This also fixes other various reference-count ownership issues that could lead to memory errors. There may still be some potential/undiscovered issues because the "outer ID" finding logic doesn't look quite right as the AST traversal descends within nested lambdas and considers their locals as "outer", but possibly the other logic for locating values in closures or cloning closures just works around that behavior. (Jon Siwek, Corelight) * Disable LeakSanitizer for btests that have known leaks E.g. ones that throw interpreter exceptions, as those are currently known to potentially cause leaks. Fixing the underlying leaks involves the larger task of more IntrusivePtr usage. Reference cycles may also cause leaks. (Jon Siwek, Corelight) * Rewrite the btest for when-statement timeouts To avoid a memory leak in DNS lookups that's hard to work around and does not otherwise effect typical operation when Zeek is allowed to continue to run after zeek_init(). (Jon Siwek, Corelight) * Fix memory leak when a logging plugin hook prevents a write (Jon Siwek, Corelight) * Fix memory leaks in various input framework error-handling cases (Jon Siwek, Corelight) * Fix memory leak in Reporter::get_weird_sampling_whitelist() BIF (Jon Siwek, Corelight) * Fix reference counting of Log::Filter "config" field Which can potentially be a minor memory leak if there's a lot of dynamic adding/removing of logging filters. (Jon Siwek, Corelight) * Fix memory leak in system_env() BIF (Jon Siwek, Corelight) * Clean up triggers awaiting global state modification at shutdown Otherwise they can be reported as memory leaks since no more global state modifications will take place to notify the trigger to clean itself up. (Jon Siwek, Corelight) * Fix memory leak in initializing log writers with no local backend (Jon Siwek, Corelight) * Fix packet filter memory leaks (Jon Siwek, Corelight) * Skip sending thread heartbeat if it alread asked to be finished Otherwise the heartbeat message may fail to be processed and show up as leaked memory. (Jon Siwek, Corelight) * Fix memory leak of sqlite input reader prepared statement (Jon Siwek, Corelight) * Prevent duplicate "finish" threading messages As they don't get processed and may show up as a memory leak. (Jon Siwek, Corelight) * Fix memory leak when table-based input stream overwrites old entries (Jon Siwek, Corelight) * Fix scripting error in a DHCP btest (Jon Siwek, Corelight) * Fix memory leaks in Kerberos ticket decryption Memory allocated to the decrypted ticket data as well as the server principal were not freed. Also fixed potential leaks in error cases that called krb5_get_error_message() without freeing the returned value. (Jon Siwek, Corelight) * Fix scripting error in an ftp btest (Jon Siwek, Corelight) * Update paraglob submodule (Jon Siwek, Corelight) * Fix malloc/delete mismatch in JSON formatting ODesc allocated with malloc() and BroString deallocated with delete[], but really the intermediate BroString wasn't even needed when copying into std::string. (Jon Siwek, Corelight) * Delete/timeout pending DNS requests during shutdown Primarily, this change prevents the pending requests showing up as memory leaks. (Jon Siwek, Corelight) * Fix memory leak in OCSP parsing functions Various OCSP parsing functions used in presence of OpenSSL 1.1 used "d2i_ASN1_SEQUENCE_ANY" which returns a "STACK_OF(ASN1_TYPE)", but used "sk_ASN1_TYPE_free" instead of "sk_ASN1_TYPE_pop_free" to free it. The former only frees the stack structure while the later frees both the structure and the elements. (Jon Siwek, Corelight) * Free the global X509 certificate root store on shutdown Otherwise LeakSanitizer reports its contents as leaked. (Jon Siwek, Corelight) * Add general LeakSanitizer macros/instrumentation (Jon Siwek, Corelight) * Improve --sanitizers configure option * Rename SANITIZERS CMake variable to ZEEK_SANITIZERS for clarity * Use -O1 by default to improve speed (set NO_OPTIMIZATIONS env. var. to override and use -O0). Uses -fno-optimize-sibling-calls with -O1 to still get "perfect stack traces". * Updates various sub-projects with sanitizer improvements: binpac and bifcl, by default, now ignore leaks reported by LeakSanitizer so that it doesn't interfere with the Zeek build (Jon Siwek, Corelight) 3.1.0-dev.319 | 2020-01-06 09:44:11 -0800 * Mark safe_snprintf and safe_vsnprintf as deprecated, remove uses of them (Tim Wojtulewicz, Corelight) * Add unit tests to util.cc and module_util.cc (Tim Wojtulewicz, Corelight) 3.1.0-dev.314 | 2019-12-18 13:36:07 -0800 * Add GitHub Action for CI notification emails (Jon Siwek, Corelight) 3.1.0-dev.313 | 2019-12-18 13:23:51 -0800 * Add Cirrus CI config (Jon Siwek, Corelight) 3.1.0-dev.309 | 2019-12-16 09:40:01 -0800 * GHI-599: avoid memory leak with default pattern matching and json formatting (Tim Wojtulewicz, Corelight) 3.1.0-dev.307 | 2019-12-16 08:20:58 -0800 * Update URL for Malware Hash Registry website (Jon Siwek, Corelight) 3.1.0-dev.300 | 2019-12-05 12:34:41 -0700 * GH-700: add zeek_args() BIF Provides access to all zeek command-line arguments ("argv"). (Jon Siwek, Corelight) 3.1.0-dev.297 | 2019-12-05 11:57:12 -0700 * GH-700: add packet_sources() BIF Provides access to properties of packet sources, like interface or pcap file name. (Jon Siwek, Corelight) 3.1.0-dev.295 | 2019-12-04 14:43:27 -0700 * Use new Zeek Logo instead of Bro Eyes on README.md (Dev Bali, Corelight) 3.1.0-dev.292 | 2019-12-02 13:37:19 -0800 * GH-619: Allow "print" statements to be redirected to a Log (Dev Bali, Corelight) 3.1.0-dev.286 | 2019-11-21 08:47:32 -0800 * GH-684: Fix parsing of RPC calls with non-AUTH_UNIX flavors The parsing logic that should be specific to the AUTH_UNIX credential flavor was previously applied unconditionally to other flavors. (Jon Siwek, Corelight) 3.1.0-dev.284 | 2019-11-21 08:29:36 -0800 * Fix ZEEK_PROFILER_FILE file format/parsing Some Zeek script statement descriptions were exceeding the hardcoded maximum length and also could contain tab characters which were supposed to be reserved for use as a delimiter in the file format. (Jon Siwek, Corelight) 3.1.0-dev.282 | 2019-11-18 12:06:13 +0000 * GH-646: Add new "successful_connection_remove" event. (Jon Siwek, Corelight) Includes: - Switch Zeek's base scripts over to using it in place of "connection_state_remove". The difference between the two is that "connection_state_remove" is raised for all events while "successful_connection_remove" excludes TCP connections that were never established (just SYN packets). There can be performance benefits to this change for some use-cases. - Add new event called ``connection_successful`` and a new ``connection`` record field named "successful" to help indicate this new property of connections. 3.1.0-dev.280 | 2019-11-15 18:27:27 -0800 * Run doctest unit tests in Travis CI (Jon Siwek, Corelight) * Fix indents/whitespace in Travis CI script (Jon Siwek, Corelight) * Adjust btests for OpenBSD portability (Jon Siwek, Corelight) * Convert pcapng test suite files to pcap format for OpenBSD compat (Jon Siwek, Corelight) * Fix undefined symbols loading libbroker on OpenBSD (Jon Siwek, Corelight) * Fix compile warnings on OpenBSD (Jon Siwek, Corelight) 3.1.0-dev.271 | 2019-11-14 19:16:31 -0800 * Add initial scaffold for unit testing via doctest (Dominik Charousset, Corelight) 3.1.0-dev.266 | 2019-11-14 17:29:00 -0800 * Add hint to run `make distclean` if configure fails (Simon Hardy-Francis, Corelight) 3.1.0-dev.264 | 2019-11-14 16:57:13 -0800 * Install test-all-policy.zeek script (Jon Siwek, Corelight) The zeekygen script, which is already installed, refers to it, and could also generally be useful for "test everything" type scenarios. 3.1.0-dev.263 | 2019-11-13 13:43:16 -0800 * Improve record_fields() BIF to recursively give full container type names (Henri DF) 3.1.0-dev.257 | 2019-11-11 13:40:11 -0800 * Update embedded CAF to 0.17.3 (Jon Siwek, Corelight) 3.1.0-dev.256 | 2019-11-07 13:42:11 -0800 * Allow record_fields() string arguments that name a record type (Jon Siwek, Corelight) 3.1.0-dev.253 | 2019-11-05 09:54:01 -0800 * Add and use new IntrusivePtr type (Dominik Charousset, Corelight) Manual memory management via Ref/Unref is verbose and prone to error. An intrusive smart pointer automates the reference counting, makes code more robust (in particular w.r.t. to exceptions) and reduces boilerplate code. A big benefit of the intrusive smart pointers for Zeek is that they can co-exist with the manual memory management. Rather than having to port the entire code base at once, we can migrate components one-by-one. In this first step, we add the new template `IntrusivePtr` and start using it in the Broker Manager. This makes the previous `unref_guard` obsolete. 3.1.0-dev.247 | 2019-11-04 17:26:32 -0800 * Cluster-ize FTP data channel analysis (Jeff Barber) The ftp_data_expected table is now synchronized across Zeek worker nodes. Note there's still a possible race condition where the traffic for the ftp-data connection proceeds before the Zeek cluster has a change to inform workers to expect it. 3.1.0-dev.245 | 2019-11-04 16:57:11 -0800 * Use CMake object libraries for subdir libs and plugins (Dominik Charousset, Corelight) 3.1.0-dev.242 | 2019-11-04 09:58:53 -0800 * Fix error handling in decode_base64() (Jon Siwek, Corelight) 3.1.0-dev.241 | 2019-11-04 09:44:20 -0800 * Revert recent changes to Base64.h API. (Jon Siwek, Corelight) This reverts commit 705210a035d4128ecdf249c7db98607c570bfcd2. The original changes broke the Base64.h API which may be used by external plugins. 3.1.0-dev.240 | 2019-11-01 15:58:55 -0700 * Use #pragma once in Base64.h (Jon Siwek, Corelight) 3.1.0-dev.239 | 2019-10-31 12:18:16 -0700 * Fix Travis CI script to run Coverity build in container (Jon Siwek, Corelight) 3.1.0-dev.237 | 2019-10-31 06:37:23 +0100 * Remove "using namespace std" from header files (Jon Siwek, Corelight) 3.1.0-dev.235 | 2019-10-30 15:04:13 -0700 * GH-649: define missing DLT_NFLOG to support OpenBSD (Jon Siwek, Corelight) 3.1.0-dev.234 | 2019-10-30 13:33:08 -0700 * Add --libdir convenience flag to configure. (Johanna Amann, Corelight) 3.1.0-dev.231 | 2019-10-30 11:02:28 -0700 * Add missing ipv6 support in patricia.c local_inet_pton() (Tim Shelton) 3.1.0-dev.229 | 2019-10-30 07:57:19 +0000 * GH-664: fix signature matching for cases where the first TCP packet seen contained payload data, regardless of its TCP flags. (Jon Siwek, Corelight) 3.1.0-dev.227 | 2019-10-29 09:39:10 -0700 * Replace build_unique with make_unique (Johanna Amann, Corelight) This was a rarely used convenience function from when we did not yet have c++17 support. * GH-626: Revert "Fix compilation on OS-X." (Johanna Amann, Corelight) Reverts workaround in cde28074a169212aa8f38fdac225ecbeac4e642d which depended on C++14 features at a time when we used only C++11. 3.1.0-dev.222 | 2019-10-28 20:18:15 -0700 * Install cmake3 from EPEL on CentOS CI system (Jon Siwek, Corelight) 3.1.0-dev.221 | 2019-10-28 18:30:24 -0700 * Require CMake 3.0+ (Dominik Charousset, Corelight) * Move CMake project() after cmake_minimum_required() (Dominik Charousset, Corelight) The call to `project` must come after `cmake_minimum_required` in CMake in order to get the correct policy settings. 3.1.0-dev.219 | 2019-10-28 20:59:12 +0100 * GH-654: allow table() in function &default expressions Table parameters of functions previously did not coerce unspecified tables used in their &default attribute to the correct type. (Jon Siwek, Corelight) * GH-654: allow table() in record &default expressions Table fields of records previously did not coerce unspecified tables used in their &default attribute to the correct type. (Jon Siwek, Corelight) * Fix up some documentation. (Keith J. Jones) 3.1.0-dev.214 | 2019-10-28 10:58:43 +0000 * GH-585: Require C++17 (Jon Siwek, Corelight) * Update Travis CI config for C++17 (Jon Siwek, Corelight) 3.1.0-dev.209 | 2019-10-28 09:17:15 +0000 * Fix HTTP::build_url. (Alexey Mokeev) * Free memory allocated by glob() in plugin manager. (Arne Welzel) * Remove always-false condition in plugin manager. (Arne Welzel) 3.1.0-dev.203 | 2019-10-25 10:41:10 -0700 * Change usage of old bro:see directive to zeek:see (Jon Siwek, Corelight) * Use explicit path name for NTP log stream (Jon Siwek, Corelight) For consistency (we do this for all other logs) and just to avoid the extra path function calls. 3.1.0-dev.201 | 2019-10-25 10:12:49 -0700 * Fixed typo. (Keith J. Jones) 3.1.0-dev.197 | 2019-10-21 09:34:18 -0700 * Update embedded CAF in Broker to 0.17.2 (Jon Siwek, Corelight) 3.1.0-dev.195 | 2019-10-21 11:42:18 +0200 * Prefer cmake3 command, add --cmake=PATH option (Dominik Charousset, Corelight) 3.1.0-dev.192 | 2019-10-17 16:32:16 -0700 * Zeekctl: change gzip compression level from 9 to default #614 (JC Connell) 3.1.0-dev.191 | 2019-10-16 10:02:45 -0700 * Update broker submodule to use pybind11 v2.4.3 (Jon Siwek, Corelight) * Improve &default validation for global vars: exclude sets (Jon Siwek, Corelight) * Remove unnecessary &optional from a Config::set_value() parameter (Jon Siwek, Corelight) * Remove check for redundant attributes (Jon Siwek, Corelight) 3.1.0-dev.185 | 2019-10-15 16:46:22 -0700 * Add weirds for invalid TCP option lengths (Jon Siwek, Corelight) * GH-618: add "tcp_options" event containing TCP option values (Jon Siwek, Corelight) * Fix tcp_option event It was not being raised in all the cases it should have been due to an incorrect/unnecessary truncation check. (Jon Siwek, Corelight) 3.1.0-dev.181 | 2019-10-14 22:05:01 -0700 * Fix redef'ing a table with a new &default attribute (Jon Siwek, Corelight) 3.1.0-dev.179 | 2019-10-14 21:33:15 -0700 * Support whitespace at end of line for config reader (Seth Hall, Corelight) 3.1.0-dev.177 | 2019-10-14 20:13:24 -0700 * Proof of Concept on Bugprone Narrowing Conversions: Base64 (Dev Bali, Corelight) 3.1.0-dev.175 | 2019-10-12 10:39:49 -0700 * Use Ubuntu 14.04 for Travis CI environment (Jon Siwek, Corelight) 3.1.0-dev.172 | 2019-10-08 15:14:35 -0700 * Add new BIF: print_raw() This is a convenience function to make it easier to print literal byte sequences to stdout without additional escaping like what may be added by the default `print` statement behavior. For example, related to GH-596, `print` currently escapes even valid UTF-8 byte sequences and makes it difficult to output valid JSON strings containing such. (Jon Siwek, Corelight) * Fix compilation on OS-X. (Johanna Amann) 3.1.0-dev.169 | 2019-10-07 10:50:35 -0700 * GH-589: improve printing of sub-microsecond intervals (Jon Siwek, Corelight) 3.1.0-dev.167 | 2019-10-07 10:42:15 -0700 * GH-591: allow Config::set_value() to use empty/unspecified table/sets (Jon Siwek, Corelight) * GH-591: fix reading set[enum] values from input files (Jon Siwek, Corelight) 3.1.0-dev.164 | 2019-10-02 11:04:13 -0700 * Fix build with external CAF but bundled Broker (Dominik Charousset) 3.1.0-dev.161 | 2019-10-01 16:41:35 -0700 * Update COPYING.3rdparty (Tim Wojtulewicz, Corelight) * Use json::emplace to avoid some extra calls to operator[] (Tim Wojtulewicz, Corelight) * Use tessil/unordered-map instead of nlohmann/fifo-map to improve JSON logging performance (Tim Wojtulewicz, Corelight) 3.1.0-dev.156 | 2019-10-01 09:05:49 +0000 * Improve RecordVal JSON formatting to no longer create a record introspection table each time when all the required information can be obtained directly in the RecordVal and RecordType objects. (Jon Siwek, Corelight) 3.1.0-dev.153 | 2019-09-30 21:09:55 -0700 * GH-602: fix paraglob header inclusion, installation, and search paths (Jon Siwek, Corelight) 3.1.0-dev.151 | 2019-09-30 17:39:50 -0700 * GH-606: Output nulls into json data if a field isn't set (Tim Wojtulewicz, Corelight) * GH-598: handle multi-key sets correctly when outputting json (Tim Wojtulewicz, Corelight) * Fix Xcode deprecation warning for std::ptr_fun (Jon Siwek, Corelight) 3.1.0-dev.143 | 2019-09-24 10:23:24 -0700 * Simplify reassembly code using DataBlockList::LastBlock() (Jon Siwek, Corelight) * Fix DataBlockList::DataSize() (Jon Siwek, Corelight) Parameters got filled opposite to what they were supposed to be and also didn't consider cutoffs that land in the middle of a block. 3.1.0-dev.141 | 2019-09-24 10:31:50 +0000 * Reorganize reassembly data structures. This replaces the previous linked list-based implementation with std::map to avoid O(n) worst case performance. (Jon Siwek, Corelight) 3.1.0-dev.131 | 2019-09-23 13:07:09 -0700 * Add --build-dir as alias for --builddir (Dominik Charousset, Corelight) * Fix overly specific include_directories usage in CMake script (Dominik Charousset, Corelight) 3.1.0-dev.127 | 2019-09-20 10:40:54 -0700 * Fix uncaught InterpreterException in Frame cloning (Robin Sommer, Corelight) Fix for CIDs 1402823 and 1394050. 3.1.0-dev.125 | 2019-09-19 16:40:10 -0700 * Update header include guard in zeek-config.h/bro-config.h (Jon Siwek, Corelight) 3.1.0-dev.124 | 2019-09-17 11:16:47 -0700 * Add speculative service script. (Jan Grashoefer) The speculative service script handles dpd_late_match events to extend conn.log with infos about potential protocol identifications. * Allow to handle late DPD matches. (Jan Grashoefer) If "dpd_match_only_beginning" is disabled, matches of protocol signatures can be handeld using protocol_late_match. To prevent further matching in this case, the new option "dpd_late_match_stop" may be activated. 3.1.0-dev.118 | 2019-09-17 17:21:58 +0000 * GH-566: Fix cases where ssh_encrypted_packet event wasn't raised. When encrypted data was bundled within the same segment as the NewKeys message, it wasn't not reported via a ssh_encrypted_package event as it should have been. (Jon Siwek, Corelight) 3.1.0-dev.116 | 2019-09-17 10:08:38 -0700 * Switch from header guards to pragma once (Dominik Charousset, Corelight) 3.1.0-dev.114 | 2019-09-17 09:32:34 -0700 * GH-580: Add additional 3rd party headers to includes (Derek Ditch) Adds FIFO Map and JSON headers to include distribution 3.1.0-dev.112 | 2019-09-16 17:11:41 -0700 * Add move assignment operator to logging::Tag (Tim Wojtulewicz, Corelight) Coverity 1357732 * Throw an error during input::Manager::CreateTableStream if val is null (Tim Wojtulewicz, Corelight) Coverity 1402730 * Use ntohl instead of manually swapping bytes in extract_XDR_uint32 (Tim Wojtulewicz, Corelight) Coverity 1375796 and others * Add check for null init variable during make_var. (Tim Wojtulewicz, Corelight) Coverity 1403419 * Remove dead check in Val::check_and_promote. (Tim Wojtulewicz, Corelight) Coverity 1401810 * Handle failure of fcntl in Pipe. (Tim Wojtulewicz, Corelight) Coverity 1241934, 1241935 * Initialize missing field in File. (Tim Wojtulewicz, Corelight) Coverity 1057851, 1057852, 1057853 * Add null check for results of dynamic_cast in AssignExpr::TypeCheck. (Tim Wojtulewicz, Corelight) Coverity 1403416 and 1403417 * Add null check when getting key size for List types. (Tim Wojtulewicz, Corelight) Coverity 1058242 3.1.0-dev.102 | 2019-09-16 10:50:47 -0700 * Stop loading scan detection in local.zeek by default (Jon Siwek, Corelight) 3.1.0-dev.98 | 2019-09-04 16:28:01 -0700 * Update Broker include dir search path (Jon Siwek, Corelight) 3.1.0-dev.96 | 2019-09-04 13:41:42 -0700 * Update embedded CAF to 0.17.1 (plus cherry-picked memory leak fix) (Jon Siwek, Corelight) 3.1.0-dev.93 | 2019-09-03 10:35:16 -0700 * Fix minor typos in code comments (The Alchemist) 3.1.0-dev.89 | 2019-08-29 13:05:09 -0700 * CID 1404734: fix NetSessions::MemoryAllocation() (Jon Siwek, Corelight) 3.1.0-dev.88 | 2019-08-28 12:48:04 -0700 * GH-545: add "addl" parameter to flow_weird and net_weird events (Jon Siwek, Corelight) * GH-554: don't init PIA endpoint matchers if there's only file-magic The logic for initializing PIA endpoint matchers was previously skipped if "there's no global rule matcher", and that's only true when no signature files get loaded. But when using `zeek -b`, some file-magic signatures still get loaded by default, so the PIA endpoint matchers still get initialized even though they don't need to be -- file-magic patterns play no part in PIA. For typical use-cases (not using the `-b` flag), this change won't help any, but we do at least use `-b` often within the test suite. (Jon Siwek, Corelight) * GH-554: remove use of file magic in protocol-based signature logic This can be a significant performance/memory improvement since otherwise the protocol-based rule matching logic ends up superfluously creating file-matching state per file-matcher per connection/endpoint. (Jon Siwek, Corelight) * GH-541: add test cases for NTLM AV Pair sequence handling (Jon Siwek, Corelight) * GH-541: fix handling of NTLM AV Pair sequences Empty AV Pair sequences or AV Pair sequences that lack a terminator could cause accesses past the end of the parsed vector. (Jon Siwek, Corelight) * Make Tag::Error values constant (Jon Siwek, Corelight) * Simplify operator bool()'s used for Tag error checks (Jon Siwek, Corelight) * Add/use unspecified IPAddr constants The analyzer-scheduling code was otherwise frequently converting the unspecified v4/v6 addresses from strings. (Jon Siwek, Corelight) 3.1.0-dev.75 | 2019-08-27 10:04:43 -0700 * Fix parsing of smb3 negotiate context (Pavel Ershov) 3.1.0-dev.73 | 2019-08-26 14:57:45 -0700 * Close down SSL/sqlite at shutdown even if net_run not used (Tim Wojtulewicz, Corelight) * Finish processing program arguments before setting up SSL/sqlite (Tim Wojtulewicz, Corelight) 3.1.0-dev.70 | 2019-08-23 14:31:17 -0700 * Make github identify our Flex source correctly. (Seth Hall, Corelight) 3.1.0-dev.68 | 2019-08-23 06:34:50 -0400 * Fix unit tests for new ordering from NetSessions::Drain (Tim Wojtulewicz, Corelight) * Change FragReassembler to use a tuple as a key and use std::map for fragments in Sessions (Tim Wojtulewicz, Corelight) * Rework Session/Connection tracking to use a std::map instead of PDict (Tim Wojtulewicz, Corelight) * Improve GitHub language identification/stats (Seth Hall, Corelight) * Zeekify thread naming. (Seth Hall, Corelight) I copied the same style that caf uses ("zk" with single dot and no space). This gives some consistency with caf and avoids us wasting more space beyond "bro: ". OSs only give 16 characters for thread names so anything we can gain here is nice. 3.1.0-dev.58 | 2019-08-16 18:45:39 +0000 * GH-532: Improve disable_analyzer BIF. (Jon Siwek, Corelight) - Add an extra "prevent" parameter (default value of false), which helps prevent the same analyzer type from being attached in the future. - Fixes disable_analyzer() to work when called even earlier, like within the protocol_confirmation event. - Fixes disable_analyzer() when called on an analyzer added to the tree via TCP_Analyzer::AddChildPacketAnalyzer. 3.1.0-dev.55 | 2019-08-14 16:18:44 -0700 * Fix misc. Coverity warnings (Jon Siwek, Corelight) 3.1.0-dev.54 | 2019-08-14 15:38:02 -0700 * Deprecate int/uint{8,16,32,64} typedefs, replace with actual cstdint types (Tim Wojtulewicz, Corelight) 3.1.0-dev.52 | 2019-08-14 13:46:40 -0700 * Change file_analysis::Manager::ignored to use std::set (Jon Siwek, Corelight) 3.1.0-dev.50 | 2019-08-14 12:32:56 -0700 * Update documentation for string_to_pattern BIF (Jon Siwek, Corelight) 3.1.0-dev.48 | 2019-08-13 20:15:17 -0700 * Cleanups related to PDict -> std::map replacements (Jon Siwek, Corelight) * Replace various uses of PDict with std::map (Tim Wojtulewicz, Corelight) 3.1.0-dev.40 | 2019-08-13 23:44:45 +0000 * Change over to whitelisting clang-tidy options instead of blacklisting. (Tim Wojtulewicz, Corelight) * Use FindClangTidy from the cmake submodule. (Tim Wojtulewicz, Corelight) 3.1.0-dev.36 | 2019-08-13 22:42:54 +0000 * Replace use of deprecated pcap_lookupdev(). (Jon Siwek, Corelight) * Use _exit() in Reporter::FatalError. (Jon Siwek, Corelight) * GH-533: Use consistent "lib" install dir. (Jon Siwek, Corelight) 3.1.0-dev.30 | 2019-08-13 13:48:47 -0700 * Add new LogAscii::gzip_file_extension option. (Tim Wojtulewicz, Corelight) This can be used with the LogAscii::gzip_level option to set the file extension of log files when they are compressed at creation time. 3.1.0-dev.28 | 2019-08-13 12:01:44 -0700 * Remove redundant buffering in ContentLine analyzer (Justin Azoff) The contentline analyzer has two code paths that buffer data: * right at the top of DeliverStream * later in DoDeliverOnce However, contentline can be in plain delivery mode, and if so, the buffer resize in DeliverStream does not need to be done just because DeliverStream was passed an 8K data chunk. This was causing contentline to resize it's buffer to fit chunks of HTTP response data. Additionally, the buffer was sized to be 3/2 of the chunk, so an 8K chunk would result in a 12K allocation. 3.1.0-dev.26 | 2019-08-13 11:25:20 -0700 * Add tests for {http,mime}_all_headers events (Jon Siwek, Corelight) And remove unnecessary check for mime_all_headers in HTTP entities (they ony raise the http_all_headers event, never mime_all_headers). 3.1.0-dev.24 | 2019-08-12 19:30:26 -0700 * Avoid buffering all http/mime headers (Justin Azoff) Only buffer all http/mime headers if the http_all_headers or mime_all_headers events are in use. 3.1.0-dev.22 | 2019-08-12 13:31:12 -0700 * GH-535: fix typo of "C_HESIOD" in DNS::classes (Jon Siwek, Corelight) 3.1.0-dev.21 | 2019-08-12 13:00:21 -0700 * Add new distro to Travis CI configuration for running leak tests (Tim Wojtulewicz, Corelight) 3.1.0-dev.18 | 2019-08-09 10:43:28 -0700 * GH-419: improve multi-protocol logging in known_services.log (Mauro Palumbo) Previously, when multiple protocols were detected on a given addr/port pair, not all protocols were always logged. 3.1.0-dev.7 | 2019-08-09 09:56:06 -0700 * Remove empty services from known_services.log (Mauro Palumbo) * Add check to log to known_services.log when removing active udp connections (Mauro Palumbo) 3.1.0-dev.5 | 2019-08-09 09:33:22 -0700 * Add Intel::read_error event to allow custom error handling (Mauro Palumbo) * Export Intel::read_entry event for general user access (Mauro Palumbo) 3.1.0-dev | 2019-08-08 16:08:50 -0700 * Change version to 3.1.0-dev (Jon Siwek, Corelight) 2.6-767 | 2019-08-07 12:27:21 -0700 * Add memory leak test of closure (un)serialization (Jon Siwek, Corelight) 2.6-765 | 2019-08-07 08:05:35 -0700 * GH-527: fix parsing of MQTT Remaining Length field Packet length is encoded in up to four bytes, with MSB (0x80) indicating if there's more bytes in the representation still to follow. The comparison/bitwise-mask wasn't correctly testing the MSB. Coverity CID 1403964 (Jon Siwek, Corelight) * GH-527: fix LambdaExpr::Traverse Coverity CID 1403966 (Jon Siwek, Corelight) * GH-527: fix ref-counting issues in Frame unserialization Coverity CIDs 1403968, 1403967 (Jon Siwek, Corelight) 2.6-760 | 2019-08-05 21:01:16 -0700 * GH-474: change MQTT::max_payload_size to be a runtime option (Jon Siwek, Corelight) 2.6-758 | 2019-08-05 18:59:53 -0700 * Fix malformed SMB documentation (Jon Siwek, Corelight) * Fix documentation warnings for MQTT identifiers (Jon Siwek, Corelight) 2.6-756 | 2019-08-05 17:32:33 -0700 * Disable MQTT by default (Johanna Amann, Corelight) To enable MQTT, one has to load policy/scripts/mqtt. Like with smb in 2.5, the consts are loaded by default. 2.6-754 | 2019-08-05 10:12:51 -0700 * GH-474: add MQTT::max_payload_size option This caps size of payload strings within mqtt_publish events and mqtt_publish.log files. A new "payload_len" field in the log file shows the real payload size in cases where it may have been truncated. (Jon Siwek, Corelight) * GH-474: use topic vectors for MQTT (un)subscribe events/logs (Jon Siwek, Corelight) * Update Certificate Transparency list (Johanna Amann) * Update CA store to NSS 3.45 (Johanna Amann) 2.6-748 | 2019-08-02 11:55:46 -0700 * GH-517: fix MQTT suback/unsuback accessing non-existent index (Jon Siwek, Corelight) 2.6-747 | 2019-08-02 11:30:52 -0700 * Fix how Broker/CAF sleep duration options are set (Jon Siwek, Corelight) * Add duration thresholding to the conn-size analyzer. (Johanna Amann, Corelight) Now, in addition to setting thresholds for bytes and packet, one can set a threshold for connection duration. Note that the threshold event is only raised once the next packet in the connection is seen. This also fixes a small pre-existing bug, in which a bunch of warnings were raised if someone just used the lower-level functions without going through the higher-level scripting API. 2.6-744 | 2019-08-01 13:33:12 -0700 * Guarantee unique internal name for each lambda function By dealing with hash collisions. (Jon Siwek, Corelight) * Use consistent hashing method for internal lambda function names The results of std::hash may vary depending on platform. E.g. test suite failed on macOS due to Linux generating different lambda function names. (Jon Siwek, Corelight) * Improve error messages from to_addr and to_subnet BIFs. (Jon Siwek, Corelight) 2.6-739 | 2019-08-01 12:28:25 -0700 * Fix a test that used a hardcoded Broker port (Jon Siwek, Corelight) 2.6-737 | 2019-08-01 11:22:57 -0700 * GH-512: add --mandir configure option (Jon Siwek, Corelight) 2.6-735 | 2019-07-31 21:29:58 -0700 * Fix the link to "good first issue" tickets. (Seth Hall, Corelight) 2.6-733 | 2019-07-31 21:23:37 -0700 * Add MQTT analyzer, updated/ported from original version by Supriya Kumar (Seth Hall, Corelight) This analyzer generates three logs to fully display what is happening over the MQTT connection. - mqtt_connect.log - mqtt_subscribe.log - mqtt_publish.log At this time it only supports MQTT 3.1 and 3.1.1 2.6-729 | 2019-07-31 14:24:44 -0700 * Rename a broxygen unit test to zeekygen (Jon Siwek, Corelight) 2.6-728 | 2019-07-31 14:15:29 -0700 * Fix hello world script in READMEs (Seth Hall, Corelight & Jon Siwek, Corelight) * Fixes a tiny Bro->Zeek renaming issue (Seth Hall, Corelight) 2.6-725 | 2019-07-31 10:44:05 -0700 * GH-506: fix NTP script errors (Seth Hall, Corelight) 2.6-723 | 2019-07-30 19:36:56 -0700 * Add LogAscii::enable_utf_8 option (Dev Bali, Corelight) This option allows valid utf8 sequences to be written directly into the ASCII logs without any escaping. 2.6-713 | 2019-07-30 18:12:49 +0000 * Fix memory leaks in expire_func introduced by recent changes (Jon Siwek, Corelight) 2.6-711 | 2019-07-29 20:15:27 -0700 * Fix duplicate TCP packets not being detected as retransmissions (Jeff Barber) 2.6-708 | 2019-07-30 02:46:39 +0000 * Add an additional license file, COPYING.3rdparty, that collects all third party software licenses. (Johanna Amann, Corelight) 2.6-706 | 2019-07-30 02:30:44 +0000 * Zeek's anonymous functions now capture their closures by reference. This means that they can use and modify variables from the scope that they were generated in. For example: local n = 3; local f = function() { n += 1; }; f(); print n; # prints 4 See NEWS and the documentations for more details. (Zeke Medley, Corelight) 2.6-674 | 2019-07-29 16:11:42 -0400 * added more options to SMB2 set-info command (Mauro Palumbo) 2.6-664 | 2019-07-29 10:01:12 -0700 * Add release branches to Travis CI whitelist (Jon Siwek, Corelight) * GH-488: teach the Version module to parse new version scheme Such as the new -rc format for release candidates (replacing "beta") and -dev.X for development versions in the master branch. (Jon Siwek, Corelight) * Improve type inference for vector-of-enum constructor (Jon Siwek, Corelight) * GHI-486: Switch over to using LLVM utf8-checking code to better validate characters (Tim Wojtulewicz, Corelight) * Fix undefined behavior via casting file analyzers to protocol analyzers When generating some events for PE and X509 file analyzers, there's an invalid cast from file_analysis::Analyzer to analyzer::Analyzer and subsequent invalid member access via analyzer::Analyzer::GetID() called on what is really a pointer to a file analyzer. (Jon Siwek, Corelight) * Fix undefined behavior via hrw_weight BIF signed int overflow (Jon Siwek, Corelight) * Fix undefined behavior via invalid TCP analyzer cast A connection's root analyzer isn't necessarily TCP and an unchecked C-style cast is undefined behavior in those cases. (Jon Siwek, Corelight) * GH-485: fix cases where DHCP log omits MAC field The field is populated in this order of preference: (1) Use a client-identifier option sent by client (2) Use the server's CHADDR field (3) Use the client's CHADDR field Case (3) did not exist before this patch. (Jon Siwek, Corelight) 2.6-649 | 2019-07-23 09:37:05 -0700 * Remove --disable-perftools from coverity builds (Jon Siwek, Corelight) 2.6-647 | 2019-07-22 12:46:04 -0700 * Ignore abs-path in test. (Zeke Medley, Corelight) * Report argument # type check failed on. (Zeke Medley, Corelight) * Update test baseline. (Zeke Medley, Corelight) * Improve func arg type checking. (Zeke Medley, Corelight) * &expire_func(table, arg1, arg2, ...) + type checking. (Zeke Medley, Corelight) 2.6-640 | 2019-07-22 12:22:08 -0700 * Mark List::insert deprecated in favor of push_front (Tim Wojtulewicz, Corelight) * Mark List::sort as deprecated, remove List::sortedinsert (Tim Wojtulewicz, Corelight) * Simplify container iterators to just use pointers directly (Tim Wojtulewicz, Corelight) 2.6-635 | 2019-07-22 10:58:56 -0700 * Add DPD::max_violations option This allows one to tune the number of protocol violations to tolerate from any given analyzer type before just disabling a given instance of it. Also removes the "disabled_aids" field from the DPD::Info record since it serves no purpose: in this case, calling disable_analyzer multiple times for the same analyzer is a no-op. (Jon Siwek, Corelight) * Abort when --enable-jemalloc is given, but JeMalloc is not found (Johanna Amann, Corelight) * GH-475: Require --enable-perftools to link in tcmalloc. (Jon Siwek, Corelight) 2.6-630 | 2019-07-19 18:01:44 -0700 * Fix typo in NEWS (Jon Siwek, Corelight) 2.6-628 | 2019-07-19 15:48:58 +0000 * Support old plugins that may still reference "bro-config.h", "bro-path-dev", or "bro" inside the build tree through compatibility symlinks/wrappers. (Jon Siwek, Corelight) 2.6-626 | 2019-07-18 19:44:31 -0700 * Remove unused expression type tag: EXPR_MATCH (Jon Siwek, Corelight) * Add missing expr_name: vector_coerce (Jon Siwek, Corelight) 2.6-624 | 2019-07-16 16:07:59 +0000 * Redo README. We now have separate plain text and Markdown versions. (Zeke Medley, Corelight) 2.6-616 | 2019-07-16 15:21:37 +0000 * Fix referecne counting bug in EnumType copy constructor. (Jon Siwek, Corelight) * Remove unused BroObj::in_ser_cache member. (Jon Siwek, Corelight) 2.6-612 | 2019-07-15 19:46:04 -0700 * Fix a potential usage of List::remove_nth(-1) (Jon Siwek, Corelight) * Change List::remote(const T&) to return a bool (Jon Siwek, Corelight) It now indicates whether the removal took place or not, depending on whether a matching element was found in the list. * Fix debug build due to old int_list usage within assert (Jon Siwek, Corelight) * Convert uses of loop_over_list to ranged-for loops (Tim Wojtulewicz, Corelight) * Remove loop_over_queue (as an example for later removing loop_over_list) (Tim Wojtulewicz, Corelight) * Change int_list in CCL.h to be a vector, fix uses of int_list to match (Tim Wojtulewicz, Corelight) * Remove List<> usage from strings.bif (Tim Wojtulewicz, Corelight) * Replace uses of the old Queue/PQueue generation code with new template versions (Tim Wojtulewicz, Corelight) * Convert BaseQueue/Queue/PQueue into templates, including iterator support (Tim Wojtulewicz, Corelight) * Replace uses of the old Dict generation code with new template versions (Tim Wojtulewicz, Corelight) * Convert PDict into template (Tim Wojtulewicz, Corelight) * Replace uses of the old List generation code with new template versions (Tim Wojtulewicz, Corelight) * Convert BaseList/List/PList into templates, including iterator support (Tim Wojtulewicz, Corelight) 2.6-598 | 2019-07-12 18:20:12 -0700 * Fix canonification of timestamps with a decisecond multiple (Jon Siwek, Corelight) 2.6-597 | 2019-07-12 15:01:56 -0700 * Fix a couple of resource leaks from JSON changes (Tim Wojtulewicz, Corelight) 2.6-595 | 2019-07-12 13:34:08 -0700 * GH-157: Mark some attributes as not allowed for global variables (Tim Wojtulewicz, Corelight) This disallows &default for global values that are not tables, and &optional for all globals. * Fix uncaught exceptions from Val cloning failures (Jon Siwek, Corelight) 2.6-591 | 2019-07-11 13:29:28 -0700 * Fix potential thread safety issue with zeekenv util function Observed segfault accessing the local static std::map of zeekenv() from a logging thread, but only in non-debug builds using Apple/Clang compiler, not in a debug build or GCC. Don't quite get this behavior since static local variable initialization is supposed to be thread-safe since C++11, but moving to a global static works and is "more efficient" anyway since there's no longer any run-time overhead. (Jon Siwek, Corelight) 2.6-589 | 2019-07-11 13:14:52 -0700 * GH-421: fix bugs/regressions in DNP3 analyzer (Hui Lin) 2.6-587 | 2019-07-11 12:13:48 -0700 * Fix a sign-compare compiler warning (Jon Siwek, Corelight) 2.6-586 | 2019-07-11 11:15:40 -0700 * Convert all JSON output to use an external library for better consistency (Tim Wojtulewicz, Corelight) See NEWS for more details; this makes to_json a bif and causes slight changes in its output, as well as the output of the JSON logger. 2.6-576 | 2019-07-10 18:38:54 -0700 * Remove unused option: chunked_io_buffer_soft_cap (Jon Siwek, Corelight) 2.6-575 | 2019-07-09 18:28:03 -0700 * Avoid a null dereference (Coverity-1402816) (Tim Wojtulewicz, Corelight) * Avoid resource leaks (Coverity-1402818, Coverity-1402812) (Tim Wojtulewicz, Corelight) * Avoid null dereference in broker (Coverity-1402824, Coverity-1402814) (Tim Wojtulewicz, Corelight) * Improve stability of a unit test (Jon Siwek, Corelight) 2.6-569 | 2019-07-03 13:03:22 -0700 * Improve stability of a unit test (Jon Siwek, Corelight) 2.6-568 | 2019-07-03 11:50:56 -0700 * Add clang-tidy rule to CMake including a base configuration (Tim Wojtulewicz, Corelight) 2.6-566 | 2019-07-03 11:08:24 -0700 * Improve Zeekygen output for long attribute expressions (Jon Siwek, Corelight) 2.6-565 | 2019-07-03 09:32:34 -0700 * GH-446: Deprecate rfb_event. (Johanna Amann, Corelight) 2.6-563 | 2019-07-03 01:57:40 -0700 * Fix CIF integration and add logging options to intel.log and added comments to code (sfinlon) 2.6-558 | 2019-07-01 01:27:50 -0700 * GH-443: fix uses of timestamp 0 in cluster diagnostic logs For broker.log and cluster.log: there was a race condition. A worker's first IOSource that it processes is potentially Broker if there were no packets available yet and thread scheduling happens to work out such that network connections (inside CAF threads) become established before we enter the main I/O loop. Such peering establishments would generate logs with timestamp 0 as there was not yet any code path taken that would update network_time. For reporter.log: any non-worker (packet-processing) node would just unnecessarily use a timestamp of 0 for their reporter messages. (Jon Siwek, Corelight) 2.6-556 | 2019-07-01 00:56:13 -0700 * Improve deprecation warning messages (Jon Siwek, Corelight) * Remove deprecated DNS events - dns_full_request - non_dns_request (Jon Siwek, Corelight) * Remove BackDoor analyzer (Jon Siwek, Corelight) * Remove InterConn analyzer (Jon Siwek, Corelight) * Remove deprecated/unused irc_servers option (Jon Siwek, Corelight) * Remove deprecated print_hook event (Jon Siwek, Corelight) * Remove dead code: dump_used_event_handlers (Jon Siwek, Corelight) * Remove unused software_version_found events - software_version_found - software_unparsed_version_found - software_parse_error (Jon Siwek, Corelight) * Remove deprecated open_log_file and log_file_name functions (Jon Siwek, Corelight) * Remove deprecated/unused "packet" type (Jon Siwek, Corelight) * Un-deprecate anonymizer BIFs (Jon Siwek, Corelight) * Un-deprecate file rotation functions - rotate_file - rotate_file_by_name - calc_next_rotate These still have use-cases even though no longer used for our logging functionality. E.g. rotate_file_by_name may be used to rotate pcap dump files. Also the log_rotate_base_time option was marked deprecated, but still used in the new logging framework. (Jon Siwek, Corelight) * Switch default CAF scheduler policy to work sharing It may generally be better for our default use-case, as workers may save a few percent cpu utilization as this policy does not have to use any polling like the stealing policy does. This also helps avoid a potential issue with the implementation of spinlocks used in the work-stealing policy in current CAF versions, where there's some conditions where lock contention causes a thread to spin for long periods without relinquishing the cpu to others. (Jon Siwek, Corelight) * Update sqlite to 3.28.0. (Johanna Amann, Corelight) * GH-320: Improve RFB (VNC) protocol parsing Parsing now stops for both client and server if either encounters any parsing error or invalid state. After a complete handshake, server messages are no longer parsed. Support for that is incomplete and not sure it's that useful anyway since it mostly contains pixel data. (Jon Siwek, Corelight) 2.6-536 | 2019-06-28 12:10:55 -0700 * Add Windows Minidump file signature (Alexander Bolshakov) 2.6-534 | 2019-06-28 11:48:41 -0700 * Change notices to be processed on worker. (Johanna Amann, Corelight) In the past they were processed on the manager - which requires big records to be sent around. This has a potential of incompatibilities if someone relied on global state for notice processing. Also may prevent notice de-duplication due to expected race condition of suppression messages taking time to proaogate out to all cluster nodes. 2.6-531 | 2019-06-27 12:09:08 -0700 * GH-375: Remove the BroFile cache (Johanna Amann, Corelight) 2.6-529 | 2019-06-27 10:12:34 -0700 * Fix creating a StringVal from std::string. (Johanna Amann, Corelight) Currently, creating a StringVal from a std::string did not work with data that contains \0 characters. This easy fix changes this - and should also lead to a small speed increase for code using this constructor. This obviously means that more data might copied now in some cases that were previously cut off at the first 0-byte. Our test-suite did not reveal any such cases. 2.6-526 | 2019-06-25 12:45:31 -0700 * Make a paraglob unit test parallelizable (Jon Siwek, Corelight) 2.6-523 | 2019-06-25 10:38:24 -0700 * GH-427: improve default ID values shown by Zeekygen The default value of an ID is now truly the one used to initialize it, unaltered by any subsequent redefs. Redefs are now shown separately, along with the expression that modifies the ID's value. (Jon Siwek, Corelight) * Unbreak build on Linux (Johanna Amann, Corelight) 2.6-519 | 2019-06-24 15:25:08 -0700 * GH-435: fix null pointer deref in RPC analyzer. (Jon Siwek, Corelight) 2.6-517 | 2019-06-24 15:20:39 -0700 * Add paraglob, a fairly quick data structure for matching a string against a large list of patterns. (Zeke Medley, Corelight) * GH-171: support warning messages alongside deprecated attributes (Tim Wojtulewicz, Corelight) 2.6-503 | 2019-06-21 11:17:58 -0700 * GH-417: Remove old, unmaintained p0f support. (Johanna Amann, Corelight) 2.6-500 | 2019-06-20 20:54:15 -0700 * Add new RDP event: rdp_client_cluster_data (Jeff Atkinson) * Added "options" field to RDP::ClientChannelDef (Jeff Atkinson) 2.6-494 | 2019-06-20 20:24:38 -0700 * Renaming src/StateAccess.{h,cc} to src/Notifier.{h,cc}. The old names did not reflect the content of the files anymore. (Robin Sommer, Corelight) * Remove MutableVal, StateAccess classes, enum Opcode. (Robin Sommer, Corelight) * Redo API for notifiers. There's now an notifier::Modifiable interface class that class supposed to signal modifications are to be derived from. This takes the place of the former MutableValue class and also unifies how Val and IDs signal modifications. (Robin Sommer, Corelight) * Redo NotfifierRegistry to no longer rely on StateAccess. We simplify the API to a simple Modified() operation. (Robin Sommer, Corelight) * Add new test for when-statement watching global variables. (Robin Sommer, Corelight) 2.6-482 | 2019-06-20 19:57:20 -0700 * Make configure complain if submodules are not checked out. (Johanna Amann, Corelight) * Improve C++ header includes to improve build time (Jon Siwek, Corelight) 2.6-479 | 2019-06-20 18:31:58 -0700 * Fix TableVal::DoClone to use CloneState cache (Jon Siwek, Corelight) 2.6-478 | 2019-06-20 14:19:11 -0700 * Remove old Broccoli SSL options (Jon Siwek, Corelight) - ssl_ca_certificate - ssl_private_key - ssl_passphrase 2.6-477 | 2019-06-20 14:00:22 -0700 * Remove unused SerialInfo.h and SerialTypes.h headers (Jon Siwek, Corelight) 2.6-476 | 2019-06-20 13:23:22 -0700 * Remove opaque of ocsp_resp. (Johanna Amann, Corelight) Only used in one event, without any way to use the opaque for anything else. At this point this just seems like a complication that has no reason to be there. * Remove remnants of event serializer. (Johanna Amann, Corelight) * Reimplement serialization infrastructure for OpaqueVals. (Robin Sommer, Corelight & Johanna Amann, Corelight) We need this to sender through Broker, and we also leverage it for cloning opaques. The serialization methods now produce Broker data instances directly, and no longer go through the binary formatter. Summary of the new API for types derived from OpaqueVal: - Add DECLARE_OPAQUE_VALUE() to the class declaration - Add IMPLEMENT_OPAQUE_VALUE() to the class' implementation file - Implement these two methods (which are declated by the 1st macro): - broker::data DoSerialize() const - bool DoUnserialize(const broker::data& data) This machinery should work correctly from dynamic plugins as well. OpaqueVal provides a default implementation of DoClone() as well that goes through serialization. Derived classes can provide a more efficient version if they want. The declaration of the "OpaqueVal" class has moved into the header file "OpaqueVal.h", along with the new serialization infrastructure. This is breaking existing code that relies on the location, but because the API is changing anyways that seems fine. * Implement a Shallow Clone operation for types. (Johanna Amann, Corelight) This is needed to track name changes for the documentation. * Remove old serialization infrastrucutre. (Johanna Amann, Corelight) 2.6-454 | 2019-06-19 09:39:06 -0700 * GH-393: Add slice notation for vectors (Tim Wojtulewicz, Corelight & Jon Siwek, Corelight) Example Syntax: local v = vector(1, 2, 3, 4, 5); v[2:4] = vector(6, 7, 8); # v is now [1, 2, 6, 7, 8, 5] print v[:4]; # prints [1, 2, 6, 7] 2.6-446 | 2019-06-17 20:26:49 -0700 * Rename bro to zeek in error messages (Daniel Thayer) 2.6-444 | 2019-06-15 19:09:03 -0700 * Add/rewrite NTP support (Vlad Grigorescu and Mauro Palumbo) 2.6-416 | 2019-06-14 20:57:57 -0700 * DNS: Add support for SPF response records (Vlad Grigorescu) 2.6-413 | 2019-06-14 19:51:28 -0700 * GH-406: rename bro.bif to zeek.bif (Jon Siwek, Corelight) 2.6-412 | 2019-06-14 19:26:21 -0700 * GH-387: update Broker topic names to use "zeek/" prefix (Jon Siwek, Corelight) * GH-323: change builtin plugin namespaces to Zeek (Jon Siwek, Corelight) 2.6-408 | 2019-06-13 11:19:50 -0700 * Fix potential null-dereference in current_time() (Tim Wojtulewicz, Corelight) * Add --sanitizers configure script to enable Clang sanitizers (Tim Wojtulewicz, Corelight) 2.6-404 | 2019-06-12 15:10:19 -0700 * Rename directories from bro to zeek (Daniel Thayer) The new default installation prefix is /usr/local/zeek 2.6-400 | 2019-06-07 20:06:33 -0700 * Adapt bro_plugin CMake macros to use zeek_plugin (Jon Siwek, Corelight) 2.6-399 | 2019-06-07 14:02:18 -0700 * Update SSL documentation. (Johanna Amann) * Support the newer TLS 1.3 key_share extension. (Johanna Amann) * Include all data of the server-hello random (Johanna Amann) Before we cut the first 4 bytes, which makes it impossible to recognize several newer packets (like the hello retry). * Parse TLS 1.3 pre-shared-key extension. (Johanna Amann) Adds new events: - ssl_extension_pre_shared_key_client_hello - ssl_extension_pre_shared_key_server_hello 2.6-391 | 2019-06-07 17:29:28 +1000 * GH-209: replace "remote_ip" field of radius.log with "tunnel_client". Also changes type from addr to string. (Jon Siwek, Corelight) 2.6-389 | 2019-06-06 20:02:19 -0700 * Update plugin unit tests to use --zeek-dist (Jon Siwek, Corelight) 2.6-388 | 2019-06-06 19:48:55 -0700 * Change default value of peer_description "zeek" (Jon Siwek, Corelight) 2.6-387 | 2019-06-06 18:51:09 -0700 * Rename Bro to Zeek in Zeekygen-generated documentation (Jon Siwek, Corelight) 2.6-386 | 2019-06-06 17:17:55 -0700 * Add new RDP event: rdp_native_encrytped_data (Anthony Kasza, Corelight) 2.6-384 | 2019-06-06 16:49:14 -0700 * Add new RDP event: rdp_client_security_data (Jeff Atkinson) 2.6-379 | 2019-06-06 11:56:58 -0700 * Improve sqlite logging unit tests (Jon Siwek, Corelight) 2.6-378 | 2019-06-05 16:23:04 -0700 * Rename BRO_DEPRECATED macro to ZEEK_DEPRECATED (Jon Siwek, Corelight) 2.6-377 | 2019-06-05 16:15:58 -0700 * Deprecate functions with "bro" in them. (Jon Siwek, Corelight) * "bro_is_terminating" is now "zeek_is_terminating" * "bro_version" is now "zeek_version" The old functions still exist for now, but are deprecated. 2.6-376 | 2019-06-05 13:29:57 -0700 * GH-379: move catch-and-release and unified2 scripts to policy/ (Jon Siwek, Corelight) These are no longer loaded by default due to the performance impact they cause simply by being loaded (they have event handlers for commonly generated events) and they aren't generally useful enough to justify it. 2.6-375 | 2019-06-04 19:28:06 -0700 * Simplify threading::Value destructor (Jon Siwek, Corelight) * Add pattern support to input framework. (Zeke Medley, Corelight) 2.6-369 | 2019-06-04 17:53:10 -0700 * GH-155: Improve coercion of expression lists to vector types (Tim Wojtulewicz, Corelight) * GH-159: Allow coercion of numeric record field values to other types (Tim Wojtulewicz, Corelight) * Allow passing a location to BroObj::Warning and BroObj::Error. (Tim Wojtulewicz, Corelight) This allows callers (such as check_and_promote) to pass an expression location to be logged if the location doesn't exist in the value being promoted. * Add CLion directories to gitignore (Tim Wojtulewicz, Corelight) * Move #define outside of max_type for clarity (Tim Wojtulewicz, Corelight) 2.6-361 | 2019-06-04 10:30:21 -0700 * GH-293: Protect copy() against reference cycles. (Robin Sommer, Corelight) Reference cycles shouldn't occur but there's nothing really preventing people from creating them, so may just as well be safe and deal with them when cloning values. 2.6-359 | 2019-05-31 13:37:17 -0700 * Remove old documentation reference to rotate_interval (Jon Siwek, Corelight) 2.6-357 | 2019-05-30 10:57:54 -0700 * Tweak to ASCII reader warning suppression (Christian Kreibich, Corelight) Warnings in the ASCII reader so far remained suppressed even when an input file changed. It's helpful to learn about problems in the data when putting in place new data files, so this change maintains the existing warning suppression while processing a file, but re-enables warnings after updates to a file. 2.6-354 | 2019-05-29 09:46:19 -0700 * Add weird: "RDP_channels_requested_exceeds_max" (Vlad Grigorescu) 2.6-352 | 2019-05-28 17:57:36 -0700 * Reduce data copying in Broker message processing (Jon Siwek, Corelight) * Improve Broker I/O loop integration: less mutex locking (Jon Siwek, Corelight) Checking a subscriber for available messages required locking a mutex, but we should never actually need to do that in the main-loop to check for Broker readiness since we can rely on file descriptor polling. * Improve processing of broker data store responses (Jon Siwek, Corelight) Now retrieves and processes all N available responses at once instead of one-by-one-until-empty. 2.6-345 | 2019-05-28 11:32:16 -0700 * RDP: Add parsing and logging of channels requested by the client. (Vlad Grigorescu) Can determine capabilities requested by the client, as well as attacks such as CVE-2019-0708. 2.6-342 | 2019-05-28 10:48:37 -0700 * GH-168: Improve type-checking for table/set list assignment. (Zeke Medley and Jon Siwek, Corelight) 2.6-340 | 2019-05-24 18:02:43 -0700 * Add support for parsing additional DHCP options (Jay Wren) The following optional fields were added to the DHCP::Options record: - time_offset (Option 2) - time_servers (Option 4) - name_servers (Option 5) - ntp_servers (Option 42) 2.6-338 | 2019-05-24 17:06:08 -0700 * Add input file name to additional ASCII reader warning messages (Christian Kreibich, Corelight) 2.6-336 | 2019-05-24 10:23:20 -0700 * GH-378: check validity of missing 'val' field in Input::add_table (Jon Siwek, Corelight) 2.6-335 | 2019-05-24 08:58:59 -0700 * Fix memory leak when no protocol_violation event handler exists (Jon Siwek, Corelight) 2.6-334 | 2019-05-23 20:40:03 -0700 * Add an internal getenv wrapper function: zeekenv (Jon Siwek, Corelight) It maps newer environment variable names starting with ZEEK to the legacy names starting with BRO. * Rename all BRO-prefixed environment variables (Daniel Thayer) For backward compatibility when reading values, we first check the ZEEK-prefixed value, and if not set, then check the corresponding BRO-prefixed value. 2.6-331 | 2019-05-23 18:03:42 -0700 * Update broker unit test output. (Jon Siwek, Corelight) Due to string representation of Broker vectors changing (they now use parentheses instead of square brackets). 2.6-330 | 2019-05-23 13:04:26 -0700 * GH-173: Support ranges of values for value_list elements in the signature parser (Tim Wojtulewicz, Corelight) * GH-173: Modify the signature parser so ID components can't start with numbers (Tim Wojtulewicz, Corelight) 2.6-327 | 2019-05-23 11:56:11 -0700 * Remove redundant RecordVal::record_type member (Jon Siwek, Corelight) 2.6-326 | 2019-05-23 10:49:38 -0700 * Fix parse-time RecordVal tracking containing duplicates (Jon Siwek, Corelight) 2.6-325 | 2019-05-22 23:56:23 -0700 * Add leak-checks for new copy operations (Johanna Amann, Corelight) * Finish implementation of new copy method. (Johanna Amann, Corelight) All types (besides EntropyVal) now support a native copy operation, which uses primitives of the underlying datatypes to perform a quick copy, without serialization. EntropyVal is the one exception - since that type is rather complex (many members) and will probably not be copied a lot, if at all, it makes sense to just use the serialization function. This will have to be slightly re-written in the near-term-future to use the new serialization function for that opaque type. This change also introduces a new x509_from_der bif, which allows to parse a der into an opaque of x509. This change removes the d2i_X509_ wrapper function; this was a remnant when d2i_X509 took non-const arguments. We directly use d2i_X509 at several places assuming const-ness, so there does not seem to ba a reason to keep the wrapper. This change also exposed a problem in the File cache - cases in which an object was brought back into the cache, and writing occurred in the file_open event were never correctly handeled as far as I can tell. * Reimplement copy(). (Robin Sommer, Corelight) The old implementation used the serialization framework, which is going away. This is a new standalone implementation that should also be quite a bit faster. 2.6-318 | 2019-05-21 09:17:53 -0700 * Remove state_dir and state_write_delay options (Jon Siwek, Corelight) * Remove a reference to &synchronized from docs (Jon Siwek, Corelight) 2.6-316 | 2019-05-20 20:56:46 -0700 * Additional Bro to Zeek renaming (Daniel Thayer) * Added a new unit test for legacy Bro Plugins (Daniel Thayer) * Added a symlink bro-path-dev.in for use by legacy Bro packages (Daniel Thayer) 2.6-314 | 2019-05-20 16:20:33 -0700 * Remove deprecated attributes. (Johanna Amann, Corelight) To be more exact: &encrypt, &mergeable, &rotate_interval, &rotate_size Also removes no longer used redef-able constants: log_rotate_interval, log_max_size, log_encryption_key 2.6-311 | 2019-05-20 09:07:58 -0700 * Add missing &optional attr to KRB record fields; also add existence checks to scripts (Jon Siwek, Corelight). 2.6-308 | 2019-05-17 14:13:46 -0700 * Always emit scripting errors to stderr during zeek_init (Jon Siwek, Corelight) 2.6-307 | 2019-05-16 13:37:24 -0700 * More bro-to-zeek renaming in scripts and other files (Daniel Thayer) * More bro-to-zeek renaming in the unit tests (Daniel Thayer) 2.6-303 | 2019-05-15 15:03:11 -0700 * Changes needed due to bro-to-zeek renaming in broker (Daniel Thayer) 2.6-301 | 2019-05-15 10:05:53 -0700 * Fix potential race in openflow broker plugin (Jon Siwek, Corelight) 2.6-300 | 2019-05-15 09:00:57 -0700 * Fixes to DNS lookup, including ref-counting bugs, preventing starvation of the DNS_Mgr in the I/O loop, dead code removal, and a fix that prevents the timeout of already resolved DNS lookups (Jon Siwek, Corelight) 2.6-292 | 2019-05-14 19:01:05 -0700 * Fix maybe-uninitialized compiler warning (Jon Siwek, Corelight) 2.6-290 | 2019-05-14 18:35:25 -0700 * Update btest.cfg path to use zeek-aux (Jon Siwek, Corelight) 2.6-288 | 2019-05-14 17:47:55 -0700 * Update CMake to use aux/zeekctl and aux/zeek-aux submodules (Jon Siwek, Corelight) 2.6-287 | 2019-05-14 17:40:40 -0700 * Rename broctl submodule to zeekctl (Jon Siwek, Corelight) 2.6-286 | 2019-05-14 13:19:12 -0700 * Undo an unintentional change to btest.cfg from a recent commit (Daniel Thayer) * Fix zeek-wrapper and improve error messages (Daniel Thayer) The script was not passing command-line arguments to the new program. * Update for renaming BroControl to ZeekControl. (Robin Sommer, Corelight) * GH-239: Rename bro to zeek, bro-config to zeek-config, and bro-path-dev to zeek-path-dev. (Robin Sommer, Corelight) This also installs symlinks from "zeek" and "bro-config" to a wrapper script that prints a deprecation warning. 2.6-279 | 2019-05-13 20:02:59 -0700 * GH-365: improve un-indexable type error message (Jon Siwek, Corelight) 2.6-277 | 2019-05-08 12:42:18 -0700 * Allow tuning Broker log batching via scripts (Jon Siwek, Corelight) Via redefining "Broker::log_batch_size" or "Broker::log_batch_interval" 2.6-276 | 2019-05-08 09:03:27 -0700 * Force the Broker IOSource to idle periodically, preventing packet IOSource starvation. (Jon Siwek, Corelight). 2.6-274 | 2019-05-08 08:58:25 -0700 * GH-353: Add `//i` case-insensitive signature syntax (Jon Siwek, Corelight) 2.6-272 | 2019-05-06 18:43:13 -0700 * Remove support for using && and || with patterns. (Johanna Amann, Corelight) This was never documented and previously deprecated. * Remove RemoteSerializer and related code/types. (Johanna Amann, Corelight) Also removes broccoli from the source tree. * Remove PersistenceSerializer. (Johanna Amann, Corelight) * Remove &synchronized and &persistent attributes. (Johanna Amann, Corelight) 2.6-264 | 2019-05-03 11:16:38 -0700 * Fix sporadic openflow/broker test failure (Jon Siwek, Corelight) 2.6-263 | 2019-05-02 22:49:40 -0700 * Install local.zeek as symlink to pre-existing local.bro (Jon Siwek, Corelight) This a convenience for those that are upgrading. If we didn't do this, then deployments can silently break until the user intervenes since BroControl now prefers to load the initially-vanilla local.zeek instead of the formerly-customized local.bro. 2.6-262 | 2019-05-02 21:39:01 -0700 * Rename Zeexygen to Zeekygen (Jon Siwek, Corelight) 2.6-261 | 2019-05-02 20:49:23 -0700 * Remove previously deprecated policy/protocols/smb/__load__ (Jon Siwek, Corelight) 2.6-260 | 2019-05-02 19:16:48 -0700 * GH-243: Remove deprecated functions/events from 2.6 and earlier (Johanna Amann, Corelight) 2.6-258 | 2019-05-02 12:26:54 -0700 * GH-340: Improve IPv4/IPv6 regexes, extraction, and validity functions. is_valid_ip() is not a BIF, the IP regular expressions are improved and extract_ip_addresses should give better results due to this. (Jon Siwek, Corelight) 2.6-255 | 2019-05-01 08:38:49 -0700 * Add methods to queue events without handler existence check Added ConnectionEventFast() and QueueEventFast() methods to avoid redundant event handler existence checks. It's common practice for caller to already check for event handler existence before doing all the work of constructing the arguments, so it's desirable to not have to check for existence again. E.g. going through ConnectionEvent() means 3 existence checks: one you do yourself before calling it, one in ConnectionEvent(), and then another in QueueEvent(). The existence check itself can be more than a few operations sometimes as it needs to check a few flags that determine if it's enabled, has a local body, or has any remote receivers in the old comm. system or has been flagged as something to publish in the new comm. system. (Jon Siwek, Corelight) * Cleanup/improve PList usage and Event API Majority of PLists are now created as automatic/stack objects, rather than on heap and initialized either with the known-capacity reserved upfront or directly from an initializer_list (so there's no wasted slack in the memory that gets allocated for lists containing a fixed/known number of elements). Added versions of the ConnectionEvent/QueueEvent methods that take a val_list by value. Added a move ctor/assign-operator to Plists to allow passing them around without having to copy the underlying array of pointers. (Jon Siwek, Corelight) 2.6-250 | 2019-04-29 18:09:29 -0700 * Remove 'dns_resolver' option, replace w/ ZEEK_DNS_RESOLVER env. var. (Jon Siwek, Corelight) 2.6-249 | 2019-04-26 19:26:44 -0700 * Fix parsing of hybrid IPv6-IPv4 addr literals with no zero compression (Jon Siwek, Corelight) 2.6-246 | 2019-04-25 10:22:11 -0700 * Add Zeexygen cross-reference links for some events (Jon Siwek, Corelight) 2.6-245 | 2019-04-23 18:42:02 -0700 * Expose TCP analyzer utility functions to derived classes (Vern Paxson, Corelight) 2.6-243 | 2019-04-22 19:42:52 -0700 * GH-234: rename Broxygen to Zeexygen along with roles/directives (Jon Siwek, Corelight) * All "Broxygen" usages have been replaced in code, documentation, filenames, etc. * Sphinx roles/directives like ":bro:see" are now ":zeek:see" * The "--broxygen" command-line option is now "--zeexygen" 2.6-242 | 2019-04-22 22:43:09 +0200 * update SSL consts from TLS 1.3 (Johanna Amann) 2.6-241 | 2019-04-22 12:38:06 -0700 * Add 'g' character to conn.log history field to flag content gaps (Vern Paxson, Corelight) There's also a small change to TCP state machine that distrusts ACKs appearing at the end of connections (in FIN or RST) such that they won't count towards revealing a true content gap. 2.6-237 | 2019-04-19 12:00:37 -0700 * GH-236: Add zeek_script_loaded event, deprecate bro_script_loaded (Jon Siwek, Corelight) Existing handlers for bro_script_loaded automatically alias to the new zeek_script_loaded event, but emit a deprecation warning. 2.6-236 | 2019-04-19 11:16:35 -0700 * Add zeek_init/zeek_done events and deprecate bro_init/bro_done (Seth Hall, Corelight) Any existing handlers for bro_init and bro_done will automatically alias to the new zeek_init and zeek_done events such that code will not break, but will emit a deprecation warning. 2.6-232 | 2019-04-18 09:34:13 +0200 * Prevent topk_merge from crashing when second argument is empty set (Jeff Barber) 2.6-230 | 2019-04-17 16:44:16 -0700 * Fix unit test failures on case-insensitive file systems (Jon Siwek, Corelight) 2.6-227 | 2019-04-16 17:44:31 -0700 * GH-237: add `@load foo.bro` -> foo.zeek fallback (Jon Siwek, Corelight) When failing to locate a script with explicit .bro suffix, check for whether one with a .zeek suffix exists and use it instead. 2.6-225 | 2019-04-16 16:07:49 -0700 * Use .zeek file suffix in unit tests (Jon Siwek, Corelight) 2.6-223 | 2019-04-16 11:56:00 -0700 * Update tests and baselines due to renaming all scripts (Daniel Thayer) * Rename all scripts to have ".zeek" file extension (Daniel Thayer) * Add test cases to verify new file extension is recognized (Daniel Thayer) * Fix the core/load-duplicates.bro test (Daniel Thayer) * Update script search logic for new .zeek file extension (Daniel Thayer) When searching for script files, look for both the new and old file extensions. If a file with ".zeek" can't be found, then search for a file with ".bro" as a fallback. * Remove unnecessary ".bro" from @load directives (Daniel Thayer) 2.6-212 | 2019-04-12 10:12:31 -0700 * smb2_write_response event added (Mauro Palumbo) 2.6-210 | 2019-04-10 09:54:27 -0700 * Add options to tune BinPAC flowbuffer policy (Jon Siwek, Corelight) 2.6-208 | 2019-04-10 11:36:17 +0000 * Improve PE file analysis (Jon Siwek, Corelight) * Set PE analyzer CMake dependencies correctly (Jon Siwek, Corelight) 2.6-205 | 2019-04-05 17:06:26 -0700 * Add script to update external test repo commit pointers (Jon Siwek, Corelight) 2.6-203 | 2019-04-04 16:35:52 -0700 * Update DTLS error handling (Johanna Amann, Corelight) - Adds tuning options: SSL::dtls_max_version_errors and SSL::dtls_max_reported_version_errors 2.6-200 | 2019-04-03 09:44:53 -0700 * Fix reporter net_weird API usage for unknown_mobility_type (Jon Siwek, Corelight) * Remove variable content from weird names This changes many weird names to move non-static content from the weird name into the "addl" field to help ensure the total number of weird names is reasonably bounded. Note the net_weird and flow_weird events do not have an "addl" parameter, so information may no longer be available in those cases -- to make it available again we'd need to either (1) define new events that contain such a parameter, or (2) change net_weird/flow_weird event signature (which is a breaking change for user-code at the moment). Also, the generic handling of binpac exceptions for analyzers which to not otherwise catch and handle them has been changed from a Weird to a ProtocolViolation. Finally, a new "file_weird" event has been added for reporting weirdness found during file analysis. (Jon Siwek, Corelight) 2.6-197 | 2019-04-03 09:08:58 -0700 * Make Syslog analyzer accept non-conformant messages that omit Priority. (Jon Siwek, Corelight) 2.6-195 | 2019-03-27 12:36:34 -0700 * Reduce weird-stats overhead (Justin Azoff, Corelight) 2.6-193 | 2019-03-27 10:53:01 -0700 * Update now-broken Broker API usages (Jon Siwek, Corelight) Related to https://github.com/zeek/broker/pull/38, see Broker's NEWS file for C++ code migration hints. 2.6-192 | 2019-03-25 17:49:18 -0700 * Deprecate str_shell_escape, add safe_shell_quote replacement (Jon Siwek, Corelight) 2.6-191 | 2019-03-25 16:43:10 -0700 * Add support for SMB filenames to the intel framework (Stephen Hosom) 2.6-186 | 2019-03-25 09:41:57 -0700 * Added policy script for intel removal. (Jan Grashoefer) * Added Intel::filter_item hook to filter intelligence items. (Jan Grashoefer) 2.6-178 | 2019-03-21 14:10:44 -0700 * Add support for parsing SMB 3.1.1 NegotiateContextList response values (Mauro Palumbo) 2.6-175 | 2019-03-20 19:25:11 -0700 * Parse SMB2 TRANSFORM_HEADER messages and generate new smb2_transform_header event (Mauro Palumbo) 2.6-172 | 2019-03-20 17:59:30 -0700 * Fix smb_files.log missing FUID field in read/write actions (Mauro Palumbo) 2.6-169 | 2019-03-19 19:12:47 -0700 * Add support for NFLOG link-layer type (Ryan Denniston) 2.6-167 | 2019-03-18 13:58:28 -0700 * GH-307: Build binpac as a shared lib, not static by default (Jon Siwek, Corelight) 2.6-166 | 2019-03-18 11:45:35 -0700 * Add source file path control options for Input and Intel frameworks (Christian Kreibich, Corelight) This introduces the following redefinable string constants, empty by default: - InputAscii::path_prefix - InputBinary::path_prefix - Intel::path_prefix 2.6-164 | 2019-03-15 19:45:48 -0700 * Migrate table-based for-loops to key-value iteration (Jon Siwek, Corelight) * GH-154: Extend for-loops to allow iteration over a table's key-value pairs (Zeke Medley) 2.6-161 | 2019-03-15 12:59:31 -0700 * Fix SSH remote_location geo-data not being logged for successful authNs. (Michael Dopheide) 2.6-159 | 2019-03-14 16:39:52 -0700 * Move NEWS file back into main repo from zeek-docs (Jon Siwek, Corelight) 2.6-158 | 2019-03-14 16:23:30 -0700 * Fix signed/unsigned comparison compiler warning (Jon Siwek, Corelight) 2.6-157 | 2019-03-14 16:18:13 +0000 * GH-250: Add VXLAN decapsulation support (Henrik Lund Kramshoej; Jon Siwek, Corelight) Zeek now automatically decapsulates VXLAN traffic on UDP port 4789. It will log such sessions as Tunnel::VXLAN in tunnel.log and proceed to analyze the inner payload. Two options allow to tune the analysis: * "Tunnel::vxlan_ports" allows to tune the set of VXLAN ports to analyze/decapsulate. * "Tunnel::validate_vxlan_checksums" allows for tuning of how checksums associated with the outer UDP header of a possible VXLAN tunnel are handled. A new "vxlan_packet" event also provides per-packet access to VXLAN traffic. 2.6-154 | 2019-03-13 17:28:26 -0700 * Decrease memory usage via deferred list/dict initialization (Justin Azoff, Corelight) 2.6-152 | 2019-03-13 13:46:17 -0700 * Add field to the default http.log for the Origin header (Nate Guagenti) 2.6-149 | 2019-03-13 18:21:59 +0000 * GH-289: Add options to limit entries in http.log file fields. The "orig_fuids", "orig_filenames", "orig_mime_types" http.log fields as well as their "resp" counterparts are now limited to having "HTTP::max_files_orig" or "HTTP::max_files_resp" entries, which are 15 by default. The limit can also be ignored case-by-case via the "HTTP::max_files_policy" hook. (Jon Siwek, Corelight) * GH-282: Remove JSON formatter's range restriction on numbers. It now produces numbers as large as is required to match the data it needs to represent. (Jon Siwek, Corelight) * GH-281: Improve parsing of Google Pixel user agent. (Jon Siwek, Corelight) * GH-286: Check for record type mismatch in ternary operator. (Jon Siwek, Corelight) 2.6-141 | 2019-03-08 18:36:25 -0800 * Improve DNS query queuing logic (Jon Siwek, Corelight) 2.6-140 | 2019-03-08 16:21:42 -0800 * Improve performance of DNS policy scripts (Justin Azoff, Corelight) 2.6-135 | 2019-03-07 13:14:00 -0800 * Fix typos in dnp3-protocol.pac (g0nzu1) 2.6-132 | 2019-03-06 15:30:58 -0800 * GH-219: revert a breaking change to |x| operator for interval/time (Jon Siwek, Corelight) 2.6-130 | 2019-02-22 14:56:41 -0600 * Make input framework parse whitespace around various data types. (Johanna Amann, Corelight) 2.6-128 | 2019-02-22 14:32:48 -0600 * Add missing libkrb5 include dir to CMake config (Jon Siwek, Corelight) 2.6-127 | 2019-02-15 17:51:51 -0600 * Skip autogenerated doc coverage test for Travis pull requests (Jon Siwek, Corelight) * Add rstrip and lstrip BIFs (Zeke Medley) * Improve format of conn_state docs (Jon Siwek, Corelight) 2.6-117 | 2019-02-13 16:14:50 -0800 * Improve format of conn_state docs (Jon Siwek, Corelight) 2.6-116 | 2019-02-07 10:32:01 -0600 * GH-208: change invalid subnet expressions to a runtime error (Jon Siwek, Corelight) * GH-211: improve consistency of how scripting errors are handled (Jon Siwek, Corelight) Scripting errors/mistakes now consistently generate a runtime error which have the behavior of unwinding the call stack all the way out of the current event handler. This also changes the behavior of the startup/initialization process to abort if there's errors during bro_init() rather than continue on to the main run loop. 2.6-113 | 2019-02-06 13:17:39 -0600 * Add validity checking/warnings for Broker messages (Jon Siwek, Corelight) * Fix crash when using debug.log. (Johanna Amann, Corelight) 2.6-111 | 2019-01-29 18:17:35 -0600 * Fix memory leak due to enum type/val circular references (Jon Siwek, Corelight) 2.6-110 | 2019-01-29 14:49:10 -0800 * Add fuid to SSL:Invalid_Server_Cert notice (Stephen Hosom) 2.6-108 | 2019-01-28 14:11:19 -0600 * GH-210: improve call stack tracking w/ argument info (Jon Siwek, Corelight) 2.6-106 | 2019-01-24 17:53:03 -0600 * Fix building with LibreSSL again (Jon Siwek, Corelight) 2.6-105 | 2019-01-24 15:22:31 -0800 * GH-167: improve error message for unclosed function at EOF (Jon Siwek, Corelight) 2.6-103 | 2019-01-24 17:09:05 -0600 * Change digest.h functions to use EVP_MD_CTX interface (Johanna Amann) * Improve support for FIPS systems (Robert Clark) 2.6-98 | 2019-01-24 12:52:18 -0800 * Added ERSPAN III testing (Stu H) 2.6-95 | 2019-01-23 09:49:35 -0800 * GH-219: fix |x| operator int overflow / floating point type inconsistency (Jon Siwek, Corelight) 2.6-92 | 2019-01-22 08:53:36 -0800 * GH-151: fix hash calculation for nested sets Hash key construction of nested sets depended on the order in which their elements are iterated, which varied even between sets containing equivalent elements. The iteration order is now sorted by each element's hash value (or, on collision, by full key) such that equivalent sets no longer hash differently. (Jon Siwek, Corelight) 2.6-89 | 2019-01-18 15:17:34 -0800 * Pre-allocate and re-use Vals for bool, int, count, enum and empty string (Jon Siwek, Corelight) * Preallocate booleans and small counts < 4096 (Justin Azoff, Corelight) 2.6-86 | 2019-01-17 18:03:10 -0600 * Improve ERSPAN Type III support (Jon Siwek, Corelight) * Implement ERSPAN type II and ERSPAN type III support (Stu H) 2.6-82 | 2019-01-17 14:09:29 -0600 * Change doc/ subdir into a git submodule (Jon Siwek, Corelight) The docs now live at https://github.com/zeek/zeek-docs 2.6-81 | 2019-01-16 19:03:07 -0600 * Add Broker::peer_counts_as_iosource option (Jon Siwek, Corelight) 2.6-80 | 2019-01-16 11:14:47 -0600 * Patch to recognized the Revoked bit in DNSKEY Flag (Fatema BW) 2.6-77 | 2019-01-15 14:24:55 -0600 * GH-170: fix segfault triggered by invalid pattern symbols (Jon Siwek, Corelight) 2.6-76 | 2019-01-15 12:12:09 -0600 * GH-172: fix broxygen not merging bif and script identifier comments (Jon Siwek, Corelight) 2.6-75 | 2019-01-15 10:30:06 -0600 * GH-213: change type of vector for-loop index to a count (Jon Siwek, Corelight) 2.6-71 | 2019-01-14 16:11:58 -0600 * GH-205: prioritize use of sigaction() over sigset() (Jon Siwek, Corelight) 2.6-70 | 2019-01-14 15:34:18 -0600 * GH-188: fix crash when shutting down with pending reporter errors(Jon Siwek, Corelight) 2.6-69 | 2019-01-14 14:49:49 -0600 * Fix compiler warning in DNS analyzer (Jon Siwek, Corelight) 2.6-68 | 2019-01-14 14:18:46 -0600 * GH-162: fix segfault when &expire_func is missing a return value (Jon Siwek, Corelight) 2.6-67 | 2019-01-14 14:01:00 -0600 * GH-161: fix segfault in &default type checking for sets (Jon Siwek, Corelight) 2.6-66 | 2019-01-14 10:26:47 -0600 * Fix performance issue due to variable reuse in table expiration (Justin Azoff, Corelight) 2.6-62 | 2019-01-10 15:45:04 -0600 * Reorganize documentation index (Jon Siwek, Corelight) 2.6-61 | 2019-01-10 13:40:04 -0600 * Add RTD yaml config file (Jon Siwek, Corelight) * Remove some Bro usages in main TOC entries (Jon Siwek, Corelight) * Remove "contents" Sphinx directive usages (Jon Siwek, Corelight) * Add a `make livehtml` target (Jon Siwek, Corelight) * Use sourcecode Sphinx directive more widely (Jon Siwek, Corelight) * Use Sphinx RTD theme for user manual (Jon Siwek, Corelight) * Remove unused Sphinx extensions (Jon Siwek, Corelight) * Remove broxygen Sphinx integration (Jon Siwek, Corelight) * Remove Sphinx btest integrations and tests (Jon Siwek, Corelight) 2.6-46 | 2019-01-10 09:10:08 -0800 * improve performance of catch and release script (Justin Azoff, Corelight) 2.6-43 | 2019-01-07 09:50:43 -0800 * GH-227: Improve LibreSSL support (Jon Siwek, Corelight) 2.6-41 | 2019-01-04 17:50:00 -0600 * Replace some bro.org usages with zeek.org (Jon Siwek, Corelight) 2.6-39 | 2019-01-02 11:26:27 -0600 * Add BIF: Reporter::fatal_error_with_core (Stephen Hosom) 2.6-27 | 2018-12-10 11:53:41 -0600 * GH-216: Add FTS dependency when building on Alpine (Jon Siwek, Corelight) * Remove unnecessary header include (Jon Siwek, Corelight) * GH-216: Improve default DNS resolution support for Alpine/musl (Jon Siwek, Corelight) /etc/resolv.conf now gets parsed for the first IPv4 nameserver that works. * Add dns_resolver option (Jon Siwek, Corelight) 2.6-22 | 2018-12-10 11:16:53 -0600 * Introduce --enable-static-broker configuration option. (Johanna Amann) This option builds the bundled broker (and caf) statically. 2.6-20 | 2018-12-07 16:36:35 -0600 * Update github/download links (Jon Siwek, Corelight) 2.6-19 | 2018-12-07 07:26:51 -0600 * Fix Travis git clone command (Jon Siwek, Corelight) 2.6-18 | 2018-12-06 20:11:01 -0600 * Update external test suite locations (Jon Siwek, Corelight) * Update submodules to use github.com/zeek (Jon Siwek, Corelight) 2.6-16 | 2018-11-29 17:05:44 -0600 * Parallelize communication tests using btest TEST-PORT (Jon Siwek, Corelight) 2.6-14 | 2018-11-29 16:27:38 -0600 * Improve introspection of Record and TypeType values (Jon Siwek, Corelight) * TypeType values are now printable and yield the type name/alias * Fix record_fields BIF to return correct type name for fields * Allow TypeType values that point to a RecordType to be used with record_fields BIF * Bro plugins should support a patch version (x.y.z) (Jon Zeolla) * GH-148: add priority to DNSSEC event handlers (Jon Siwek, Corelight) * DNSSEC support (Fatema Bannat Wala) 2.6 | 2018-11-29 10:03:33 -0600 * Release 2.6. 2.6-beta3-2 | 2018-11-22 07:56:17 -0600 * GH-218: Add missing ICMP router advertisement counterpart (Jon Siwek, Corelight) 2.6-beta3 | 2018-11-14 17:09:42 -0600 * Release 2.6-beta3 2.6-beta2-83 | 2018-11-08 12:25:21 -0600 * Fix SumStats "last" plugin in cluster mode (Jon Siwek, Corelight) 2.6-beta2-82 | 2018-11-08 09:38:52 -0600 * Remove unnecessary Bloom filter empty check (Matthias Vallentin) 2.6-beta2-80 | 2018-11-07 11:46:34 -0600 * Support appending to vector of any (Jon Siwek, Corelight) 2.6-beta2-79 | 2018-11-07 10:27:00 -0600 * Fix coding conventions nits/typos (Vern Paxson, Corelight) 2.6-beta2-77 | 2018-11-06 09:32:17 -0600 * Switch GridFTP options from redef to option (Vlad Grigorescu) * Improve error handling in x509_ocsp_verify function (Jon Siwek, Corelight) 2.6-beta2-68 | 2018-11-02 18:30:01 -0500 * Fix a unit test relying on a bash-ism (Jon Siwek, Corelight) 2.6-beta2-67 | 2018-11-02 17:41:46 -0500 * Add script-layer call stack to internal errors messages that abort (Jon Siwek, Corelight) * Improve error message of index assignment expression failures (Jon Siwek, Corelight) 2.6-beta2-65 | 2018-11-02 09:36:30 -0500 * Improve Travis script to show multiple core dump stacks (Jon Siwek, Corelight) 2.6-beta2-64 | 2018-11-02 08:56:59 -0500 * Improve a weird stats unit test (Jon Siwek, Corelight) 2.6-beta2-62 | 2018-11-01 20:39:07 -0500 * Fix Travis script typo (Jon Siwek, Corelight) 2.6-beta2-61 | 2018-11-01 19:57:32 -0500 * Add more debug output to Travis script (Jon Siwek, Corelight) 2.6-beta2-60 | 2018-11-01 18:35:26 -0500 * Add core file search and stack trace output for Travis builds (Jon Siwek, Corelight) * Update license year for 2018 (Vlad Grigorescu) 2.6-beta2-57 | 2018-10-31 22:26:24 -0500 * GH-199: change `bro --help` exit status from 1 to 0 (Jon Siwek, Corelight) 2.6-beta2-55 | 2018-10-30 09:59:44 -0500 * Add a test with an encrypted MySQL connection (Vlad Grigorescu) * Fix parsing of MySQL NUL Strings (Vlad Grigorescu) 2.6-beta2-51 | 2018-10-26 10:41:42 -0500 * Add missing record field comment (Jon Siwek, Corelight) 2.6-beta2-50 | 2018-10-26 10:23:57 -0500 * Add missing record field comments (Jon Siwek, Corelight) 2.6-beta2-49 | 2018-10-25 18:56:02 -0500 * Fix minor documentation mistakes (Jon Siwek, Corelight) 2.6-beta2-46 | 2018-10-23 13:01:28 -0500 * GH-192: Generate ssh_auth_attempted for the 'none' authentication method. (Vlad Grigorescu) 2.6-beta2-43 | 2018-10-19 11:15:44 -0500 * Improve scripts/base/utils/dir unit test (Jon Siwek, Corelight) 2.6-beta2-42 | 2018-10-18 10:21:01 -0500 * Fix documentation link for notice_alarm.log fields (Jon Siwek, Corelight) 2.6-beta2-40 | 2018-10-16 15:37:49 -0500 * Change DNP3::function_codes name for request 0x21 (Dale Lakes) * Fix resource record type names in DNS::query_types for 41 and 100 (Dale Lakes) * Add missing DNS resource record types to DNS::query_types (Dale Lakes) * Refactor DCE_RPC constants to be specified in hex instead of decimal (Dale Lakes) 2.6-beta2-35 | 2018-10-16 13:41:15 -0500 * Update baselines for SSH capabilities fix (Vlad Grigorescu) * Fix SSH analyzer bug where is_server in capabilities is wrong. (Vlad Grigorescu) 2.6-beta2-32 | 2018-10-16 09:22:54 -0700 * Fix typo in Sessions.h (Eiji Yanagi (Cisco)) 2.6-beta2-31 | 2018-10-15 16:42:36 -0500 * Add DCE_RPC exchange_mapi operations to relevant consts.bro file (Dale Lakes) 2.6-beta2-29 | 2018-10-12 21:30:19 +0000 * GH-186: fix JSON formatting of timestamps before Unix epoch (Jon Siwek, Corelight) 2.6-beta2-28 | 2018-10-12 12:48:33 -0400 * Fix test baseline for plugin skeleton update (Jon Siwek, Corelight) 2.6-beta2-27 | 2018-10-12 12:18:02 -0400 * Convert site::local_nets, etc. into options. (Johanna Amann) 2.6-beta2-25 | 2018-10-12 08:33:32 -0400 * Fix crash when modifying a table from within its &expire_func (Jon Siwek, Corelight) 2.6-beta2-24 | 2018-10-05 14:24:34 -0500 * GH-184: add `bro-config --build_type`, outputs CMake build type (Jon Siwek, Corelight) 2.6-beta2-22 | 2018-10-04 11:31:48 -0500 * Add return value checks for some RPC parsing functions (Jon Siwek, Corelight) * Add 'fallthrough' comment to a switch/case block (Jon Siwek, Corelight) 2.6-beta2-20 | 2018-10-03 15:47:26 -0500 * Improve broker.remote_id unit test (Jon Siwek, Corelight) * Increase broker unit test timeout intervals (Jon Siwek, Corelight) 2.6-beta2-18 | 2018-10-03 11:09:04 -0500 * Fix memory leak in broker type checking (Jon Siwek, Corelight) 2.6-beta2-17 | 2018-10-02 16:05:10 -0500 * Update testing/btest/README (Jon Siwek, Corelight) 2.6-beta2-14 | 2018-09-25 16:38:29 -0500 * Add some missing @TEST-REQUIRES to a few tests (Daniel Thayer) 2.6-beta2-12 | 2018-09-24 10:56:09 -0500 * Fix BasicThread::SetOSName on FreeBSD (Dominik Charousset) 2.6-beta2-10 | 2018-09-21 13:29:15 -0500 * Fix some broken @TEST-REQUIRES in unit tests (Daniel Thayer) 2.6-beta2-8 | 2018-09-21 13:25:50 -0500 * Emit missing GeoIP database errors only once at startup (Jon Siwek, Corelight) 2.6-beta2-7 | 2018-09-21 10:18:55 -0500 * Fix compile error in MMDB GeoIP code (Jon Siwek, Corelight) 2.6-beta2-6 | 2018-09-20 13:15:15 -0500 * Add a missing "break" in OSFinger.cc (Daniel Thayer) * Fix buffer sizes in the rotate_file function (Daniel Thayer) 2.6-beta2-3 | 2018-09-19 15:21:00 -0500 * Add HTTP::sqli_policy hook to ignore counting a request as a SQL injection (Justin Azoff) 2.6-beta2 | 2018-09-18 16:52:34 -0500 * Release 2.6-beta2 2.6-beta | 2018-09-18 15:05:24 -0500 * Release 2.6-beta 2.5-1001 | 2018-09-12 19:47:57 -0500 * Fix IRC names command parsing (Jon Siwek, Corelight) 2.5-996 | 2018-09-11 13:04:20 -0500 * Fix raw input reader not removing streams for dead processes. (Seth Hall, Corelight) 2.5-994 | 2018-09-10 19:47:03 -0500 * Try to fix a rare broker test instability (Jon Siwek, Corelight) * Fix invalid memory free when using Log::default_field_name_map (Jon Siwek, Corelight) 2.5-992 | 2018-09-10 18:34:09 -0500 * Stabilize a unit test. (Jon Siwek, Corelight) * Fix potential memory leak in Kerberos scripts (reported by Maksim Shudrak and fixed by Jon Siwek, Corelight) 2.5-990 | 2018-09-10 14:55:13 -0500 * Fix recursive type checks/casts of broker data into type 'any' (Jon Siwek, Corelight) * Fix is/as operators on vector values (Jon Siwek, Corelight) 2.5-988 | 2018-09-07 17:49:34 -0500 * Update default Broker/CAF thread tuning (Jon Siwek, Corelight) 2.5-987 | 2018-09-07 11:24:34 -0500 * Update NEWS explaining Bro runs as 1 process instead of 2 (Daniel Thayer) * Update NEWS for changes to broctl "top" command output (Daniel Thayer) 2.5-984 | 2018-09-07 09:57:52 -0500 * Give Cluster::rr_topic "key" argument a default value (Jon Siwek, Corelight) 2.5-983 | 2018-09-06 18:26:20 -0500 * Disable broker message forwarding by default (Jon Siwek, Corelight) 2.5-982 | 2018-09-06 08:58:09 -0500 * Documentation updates (Daniel Thayer) * Fix a typo and indentation in the configure script (Daniel Thayer) * Add krb5 devel package to Travis docker containers (Daniel Thayer) 2.5-975 | 2018-09-05 16:52:32 -0500 * Allow weird sampling settings to be updateable at runtime (Johanna Amann, Corelight) * Permit weird sampling rate of 0, which suppresses all weirds (Johanna Amann, Corelight) * Switch packet stats to uint64. (Robin Sommer, Corelight) 2.5-969 | 2018-09-05 15:11:48 -0500 * BIT-1208: remove unused weirds from Weird::actions table (Jon Siwek, Corelight) * BIT-1779: use BRO_LOG_SUFFIX env var in ascii log rotation function (Jon Siwek, Corelight) 2.5-967 | 2018-09-05 19:30:48 +0000 * Fix printf format specification for reporting packet stats. (Robin Sommer, Corelight) 2.5-965 | 2018-09-04 17:17:36 -0500 * Updates to NTLM script handling. (Seth Hall, Corelight) - This separates NTLM handling away from SMB. - It logs more accurately when logins are succeed or fail or even if the resulting status of an authentication is unknown. - Adds some new fields where the server is indicating information about itself (server_nb_computer_name, server_dns_computer_name, and server_tree_name) 2.5-962 | 2018-09-04 12:11:14 -0500 * Improve update-changes output (Jon Siwek, Corelight) 2.5-961 | 2018-09-04 12:07:54 -0500 * Sort output of a coverage unit test (Jon Siwek, Corelight) * Remove non-ascii char from rdp/consts.bro (Jon Siwek, Corelight) 2.5-957 | 2018-09-04 09:28:47 -0500 * Fix/improve the find-bro-logs.test (Daniel Thayer) * Fix typos/formatting in NEWS (Daniel Thayer) * Clarify 'old_comm_usage_is_ok' error message (Jon Siwek, Corelight) * Update Mozilla CA list to NSS 3.39 (Johanna Amann, Corelight) 2.5-952 | 2018-08-31 17:30:21 -0500 * Update NEWS (finalizations/formatting) (Jon Siwek, Corelight) 2.5-951 | 2018-08-31 15:33:31 -0500 * Improve `make dist` (Jon Siwek, Corelight) 2.5-950 | 2018-08-31 08:54:36 -0700 * Add @deprecated directive and deprecate policy/protocols/smb/__load__.bro @deprecated a warning stating that the script is deprecated. (Jon Siwek, Corelight) 2.5-947 | 2018-08-30 16:05:36 -0500 * Allow loading policy/protocols/smb once again (Jon Siwek, Corelight) 2.5-946 | 2018-08-30 09:51:16 -0500 * Update NEWS with more info about runtime options (Daniel Thayer) 2.5-944 | 2018-08-30 09:28:41 -0500 * Introduce ssl_plaintext_data event, replacing ssl_application_data event. (Johanna Amann) * Add record layer version to event ssl_encrypted_data. (Johanna Amann) * Add compression methods to ssl_client_hello event. (Johanna Amann) 2.5-932 | 2018-08-30 00:08:58 +0000 * Add Broker::forward() function. This enables explicit forwarding of events matching a given topic prefix. Even if a receiving node has an event handler, it will not be raised if the event was sent along a topic that matches a previous call to Broker::forward(). (Jon Siwek, Corelight) * Enable implicit Broker message forwarding by default. (Jon Siwek, Corelight) * Remove Cluster::broadcast_topic. As enabling Broker forwarding would cause routing loops with messages sent to such a topic (one subscribed to on all nodes). (Jon Siwek, Corelight) * Remove Intel Broker topics, re-use existing Cluster topics. (Jon Siwek, Corelight) * Update broker docs to reflect best-practice/convention for declaring new topics. * Remove "relay" family of Broker functions. (Jon Siwek, Corelight) Namely these are now removed: - Broker::relay - Broker::publish_and_relay - Cluster::relay_rr - Cluster::relay_hrw The idea being that Broker may eventually implement the necessary routing (plus load balancing) functionality. For now, code that used these should "manually" handle and re-publish events as needed. 2.5-924 | 2018-08-29 18:21:37 -0500 * Allow event/function headers to be wrapped in directives. (Johanna Amann) For example: @if ( conditions ) event a(...) @else event b(...) @endif { ... } 2.5-922 | 2018-08-29 17:22:20 -0500 * Fix unit tests (Jon Siwek, Corelight) * Fix strict-aliasing compiler warning (Jon Siwek, Corelight) 2.5-919 | 2018-08-29 14:58:06 -0500 * Update unit test baseline for new BinPAC output (Jon Siwek, Corelight) * CT List update - a few more logs. (Johanna Amann) * Update certificate list to NSS 3.38 (Johanna Amann) 2.5-915 | 2018-08-28 14:22:25 -0700 * Improve input framework re-read logic Changed from checking for "has newer modification time" to "has different modification time or inode number". (Jon Siwek, Corelight) * Convert more redef-able constants to runtime options (Daniel Thayer) 2.5-911 | 2018-08-24 17:47:03 -0700 * Add a missing initializer to a runtime option (Daniel Thayer) * Convert more redef-able constants to runtime options (Daniel Thayer) 2.5-907 | 2018-08-24 17:23:46 -0700 * Fix base/misc/version.bro version parsing. (Johanna Amann) 2.5-906 | 2018-08-24 14:57:55 -0500 * Stabilize a cluster logging unit test (Jon Siwek, Corelight) 2.5-905 | 2018-08-24 10:21:35 -0500 * Detect MaxMind DB changes and auto-reload (Jonathan Perkins, Corelight) 2.5-903 | 2018-08-23 16:54:24 -0500 * Fix finding of kerberos and libmaxminddb CMake < 3.3 (Daniel Thayer) * BIT-1885: fix "kill" threading message (Jon Siwek, Corelight) 2.5-900 | 2018-08-23 15:18:48 -0500 * Improve readability of the Travis job log (Daniel Thayer) * Fix tracking of DCE-RPC context identifier mappings This adds previously-missing support for "Alter Context" request/response PDUs (initial patch contributed by Mark Fernandez). Also, context ID arguments were added to dce_rpc_bind, dce_rpc_request, and dce_rpc_response in order to properly track what endpoint/operation a given opnum maps to. (Jon Siwek, Corelight) 2.5-897 | 2018-08-23 15:53:16 +0000 * BIT-1885: Fix input framework memory leak. For input threads that get joined during run-time, messages could remain in the thread's queue and leak. (Jon Siwek, Corelight) * Increase timeout for a memleak test. (Jon Siwek, Corelight) 2.5-894 | 2018-08-22 12:05:19 -0500 * Ensure external test repo hashes track origin/master (Jon Siwek, Corelight) 2.5-892 | 2018-08-22 11:49:12 -0500 * Fix "unused CMake variable" configuration warnings (Jon Siwek, Corelight) 2.5-890 | 2018-08-21 16:47:52 -0500 * Fix Travis CI script to checkout particular commits of external tests (Jon Siwek, Corelight) * Fix signed/unsigned comparison warning (Jon Siwek, Corelight) 2.5-888 | 2018-08-21 15:54:56 -0500 * Add --with-broker configure option (Jon Siwek, Corelight) 2.5-887 | 2018-08-21 14:54:12 -0500 * Change default snaplen to 9216 bytes to better accommodate jumbo frames (Justin Azoff) 2.5-884 | 2018-08-20 15:39:21 -0500 * Fix outdated documentation test baselines (Jon Siwek, Corelight) * Add 'smtp_excessive_pending_cmds' weird (Jon Siwek, Corelight) * Fix SMTP command string comparisons (Jon Siwek, Corelight) * Improve handling of empty lines in several text protocol analyzers (Jon Siwek, Corelight) * Add rate-limiting sampling mechanism for weird events The generation of weird events, by default, are now rate-limited according to these tunable options: - Weird::sampling_whitelist - Weird::sampling_threshold - Weird::sampling_rate - Weird::sampling_duration The new get_reporter_stats() BIF also allows one to query the total number of weirds generated (pre-sampling) which the new policy/misc/weird-stats.bro script uses periodically to populate a weird_stats.log. There's also new reporter BIFs to allow generating weirds from the script-layer such that they go through the same, internal rate-limiting/sampling mechanisms: - Reporter::conn_weird - Reporter::flow_weird - Reporter::net_weird Some of the code was adapted from previous work by Johanna Amann. (Jon Siwek, Corelight) * Teach timestamp canonifier about timestamps before ~2001 (Jon Siwek, Corelight) 2.5-877 | 2018-08-20 14:58:58 -0500 * Remove the node-specific local-*.bro scripts (Daniel Thayer) 2.5-875 | 2018-08-20 12:45:32 -0500 * Improve diff-remove-abspath canonifier: collapse '/' sequences (Jon Siwek, Corelight) * Remove unused redef-able constants (Daniel Thayer) * Convert some redef-able constants to runtime options (Daniel Thayer) 2.5-870 | 2018-08-17 17:07:57 -0500 * Documentation improvements (Daniel Thayer) 2.5-855 | 2018-08-17 16:34:51 -0500 * Add script to support the old DHCP events (Vlad Grigorescu) 2.5-852 | 2018-08-17 15:15:55 -0500 * BIT-466: add redef += support to vectors (Jon Siwek, Corelight) 2.5-850 | 2018-08-17 11:12:53 -0500 * BIT-1815: move SMB::write_cmd_log functionality into policy/ script The option is removed, but same functionality is now enabled simply by loading policy/protocols/smb/log-cmds.bro (Jon Siwek, Corelight) 2.5-849 | 2018-08-17 10:29:58 -0500 * Fix possible race in netcontrol acld/broker plugins (Jon Siwek, Corelight) 2.5-848 | 2018-08-16 17:21:28 -0500 * Enable SMB by default by moving scripts from policy/ to base/ (Jon Siwek, Corelight) 2.5-847 | 2018-08-16 16:07:14 -0500 * BIT-1924: add DHCP port to software.log for completeness (Jon Siwek, Corelight) 2.5-846 | 2018-08-16 14:11:02 -0500 * BIT-1858: fix logged-names for DNS RR types 44 and 45 (Jon Siwek, Corelight) * BIT-1850: add missing DCE/RPC PDU type enum values (Jon Siwek, Corelight) 2.5-844 | 2018-08-16 12:13:16 -0500 * Add env. variables to override Broker listen/connect retry intervals And use them to default retries to 1sec for all unit tests. (Jon Siwek, Corelight) 2.5-843 | 2018-08-15 18:01:56 -0500 * BIT-1544: allow NULs in file analysis handles (Jon Siwek, Corelight) 2.5-842 | 2018-08-15 11:00:20 -0500 * Fix seg fault on trying to type-cast invalid/nil Broker::Data (Jon Siwek, Corelight) 2.5-841 | 2018-08-14 16:45:09 -0500 * BIT-1798: fix PPTP GRE tunnel decapsulation (Jon Siwek, Corelight) 2.5-840 | 2018-08-13 17:40:06 -0500 * Fix SumStats::observe key normalization logic (reported by Jim Mellander and fixed by Jon Siwek, Corelight) 2.5-839 | 2018-08-13 10:51:43 -0500 * Make options redef-able by default. (Johanna Amann, Corelight) * Fix incorrect input framework warnings when parsing ports. (Johanna Amann, Corelight) * Allow input framework to accept 0 and 1 as valid boolean values. (Johanna Amann, Corelight) * Improve the travis-job script to work outside of Travis (Daniel Thayer) * Fix validate-certs.bro comments (Jon Siwek, Corelight) 2.5-831 | 2018-08-10 17:12:53 -0500 * Immediately apply broker subscriptions made during bro_init() (Jon Siwek, Corelight) * Update default broker threading configuration to use 4 threads and allow tuning via BRO_BROKER_MAX_THREADS env. variable (Jon Siwek, Corelight) * Misc. unit test improvements (Jon Siwek, Corelight) 2.5-826 | 2018-08-08 13:09:27 -0700 * Add support for code coverage statistics for bro source files after running btest test suite This adds --enable-coverage flag to configure Bro with gcov. A new directory named /testing/coverage/ contains a new coverage target. By default a coverage.log is created; running make html in testing/coverage creates a HTML report. (Chung Min Kim, Corelight) 2.5-819 | 2018-08-08 13:03:22 -0500 * Fix cluster layout graphic and doc warnings (Jon Siwek, Corelight) * Added missing tcp-state for signature dpd_rfb_server (Zhongjie Wang) 2.5-815 | 2018-08-06 17:07:56 -0500 * Fix an "uninitialized" compiler warning (Jon Siwek, Corelight) * Fix (non)suppression of proxy-bound events in known-*.bro scripts (Jon Siwek, Corelight) 2.5-811 | 2018-08-03 11:33:57 -0500 * Update scripts to use vector "+=" append operation (Vern Paxson, Corelight) * Add vector "+=" append operation (Vern Paxson, Corelight) * Improve a travis output message in pull request builds (Daniel Thayer) * Use default version of OpenSSL on all travis docker containers (Daniel Thayer) 2.5-802 | 2018-08-02 10:40:36 -0500 * Add set operations: union, intersection, difference, comparison (Vern Paxson, Corelight) 2.5-796 | 2018-08-01 16:31:25 -0500 * Add 'W' connection history indicator for zero windows (Vern Paxson, Corelight) * Allow logarithmic 'T'/'C'/'W' connection history repetitions, which also now raise their own events (Vern Paxson, Corelight) 2.5-792 | 2018-08-01 12:15:31 -0500 * fix NTLM NegotiateFlags field offsets (Jeffrey Bencteux) 2.5-790 | 2018-08-01 11:25:27 -0500 * Fix --with-binpac configure option (Jon Siwek, Corelight) * Update CAF-finding logic (Jon Siwek, Corelight) 2.5-787 | 2018-07-31 16:50:55 -0500 * Add Cisco FabricPath support (Damani Wade, Corelight) * Replace GeoIP Legacy DB support with MaxMind DB support (Jonathan Perkins, Corelight) 2.5-782 | 2018-07-31 11:53:22 +0200 * Update install instructions for OpenSSL 1.1 compat (Jon Siwek, Corelight) * Remove requestorName parameter of ocsp_request event This field isn't publicly available via the OpenSSL 1.1 API, not used in the base scripts, and has no example in the test suit, so removing it is simpler than trying to support manually parsing it out of the raw data. (Jon Siwek, Corelight) * Adjust x509 unit tests to work around OpenSSL 1.0 vs. 1.1 differences (Jon Siwek, Corelight) * Fixes for OpenSSL 1.1 support (Jon Siwek, Corelight) 2.5-775 | 2018-07-24 16:39:34 -0500 * Add broker/binpac/caf dirs to bro-config script (Jon Siwek, Corelight) * Exclude CMakeFiles from header installation path (Jon Siwek, Corelight) 2.5-773 | 2018-07-24 15:04:41 +0000 * BIT-1950: Support PPPoE over QinQ (Jon Siwek, Corelight) 2.5-771 | 2018-07-24 02:26:17 +0000 * Support building plugins from Bro installation prefix so that it does no longer need access to a Bro source/build tree. This required installing various Bro headers, BinPAC and it's headers, bifcl, and Bro's custom CMake modules. (Jon Siwek, Corelight) * Add binpac to install process. (Jon Siwek, Corelight) * Move bifcl to a separate repo. (Jon Siwek, Corelight) 2.5-766 | 2018-07-24 01:39:07 +0000 * Clusterization of configureation framework. (Johanna Amann, Corelight) * Fix special-case-bug for vectors in UnaryExpr. (Johanna Amann, Corelight) * Teach Option::set to unwrap Broker::Data values (Jon Siwek, Corelight) * Fix some compiler warnings. (Robin Sommer, Corelight) 2.5-749 | 2018-07-20 12:08:06 -0500 * Make Broker congestion queue size tunable and increase default (Jon Siwek, Corelight) * Improve control framework id-update/test output (Jon Siwek, Corelight) 2.5-747 | 2018-07-18 09:51:13 -0500 * Improve some netcontrol unit tests (Jon Siwek, Corelight) 2.5-746 | 2018-07-17 17:51:13 -0500 * Improve an input framework unit test (Jon Siwek, Corelight) 2.5-745 | 2018-07-17 16:46:16 -0500 * Add explicit key in Travis known_hosts (Jon Siwek, Corelight) 2.5-743 | 2018-07-17 14:20:19 -0500 * Port broker::data variant usages to use CAF API directly (Jon Siwek, Corelight) 2.5-741 | 2018-07-16 16:06:02 -0500 * Improve Specific_RE_Matcher::CompileSet() error condition cleanup (Jon Siwek, Corelight) 2.5-740 | 2018-07-16 16:01:31 -0500 * Add support for case-insensitive patterns (Vern Paxson, Corelight) 2.5-730 | 2018-07-16 10:39:33 -0500 * de-restrict pattern-oriented BiFs to no longer require only running at init (Vern Paxson) * Add option to toggle extraction of subject alternate names from X509 SAN DNS field (Liviu Valsan) 2.5-725 | 2018-07-03 14:56:10 -0500 * BIT-1941: improve unit test stability (Corelight) 2.5-723 | 2018-07-03 09:34:10 -0500 * Fix unstable config framework test (Corelight) 2.5-722 | 2018-07-03 09:16:37 -0500 * BIT-1941: teach diff-remove-timestamps about time 0 (Corelight) 2.5-721 | 2018-07-02 16:29:21 -0500 * BIT-1941: improve reliability of broker.disconnect unit test (Corelight) 2.5-719 | 2018-06-27 20:02:52 -0500 * Fix some typos and formatting in NEWS and other documentation (Daniel Thayer) * Add documentation for type-based switch statment, as/is operators, bitwise operators, and pattern operators (Daniel Thayer) 2.5-711 | 2018-06-27 19:11:58 -0500 * Prevent double-wrapping Broker::Data in published event args (Corelight) 2.5-710 | 2018-06-26 18:06:22 -0500 * Add memory leak unit test for pattern operations (Corelight) * fixed 3 leaks in creating pattern values (Vern Paxson) * add & and | operators for patterns (Vern Paxson) * deprecate merge_patterns() (Vern Paxson) * deprecate boolean scalar+vector operations (Vern Paxson) * deprecate mixing scalars and vectors (Vern Paxson) * deprecate && / || operators for patterns (Vern Paxson) 2.5-690 | 2018-06-26 15:05:23 -0500 * Fix deprecated actor_system_config field usages (Corelight) 2.5-689 | 2018-06-26 11:45:52 -0500 * Remove header self-inclusions (Corelight) * Fix travis-job script to not fail when all tests succeed (Daniel Thayer) 2.5-687 | 2018-06-25 16:35:25 -0500 * Reorganize internal + private broker/Manager.h bits (Corelight) * Reduce proliferation of including broker header files (Corelight) 2.5-684 | 2018-06-25 11:26:55 -0500 * Use docker containers to run Bro tests on Travis CI (Daniel Thayer) * Travis CI fewer failures and improved output messages (Daniel Thayer) 2.5-681 | 2018-06-22 20:17:06 -0500 * Fix null pointer deref in AST traversal (Corelight) * Fix for ancient reference-counting bug in NFA.cc (Vern Paxson) 2.5-679 | 2018-06-21 16:00:48 -0500 * Add support for bitwise operations (&, |, ^, ~) on "count" values. (Vern Paxson) 2.5-671 | 2018-06-21 11:55:39 -0500 * Add ability for BroControl to skip cluster setup (Corelight) * BIT-1938: fix crash in Broker manager shutdown (Corelight) * Disable broxygen when running unit tests (Daniel Thayer) 2.5-668 | 2018-06-15 17:14:33 -0500 * Make old comm. system usages an error unless old_comm_usage_is_ok is set (Corelight) 2.5-667 | 2018-06-15 15:30:11 -0500 * Add --disable-broker-tests configure option (Corelight) 2.5-663 | 2018-06-14 12:51:28 -0500 * Add Broker::max_threads and Broker::max_sleep tuning options, remove Broker::max_live_threads and Broker::max_pcap threads (Corelight) * Minor optimization to bro_broker::Manager::FlushPendingQueries (Corelight) 2.5-660 | 2018-06-12 13:49:39 -0500 * Add Broker::max_live_threads and Broker::max_pcap_threads tunables (Corelight) 2.5-658 | 2018-06-08 16:41:07 +0000 * Allow BRO_DEFAULT_LISTEN_ADDRESS to control broker listen address. This environment variable is now set to listen only on IPv4 loopback when running unit tests (instead of using the default INADDR_ANY). (Corelight) * Move some of the @loads out from init-bare.bro into a new init-frameworks-and-bifs.bro in order to better support calling BIFs (like `getenv`) from variable initializations in those particular frameworks. (Corelight) 2.5-655 | 2018-06-08 10:43:03 -0500 * Correct conn history field documentation (Corelight) 2.5-652 | 2018-06-07 13:57:23 -0500 * GH-131: disable krb ticket decryption on non-Linux (Corelight) 2.5-651 | 2018-06-07 09:57:29 -0500 * Fix signed/unsigned comparison compiler warning (Corelight) 2.5-650 | 2018-06-06 16:20:18 -0500 * Improve Broker performance (Corelight) 2.5-648 | 2018-06-05 17:32:47 -0500 * BIT-1936: improve Broxygen warnings (Corelight) 2.5-647 | 2018-06-05 15:19:16 -0500 * Update `make doc`: don't copy broker docs (Corelight) 2.5-646 | 2018-06-05 11:31:43 -0500 * Add NCP::max_frame_size tuning option (Corelight) * Migrate NCP analyzer to use latest analyzer API (Corelight) * Fix read at invalid address in X509 extension parser (Johanna Amann) 2.5-642 | 2018-06-04 13:52:46 -0500 * Make 0 be a valid packet source timestamp (Corelight) 2.5-641 | 2018-06-04 09:18:59 -0700 * Add Broker::publish_and_relay BIF Like Broker::relay, except the relaying-node also calls event handlers. (Corelight) * Document variable argument list BIFs using ellipsis. (Corelight). * Support unserializing broker data into type 'any' The receiver side will wrap the data as a Broker::Data value, which can then be type-checked/cast via 'is' or 'as' operators to a specific Bro type. For example: Sender: Broker::publish("topic", my_event, "hello") Receiver: event my_event(arg: any) { if ( arg is string ) print arg as string; } (Corelight) * Fix a bug in broker data type-casting check (Corelight) * Remove dead code in broker data/val conversion function (Corelight) * SSH protocol now assesses the packet length at an earlier stage within binpac (Andrew Woodford). * Remove some UTF-8 characters that snuck into a few scripts. (Corelight) * Decrypt the krb ticket and extract authentication data. (Julien Wallior) 2.5-619 | 2018-06-01 11:29:15 -0500 * Relocate temporary script coverage files (Corelight) 2.5-618 | 2018-06-01 10:03:24 -0500 * BIT-1635: fix `make doc` warnings (Corelight) * Add smb2_file_sattr event (Devin Trejo) * Add bad ARP tests (Pierre LATET) * Fix SCT validation when invalid certificates are in chain. (Johanna Amann) 2.5-611 | 2018-05-29 10:13:17 -0500 * Fix NEWS file formatting (Corelight) * Improve Broker docs with reminder about modules and event namespace scoping interactions. (Michael Dopheide) * Change Intel framework to round-robin insertion events across proxies (Corelight) * Add a counter for number of alive nodes within a given cluster pool (Corelight) * Fix how cluster framework tracks worker count (Corelight) 2.5-599 | 2018-05-23 16:50:12 -0500 * Documentation improvements/fixes (Corelight) 2.5-598 | 2018-05-22 15:05:24 -0500 * Fixes for MySQL and SMB protocol parsers (Corelight) * MySQL: the parser for this was generally broken (not following the specification well) and needed many changes. One addition is a new "mysql_result_row" event that provides access to the results of queries. * SMB: the spec seems to explitly call out the omission of the PrimaryDomain field on SMB_COM_SESSION_SETUP_ANDX responses (and I don't see that field in pcaps either), so this may have just been a typo that used to work fine in the past only due to faulty array parsing behavior in binpac. * BIT-1829: add unit test for modbus parser issue (Corelight) 2.5-591 | 2018-05-22 09:19:59 -0500 * Make Reassembler::TotalSize a constant time operation (Corelight) 2.5-589 | 2018-05-21 21:37:54 +0000 * Switch Bro's communication over to Broker; deprecate the old communication system, including Broccoli. See NEWS for more. (Many people contributed to this effort. Broker library: Jon Siwek, Matthias Vallentin, Robin Sommer, Dominik Charousset. Porting Bro to Broker: Daniel Thayer, Robin Sommer, Jon Siwek. Further contributions by: Johanna Amann, Justin Azoff, Matthias Fischer, Jan Grashoefer, and Seth Hall. The final integration was supported by Corelight.) * Extend switch statement to branch by type of the operand. See NEWS for more. (Robin Sommer) * Add new operators "is" and "as" for dynamic type casting and type checking. See NEWS for more. (Robin Sommer) 2.5-582 | 2018-05-21 13:34:16 -0500 * Update link to flex pattern docs (Corelight) * Add non-standard experimental Google post-quantum ciphers (Johanna Amann) * ARP: fix the l2 source address check for ARP over Wi-Fi (Pierre LALET) * Support 802.11 monitor mode (Pierre LALET) 2.5-569 | 2018-05-10 11:24:07 -0500 * BIT-1927: relocate notice/extend-email/ scripts to policy/ dir and load it from local.bro to allow users to control whether it is used. (Stephen Hosom) * Sort output of the missing-file-initially.bro test (Daniel Thayer) 2.5-565 | 2018-05-08 15:29:53 -0500 * BIT-1926: add unit tests for misc. HTTP patches (Corelight) * Fix case insensitive HTTP/MIME header name comparisons (Jeffrey Bencteux) * Don't use chunked mode Transfer-Encoding with HTTP/1.0 (Jeffrey Bencteux) * Fix handling of HTTP body length when Content-Range length differs from Content-Length. (Jeffrey Bencteux) * Decode 'x-gzip' HTTP Content-Encoding the same as 'gzip' (Jeffrey Bencteux) 2.5-559 | 2018-05-08 11:23:28 -0700 * Add test for dump_current_packet bif. (Johanna Amann) * Fix dump_packet & dump_current_packet to work with several filenames. (Assaf Morami) 2.5-553 | 2018-05-03 14:59:53 -0500 * Make BinPAC exception handling more consistent (Vlad Grigorescu) 2.5-551 | 2018-05-01 18:27:38 -0500 * Fix the ip-broken-header.bro test on macOS due to missing 'xzcat' (Daniel Thayer) * Improve reliability of the logging rotate.bro test (Daniel Thayer) * Improve reliability of missing-file-initially.bro test (Daniel thayer) 2.5-547 | 2018-05-01 18:17:14 -0500 * Update install instructions for Ubuntu 18.04 (Daniel Thayer) 2.5-545 | 2018-05-01 18:09:30 -0500 * Improve canonicalization of build dir path in a coverage unit test (Corelight) 2.5-544 | 2018-05-01 17:57:15 -0500 * Rewrite the DHCP analyzer and accompanying script-layer API. (Valerio G, Corelight) * Reduced all DHCP events into a single dhcp_message event. (removed legacy events since they weren't widely used anyway) - Support many more DHCP options. - DHCP log is completely reworked and now represents DHCP sessions based on the transaction ID (and works on clusters). - Removed the known-devices-and-hostnames.bro and known-devices.bro scripts since it's generally less relevant now with the updated log. * Change include directory search order to better support --with-openssl. (Johanna Amann) 2.5-535 | 2018-04-30 16:22:30 -0500 * Improve how coverage unit tests handle name of build dir (Corelight) 2.5-534 | 2018-04-27 19:59:46 -0400 * Fix subnet expiration in the intel framework. (Seth Hall) * BIT-1909: fix invalid redef'd record field accesses (Jon Siwek) 2.5-527 | 2018-04-27 11:01:03 -0500 * BIT-1430: Improve cross compilation support (Corelight) * Add --toolchain= configure option * Add --with-bifcl= configure option * Change --with-binpac= configure option to mean "path to binpac executable" 2.5-526 | 2018-04-25 11:06:50 -0500 * BIT-1914: comment out &check usages now that they emit warnings and convert some to &enforce (Corelight) * Removed the "start" parameter of the dnp3_header_block event since it's always the same value. (Corelight) 2.5-522 | 2018-04-25 10:48:38 -0500 * Improve dce-rpc/consts.bro operations table organization (Luciano Mammino) 2.5-519 | 2018-04-20 07:46:07 -0700 * Reduce number of btest threads running tests on Travis CI. (Daniel Thayer) 2.5-515 | 2018-04-18 11:44:36 -0500 * Improve std::map usages in SMB code. (Corelight) 2.5-514 | 2018-04-18 10:54:24 -0500 * Improve HLL cardinality estimate unit test. (Corelight) 2.5-513 | 2018-04-18 10:38:41 -0500 * Updating the defined SMB2 dialects to match Microsofts current docs. (Corelight) * BIT-1862: Improve handling SMB pending commands and read response tree id. (Stefano Rinaldi, Corelight) On rare occasions, SMB server doesn't return the tree id on read responses. * Better reporter for Brostring with embedded NUL (Philippe Antoine) * Fix config input reader on systems with gcc 4.8 (Daniel Thayer) 2.5-504 | 2018-04-06 10:51:19 -0700 * Trim the Travis CI build log output. (Daniel Thayer) * Remove unneeded lines from .travis.yml. (Daniel Thayer) 2.5-501 | 2018-04-06 10:49:54 -0700 * Fix NETBIOSSSN analyzer name (Vladimir Ruzanov) * Additional fix for Kerberos in GSSAPI. (Seth Hall) 2.5-498 | 2018-04-03 01:59:46 -0400 * Improvements to GSSAPI handling of Kerberos messages (John E. Rollinson, Seth Hall, juno0812, Justin Oursler) * Improve SMB2 Create command events and add newly parsed data. (Julien Wallior) 2.5-483 | 2018-03-29 14:10:48 -0700 * Source code clean up (Johanna Amann) - Mark one-parameter constructors as 'explicit' & use 'override' where possible - Remove unimplemented & unused functions from header files. - Make some data flows more explicit for compilers. 2.5-478 | 2018-03-29 12:59:49 -0700 * Recognize TLS 1.3 negotiation correctly. The way in which TLS 1.3 is negotiated was changed slightly in later revisions of the standard. (Johanna Amann) * Fix the travis-job script to always run external tests. (Daniel Thayer) * Fix information leak in the update-traces script. (Daniel Thayer) * Add Coverity scan and private testing to Travis CI. (Daniel Thayer) 2.5-471 | 2018-03-21 13:56:57 -0700 * Fix a memory leak in SMBv1 share mapping. (Corelight) * Fix one new minor typo in the config framework docs. (Daniel Thayer) 2.5-467 | 2018-03-15 14:58:40 -0700 * Configure Travis CI email recipients and build branches. (Daniel Thayer) * Add documentation of the configuration framework, and improve existing script comments. (Daniel Thayer) 2.5-459 | 2018-03-07 12:46:57 -0600 * Update a doc test/baseline (Corelight) * Add removed root certificate back to test that requires it. Test has a trace that contains a WoSign certificate - they are no longer recognized by pretty much anyone. (Johanna Amann) 2.5-457 | 2018-02-18 17:35:50 -0600 * Fix another warning when building the documentation (Daniel Thayer) * Fix a warning when building documentation (Daniel Thayer) * Fix the config framework several-files.bro test (Daniel Thayer) * Update Mozilla CA list to state of NSS 3.35. (Johanna Amann) * Update list of Certificate Transparency logs. (Johanna Amann) 2.5-449 | 2018-02-14 08:49:27 -0800 * Patch in Binpac submodule that fixes an integer overflow (Philippe Antoine/Catena cyber). 2.5-448 | 2018-02-12 11:09:00 -0600 * Fix pessimizing-move compiler warning. (Corelight) 2.5-447 | 2018-02-12 11:00:44 -0600 * Add limit to number of auth flavors parsed out of MNT replies (Corelight) * Treat LibreSSL as an older OpenSSL (Xiaogrill) 2.5-445 | 2018-02-07 14:20:59 -0800 * Add new configuration framework for dynamically changing script options at runtime. See NEWS for more. (Corelight) * Allow the empty field separator to be empty when reading through the input frameworkk. (Corelight) 2.5-435 | 2018-02-06 08:40:38 -0800 * BIT-1854: Improve reassembly overlap checking. (Corelight) * BIT-1854: Fix the 'tcp_excessive_data_without_further_acks' option. (Corelight) * Make parsing of ServerKeyExchange work for D(TLS) < 1.2. (Johanna Amann) * Add more details to ssl_server_signature. (Johanna Amann) 2.5-427 | 2018-02-05 15:09:14 -0800 * BIT-1898: Fix problems with SumStats non-cluster.bro script. Reported by Jim Mellander. (Corelight) 2.5-424 | 2018-02-05 15:07:20 -0800 * Add a .travis.yml. file (Daniel Thayer) 2.5-422 | 2018-02-05 16:28:25 -0600 * fix setup field handling in smb1_com_transaction_request messages This field is an array of 16 bit words and was parsed as an array of 32 bit words. Moreover, one can not assume the format is going to be a 16 bits opcode followed by a 16 bit file ID, the content of the setup field is different according to its first 16 bits word that defines the subcommand code. See MS-CIFS section 2.2.4.33.1 : Setup (variable): An array of two-byte words that provides transaction context to the server. The size and content of the array are specific to individual subcommands. (Jeffrey Bencteux) * add smb1_transaction2_secondary_request event parse and expose SMB_COM_TRANSACTION2_SECONDARY (0x33) message to script level. See MS-CIFS section 2.2.4.47.1. (Jeffrey Bencteux) * add smb1_transaction_secondary_request event expose SMB_COM_TRANSACTION_SECONDARY (0x26) message to script language. See MS-CIFS section 2.2.4.34.1. (Jeffrey Bencteux) * add parameters and data to smb1_transaction_request/response messages expose SMB_Data.Trans_Parameters and SMB_Data.Trans_Data fields of SMB_COM_TRANSACTION (0x25) message type. See MS-CIFS section 2.2.4.33.1. These fields are exposed to the script level as Bro strings. Note that this commit also expose a new event smb1_transaction_response. (Jeffrey Bencteux) * add SMB_Parameters.Words to smb1_transaction2_request event expose the fields contained in SMB_Parameters.Words of the SMB_COM_TRANSACTION2 (0x32) message to the script language. See MS-CIFS section 2.2.46.1. (Jeffrey Bencteux) 2.5-410 | 2018-02-05 15:18:41 -0600 * Fix warnings when building sphinx docs (Corelight) 2.5-409 | 2018-02-05 14:12:21 -0600 * Bug fix: nfs3_writeargs didn't properly return filehandle. (Devin Trejo) * Add NFS events and unit tests: nfs_proc_symlink, nfs_proc_link, nfs_proc_sattr. (Devin Trejo) 2.5-405 | 2018-02-05 13:29:39 -0600 * Add MOUNT3 protocol parser. It's not activated by default. New events available: mount_proc_null, mount_proc_mnt, mount_proc_umnt, mount_proc_umnt_all, mount_proc_not_implemented, mount_reply_status. (Devin Trejo) 2.5-402 | 2018-02-05 10:43:59 -0600 * Fix (unlikely) memory leak in nb_dns.c (Corelight) * Remove bro-plugins submodule from .gitmodules file (Daniel Thayer) 2.5-399 | 2018-01-30 14:31:45 -0800 * Adapt the X509 analyzer to partially support OpenSSL 1.1. (Hilgo Bengen, Johanna Amann) 2.5-395 | 2018-01-26 15:46:05 -0600 * BIT-1894: fix bad integer casts in BIFs: sort, rand, order, to_int (Corelight) 2.5-392 | 2018-01-19 11:39:34 -0600 * Make nearly all bool conversion operators explicit. (Corelight) 2.5-390 | 2018-01-17 16:09:55 -0600 * Logging: implement get_filter_names and small fixes. get_filter_names(id: ID) : set[string] returns the names of the current list of filters for a specified log stream. Furthermore this commit makes a number of logging functions more robust by checking existence of values before trying to modify them. This commit also really implements (and tests) the enable_stream function. (Corelight) 2.5-388 | 2018-01-16 15:35:21 -0600 * HTTP: Recognize and skip upgrade/websocket connections. This patches the HTTP analyzerto recognize when a connections is upgraded to a different protocol (e.g. client uses the Upgrade header and gets a 101 reply). In this case, the analyzer stops further processing (which would have resulted in DPD errors) and raises a new event: event http_connection_upgrade(c: connection, protocol: string); The protocol parameter contains the name of the protocol that is being upgraded to, as specified in one of the header values. (Johanna Amann) 2.5-385 | 2018-01-16 14:51:07 -0600 * Make tunnel_parents in conn.log optional. This makes conn.logs a bit prettier (and smaller) because all lines tha do not use a tunnel will now have a "-" instead of the "(empty)" for tunnel_parents. (Johanna Amann) * Correct include-path in x509Common.h (Johanna Amann) 2.5-381 | 2018-01-12 10:03:21 -0800 * Preallocate all possible PortVals, mark PortVal ctors deprecated. The performance benefit is small (maybe ~1% at most), however, it's a trivial change without downsides. (Jon Siwek) * Add BRO_DEPRECATED macro. (Jon Siwek) * Add functions for retrieving files by their id. There are two new script level functions to query and lookup files from the core by their IDs. These are adding feature parity for similarly named functions for files. The function prototypes are as follows: Files::file_exists(fuid: string): bool Files::lookup_File(fuid: string): fa_file (Seth Hall) 2.5-375 | 2018-01-11 11:47:01 -0600 * Fix a test that fails in some environments (Daniel Thayer) * Add CVE ID for BIT-1856. (Johanna Amann) 2.5-372 | 2017-12-15 15:08:51 -0600 * Remove some DNS weirds that caused volume and are generally not useful: dns_unmatched_msg, dns_unmatched_msg_quantity, dns_unmatched_reply. (Corelight) 2.5-369 | 2017-12-13 14:22:47 -0600 * Fix typo in analyzer::Manager API docs (Corelight) 2.5-368 | 2017-12-08 13:09:25 -0600 * Improve for-loop iteration performance over empty tables. (Justin Azoff) * Fix gcc7 warnings. (Johanna Amann) 2.5-363 | 2017-12-05 11:00:09 -0600 * Fix documentation for ReassemblerStats. (Corelight) 2.5-362 | 2017-12-02 09:45:04 -0600 * BIT-1791: Do not log SOCKS passwords by default and add SOCKS::default_capture_password option. (Johanna Amann) * Add missing ; in SSL binpac parser, found by Luke Valenta. (Johanna Amann) 2.5-359 | 2017-11-29 14:01:37 -0600 * Add --ccache option to configure script (requires CMake 3.10+). (Corelight) 2.5-358 | 2017-11-28 12:28:14 -0800 * Extend the TLS analyzer with several events containing cryptographic parameters from the client and server key exchanges. The new events are: ssl_ecdh_server_params, ssl_dh_server_params, ssl_server_signature, ssl_ecdh_client_params, ssl_dh_client_params, ssl_rsa_client_pms Since ssl_ecdh_server_params contains more information than the old ssl_server_curve event, ssl_server_curve is now marked as deprecated. (Luke Valenta) 2.5-352 | 2017-11-21 13:21:51 -0600 * Fix assignments to event arguments becoming visible to subsequent handlers. (Robin Sommer) 2.5-350 | 2017-11-21 12:19:28 -0600 * Add HookReporter plugin hook function. This hook gives access to basically all information that is available in the function in Reporter.cc that performs the logging. The hook is called each time when anything passes through the reporter in the cases in which an event usually would be called. This includes weirds. The hook can return false to prevent the normal reporter events from being raised. (Corelight) 2.5-348 | 2017-11-21 11:30:55 -0600 * Fix a nb_dns.c compile error (older OSs) due to C90 vs C99. (Corelight) 2.5-347 | 2017-11-20 14:00:37 -0600 * Fix and extend behavior of HookLoadFile. (Corelight) 2.5-345 | 2017-11-20 11:28:59 -0600 * BIT-1827: fix error on initializing DNS w/ IPv6 nameserver. (Corelight) * Add --build-type flag to configure wrapper. (Corelight) 2.5-343 | 2017-11-17 15:27:04 -0800 * Fix ASCII logging of very large values of type "double". Previously, the nonsensical "NAN.0" would be written to ASCII logs for any value >= 1e248). (Daniel Thayer) * Add more test cases to ascii-double.bro (Daniel Thayer) * Enforce a maximum line length in ContentLine analyzer. (Justin Azoff) * Fix OOB read with IP packets that have a header length greater than the total length of their packet. (Johanna Amann) * Verify version field of IP packets read from tunnels. (Johanna Amann) 2.5-332 | 2017-10-27 13:27:16 -0700 * Bro docs tweaks for correctness and readability. (Christian Kreibich) * Fix use-after-free in Trigger.cc. (Johanna Amann) 2.5-328 | 2017-10-16 13:13:41 -0700 * Patch OOB write in content-line analyzer. A combination of packets can trigger an out of bound write of '0' byte in the content-line analyzer. Addresses BIT-1856 / CVE-2017-1000458. (Frank Meier/Johanna Amann) 2.5-327 | 2017-10-16 12:21:01 -0700 * Updating submodule(s). 2.5-326 | 2017-10-05 14:34:20 -0700 * Update the SSH analyzer to support the "curve25519-sha256" KEX. (Vlad Grigorescu) 2.5-321 | 2017-10-03 12:00:29 -0500 * Add "-B scripts" flag to allow debug output of script load order. (Corelight) * Fix segmentation fault on eval condition with no return value. (Corelight) 2.5-317 | 2017-09-29 09:54:50 -0400 * BIT-1853 - Fix an issue with broctl triggering reporter error in the intel framework. (Justin Azoff) * BIT-1845 - Make "in" keyword work with binary data. (Johanna Amann) * Add TLS 1.3 fix and testcase due to Google Chrome's use of TLS 1.3. It turns out that Chrome supports an experimental mode to support TLS 1.3, which uses a non-standard way to negotiate TLS 1.3 with a server. This non-standard way to negotiate TLS 1.3 breaks the current draft RFC and re-uses an extension on the server-side with a different binary formatting, causing us to throw a binpac exception. This patch ignores the extension when sent by the server, continuing to correctly parse the server_hello reply (as far as possible). From what I can tell this seems to be google working around the fac that MITM equipment cannot deal with TLS 1.3 server hellos; this change makes the fact that TLS 1.3 is used completely opaque unless one looks into a few extensions. We currently log this as TLS 1.2. (Johanna Amann) 2.5-310 | 2017-09-21 09:10:21 -0700 * fix interaction of gridftp scripts with other thresholds. (Justin Azoff) 2.5-307 | 2017-09-20 10:51:09 -0500 * BIT-1846: Updating broctl submodule to include fix for symlinking issue (Jon Siwek) 2.5-306 | 2017-09-18 14:43:42 -0700 * Make strerror_r portable, supporting XSI/gnu versions. (Thomas Petersen) * Prevent crash when calling bro -U. (Thomas Petersen) * Remove annoying error message from connsize bifs. (Johanna Amann) * Add test to verify that log rotation works with gzipped logs (Daniel Thayer) * Fix ascii writer to not discard a ".gz" file extension. (Daniel Thayer) When Bro writes a compressed log, it uses a file extension of ".gz". However, upon log rotation the ascii writer script function "default_rotation_postprocessor_func" was discarding the ".gz" file extension. Fixed so that the correct file extension is preserved after rotation. (Daniel Thayer) 2.5-297 | 2017-09-11 09:26:33 -0700 * Fix small OCSP parser bug; serial numbers were not passed to events (Johanna Amann) * Fix expire-redef.bro test. (Daniel Thayer) 2.5-294 | 2017-08-11 13:51:49 -0500 * Fix core.truncation unit test on macOS. (Jon Siwek) * Fix a netcontrol test that often fails (Daniel Thayer) * Update install instructions for Fedora 26 (Daniel Thayer) 2.5-288 | 2017-08-04 14:17:10 -0700 * Fix field not being populated, which resulted in a reporter messsage. Addresses BIT-1831. Reported by Chris Herdt. (Seth Hall) * Support for OCSP and Signed Certificate Timestamp. (Liang Zhu/Johanna Amann) - OCSP parsing is added to the X.509 module. - Signed Certificate Timestamp extraction, parsing, & validation is added to the SSL, X.509, and OCSP analyzers. Validation is added to the X.509 BIFs. This adds the following events and BIFs: - event ocsp_request(f: fa_file, version: count, requestorName: string); - event ocsp_request_certificate(f: fa_file, hashAlgorithm: string, issuerNameHash: string, issuerKeyHash: string, serialNumber: string); - event ocsp_response_status(f: fa_file, status: string); - event ocsp_response_bytes(f: fa_file, resp_ref: opaque of ocsp_resp, status: string, version: count, responderId: string, producedAt: time, signatureAlgorithm: string, certs: x509_opaque_vector); - event ocsp_response_certificate(f: fa_file, hashAlgorithm: string, issuerNameHash: string, issuerKeyHash: string, serialNumber: string, certStatus: string, revokeTime: time, revokeReason: string, thisUpdate: time, nextUpdate: time); - event ocsp_extension(f: fa_file, ext: X509::Extension, global_resp: bool); - event x509_ocsp_ext_signed_certificate_timestamp(f: fa_file, version: count, logid: string, timestamp: count, hash_algorithm: count, signature_algorithm: count, signature: string); - event ssl_extension_signed_certificate_timestamp(c: connection, is_orig: bool, version: count, logid: string, timestamp: count, signature_and_hashalgorithm: SSL::SignatureAndHashAlgorithm, signature: string); - function sct_verify(cert: opaque of x509, logid: string, log_key: string, signature: string, timestamp: count, hash_algorithm: count, issuer_key_hash: string &default=""): bool - function x509_subject_name_hash(cert: opaque of x509, hash_alg: count): string - function x509_issuer_name_hash(cert: opaque of x509, hash_alg: count): string - function x509_spki_hash(cert: opaque of x509, hash_alg: count): string This also changes the MIME types that we use to identify X.509 certificates in SSL connections from "application/pkix-cert" to "application/x-x509-user-cert" for host certificates and "application/x-x509-ca-cert" for CA certificates. * The SSL scripts provide a new hook "ssl_finishing(c: connection)" to trigger actions after the handshake has concluded. (Johanna Amann) * Add an internal API for protocol analyzers to provide the MIME type of file data directly, disabling automatic inferrence. (Johanna Amann). 2.5-186 | 2017-07-28 12:22:20 -0700 * Improved handling of '%' at end of line in HTTP analyzer. (Johanna Amann) * Add canonifier to catch and release test that should fix test failures. (Johanna Amann) 2.5-181 | 2017-07-25 16:02:41 -0700 * Extend plugin infrastructure to catch Bro version mismatches at link time. The version number used for the function name is slightly normalized to skip any git revision postfixes (i.e., "2.5-xxx" is always treated as "2.5-git") so that one doesn't need to recompile all plugins after every master commit. That seems good enough, usually people run into this when upgrading to a new release. The Plugin API version is also part of the version number. If one loads an old plugin into a new Bro, the error message looks like this: $ bro -NN Demo::Foo fatal error in /home/robin/bro/master/scripts/base/init-bare.bro, line 1: cannot load plugin library /home/robin/tmp/p/build//lib/Demo-Foo.linux-x86_64.so: /home/robin/tmp/p/build//lib/Demo-Foo.linux-x86_64.so: undefined symbol: bro_version_2_5_git_debug (Robin Sommer) * Several fixes and improvements for software version parsing. - Addresses Philip Romero's question from the Bro mailing list. - Adds Microsoft Edge as a detected browser. - We are now unescaping encoded characters in software names. (Seth Hall) * Remove another reference to now removed bro-plugins. (Johanna Amann) 2.5-175 | 2017-07-07 14:35:11 -0700 * Removing aux/plugins. Most of the plugins are now Bro packages. (Robin Sommer) * Update install instructions for Debian 9. (Daniel Thayer) 2.5-170 | 2017-07-07 12:20:19 -0700 * Update krb-protocol.pac (balintm) This fixes parsing of KRB_AP_Options where the padding and flags were reversed. * Add new cipher suites from draft-ietf-tls-ecdhe-psk-aead-05 (Johanna Amann) * Test changes: remove loading of listen.bro in tests that do not use it, serialize tests that load listen.bro, fix race conditions in some tests. (Daniel Thayer) * The broccoli-v6addrs "-r" option was renamed to "-R" (Daniel Thayer) 2.5-156 | 2017-06-13 11:01:56 -0700 * Add 2.5.1 news file to master. (Johanna Amann) * Remove link to no longer existing myricom plugin. (Johanna Amann) 2.5-152 | 2017-06-05 15:16:49 -0700 * Remove non-existing links; this broke documentation build. (Johanna Amann) * Fix at_least in Version.bro - it did exactly the oposite of the documented behavior. (Johanna Amann) 2.5-147 | 2017-05-22 20:32:32 -0500 * Add nfs unittest. (Julien Wallior) * Added nfs_proc_rename event to rpc/nfs protocol analyzer. (Roberto Del Valle Rodriguez) * Expand parsing of RPC Call packets to add Uid, Gid, Stamp, MachineName and AuxGIDs (Julien Wallior) * Fix NFS protocol parser. (Julien Wallior) 2.5-142 | 2017-05-22 00:08:52 -0500 * Add gzip log writing to the ascii writer. This feature can be enabled globally for all logs by setting LogAscii::gzip_level to a value greater than 0. This feature can be enabled on a per-log basis by setting gzip-level in $config to a value greater than 0. (Corelight) 2.5-140 | 2017-05-12 15:31:32 -0400 * Lessen cluster load due to notice suppression. (Johanna Amann, Justin Azoff) 2.5-137 | 2017-05-04 11:37:48 -0500 * Add plugin hooks for log init and writing: HookLogInit and HookLogWrite. (Corelight) * TLS: Fix compile warning (comparison between signed/unsigned). This was introduced with the addition of new TLS1.3 extensions. (Johanna Amann) 2.5-134 | 2017-05-01 10:34:34 -0500 * Add rename, unlink, and rmdir bifs. (Corelight) 2.5-131 | 2017-04-21 14:27:16 -0700 * Guard more format strings with __attribute__((format)). (Johanna Amann) * Add support for two TLS 1.3 extensions. New events: - event ssl_extension_supported_versions(c: connection, is_orig: bool, versions: index_vec) - event ssl_extension_psk_key_exchange_modes(c: connection, is_orig: bool, modes: index_vec) (Johanna Amann) 2.5-125 | 2017-04-17 22:02:39 +0200 * Documentation updates for loading Bro scripts. (Seth Hall) 2.5-123 | 2017-04-10 13:30:14 -0700 * Fix some failing tests by increasing delay times. (Daniel Thayer) * Threading Types: add a bit of documentation to subnet type. (Johanna Amann) * Fixing couple issues reported by Coverity. (Robin Sommer) 2.5-119 | 2017-04-07 10:30:09 -0700 * Fix the test group name in some broker test files. (Daniel Thayer) * NetControl: small rule_error changes (test, call fix). (Johanna Amann) * SSL: update dpd signature for TLS1.3. (Johanna Amann) 2.5-115 | 2017-03-23 07:25:41 -0700 * Fix a test that was failing on some platforms. (Daniel Thayer) * Remove test for cluster catch and release. This test keeps failing intermittently because of timing issues that are surprisingly hard to fix. (Johanna Amann) * Fix some Coverity warnings. (Daniel Thayer) 2.5-106 | 2017-03-13 11:19:03 -0700 * print version string to stdout on --version, instead of printing it to stderr. Output is not an error output. (Pete) * Fix compiler warning raised by llvm8. (Johanna Amann) * Fix coverity warning in Ascii reader. (Johanna Amann) 2.5-101 | 2017-03-09 12:20:11 -0500 * Input's framework's ascii reader is now more resilient. By default, the ASCII reader does not fail on errors anymore. If there is a problem parsing a line, a reporter warning is written and parsing continues. If the file is missing or can't be read, the input thread just tries again on the next heartbeat. (Seth Hall, Johanna Amann) 2.5-92 | 2017-03-03 10:44:14 -0800 * Move most threading to C++11 primitives (mostly). (Johanna Amann) * Fix a test that sometimes fails on FreeBSD. (Daniel Thayer) * Remove build time warnings. (Seth Hall) 2.5-84 | 2017-02-27 15:08:55 -0500 * Change semantics of Broker's remote logging to match old communication framework. (Robin Sommer) * Add and fix documentation for HookSetupAnalyzerTree (Johanna Amann) 2.5-76 | 2017-02-23 10:19:57 -0800 * Kerberos ciphertext had some additional ASN.1 content being lumped in. (Vlad Grigorescu) * Updated Windows version detection to include Windows 10. (Fatema Bannatwala, Keith Lehigh, Mike, Seth Hall). 2.5-70 | 2017-02-20 00:20:02 -0500 * Rework the RADIUS base script. Fixes BIT-1769 which improves logging behavior when replies aren't seen. Also added a `framed_addr` field to indicate if the radius server is hinting at an address for the client and a `ttl` field to show how quickly the server is responding. (Seth Hall) 2.5-68 | 2017-02-18 13:59:05 -0500 * Refactored base krb scripts. (Seth Hall) * New script to log ticket hashes in krb log (policy/protocols/krb/ticket-logging.bro). Also, add ciphertext to ticket data structure. (John E. Rollinson) 2.5-62 | 2017-02-15 15:56:38 -0800 * Fix case in which scripts were able to access unitialized variables in certain cases. Addresses BIT-1785. (Jon Siwek) 2.5-60 | 2017-02-15 15:19:20 -0800 * Implement ERSPAN support. There is a small caveat to this implementation. The ethernet header that is carried over the tunnel is ignored. If a user tries to do MAC address logging, it will only show the MAC addresses for the outer tunnel and the inner MAC addresses will be stripped and not available anywhere. (Seth Hall) * Tiny mime-type fix from Dan Caselden. (Seth Hall) * Update failing intel framework test. (Johanna Amann) 2.5-55 | 2017-02-10 09:50:43 -0500 * Fixed intel expiration reset. Reinserting the same indicator did not reset the expiration timer for the indicator in the underlying data store. Addresses BIT-1790. (Jan Grashoefer) 2.5-51 | 2017-02-06 10:15:56 -0500 * Fix memory leak in file analyzer. (Johanna Amann) * Fix a series of problems with the to_json function. Addresses BIT-1788. (Daniel Thayer) 2.5-44 | 2017-02-03 16:38:10 -0800 * Change snap lengths of some tests. (Johanna Amann) * Fix layer 2 connection flipping. If connection flipping occured in Sessions.cc code (invoked e.g. when the original SYN is missing), layer 2 flipping was not performed. (Johanna Amann) 2.5-39 | 2017-02-01 14:03:08 -0800 * Fix file analyzer memory management, and a delay in disabling file analyzers. File analyzers are no longer deleted immediately; this is delayed until a file opject is destroyed. Furthermore, no data is sent to file analyzers anymore after they have been disabled. 2.5-33 | 2017-02-01 10:07:47 -0500 * New file types sigs. (Keith Lehigh) * Change snaplen of test trace from 1,000,000 to 10,000 Recent versions of libpcap are unhappy with values bigger than 262,144 and will refuse reading the file. (Johanna Amann) 2.5-30 | 2017-01-26 13:24:36 -0800 * Extend file extraction log, adding extracted_cutoff and extracted_size fields. (Seth Hall) * Add new TLS extension type (cached_info) (Johanna Amann) * Remove brocon event; it caused test failures. (Johanna Amann) * Add missing paths to SMB Log::create_streams calls. (Johanna Amann) * Tiny xlsx file signature fix. (Dan Caselden) * Allow access to global variables using GLOBAL:: namespace. Addresses BIT-1758. (Francois Pennaneac) 2.5-17 | 2016-12-07 14:51:37 -0800 * Broxygen no longer attempts to do tilde expansion of PATH, giving an error message instead if bro is located in a PATH component that starts with a tilde. Broxygen also no longer attempts to get the mtime of the bro executable when bro is not invoced with the "-X" option. (Daniel Thayer) * Fix failing tests, compiler warnings and build issues on OpenBSD. (Daniel Thayer) 2.5-9 | 2016-12-05 11:39:54 -0800 * Fix validation of OCSP replies inside of Bro. (Johanna Amann) At one place in the code, we did not check the correct return code. This makes it possible for a reply to get a response of "good", when the OCSP reply is not actually signed by the responder in question. This also instructs OCSP verication to skip certificate chain validation, which we do ourselves earlier because the OCSP verify function cannot do it correctly (no way to pass timestamp). 2.5-6 | 2016-11-29 12:51:04 -0800 * Fix a build failure on OpenBSD relating to pcap_pkthdr. Also fixes an include issue on OpenBSD. (Daniel Thayer) * Fix compile error in krb-types.pac. (Johanna Amann) * Update krb-types.pac: KerberosString formatting for the principal principal name is now compliant with RFC 4120 section 5.2.2. (jamesecorrenti) 2.5 | 2016-11-16 14:51:59 -0800 * Release 2.5. 2.5-beta2-17 | 2016-11-14 17:59:19 -0800 * Add missing '@load ./pubkey-hashes' to policy/frameworks/intel/seen. (Robin Sommer) 2.5-beta2-15 | 2016-11-14 17:52:55 -0800 * Remove unused "bindist" make target. (Daniel Thayer) * Improve the "How to Upgrade" page in the Bro docs. (Daniel Thayer) * Update the quickstart guide for the deploy command. (Daniel Thayer) * Improved installation instructions for Mac OS X. (Daniel Thayer) * Lots of more small updates to documentation. (Daniel Thayer) 2.5-beta2 | 2016-11-02 12:13:11 -0700 * Release 2.5-beta2. 2.5-beta-135 | 2016-11-02 09:47:20 -0700 * SMB fixes and cleanup. Includes better SMB error handling, improved DCE_RPC handling in edge cases where drive_mapping is not seen. The concept of unknown shares has been removed with this change. Also fixes SMB tree connect handling and removes files that are not parsed. SMB2 error parsing is disabled because it never was implemented correctly. (Seth Hall) * Including a test for raw NTLM in SMB (Seth Hall) * Updates for SMB auth handling from Martin van Hensbergen. - Raw NTLM (not in GSSAPI) over SMB is now handled correctly. - The encrypted NTLM session key is now passed into scriptland through the ntlm_authenticate event. (Seth Hall) * Add a files framework signature for VIM tmp files. (Seth Hall) * Version parsing scripts now supports several beta versions. (Johanna Amann) 2.5-beta-123 | 2016-11-01 09:40:49 -0700 * Add a new site policy script local-logger.bro. (Daniel Thayer) 2.5-beta-121 | 2016-10-31 14:24:33 -0700 * Python 3 compatibility fixes for documentation building. (Daniel Thayer) 2.5-beta-114 | 2016-10-27 09:00:24 -0700 * Fix for Sphinx >= 1.4 compability. (Robin Sommer) 2.5-beta-113 | 2016-10-27 07:44:25 -0700 * XMPP: Fix detection of StartTLS when using namespaces. (Johanna Amann) 2.5-beta-110 | 2016-10-26 09:42:11 -0400 * Improvements DCE_RPC analyzer to make it perform fragment handling correctly and generally be more resistent to unexpected traffic. (Seth Hall) 2.5-beta-102 | 2016-10-25 09:43:45 -0700 * Update number of bytes in request/response of smb1-com-open-andx.pac. (balintm) * Fix a IPv4 CIDR specifications and payload-size condition of signature matching. (Robin Sommer) * Python 3 compatibility fix for coverage-calc script. (Daniel Thayer) 2.5-beta-93 | 2016-10-24 11:11:07 -0700 * Fix alignment issue of ones_complement_checksum. This error occured reproducibly newer compilers when called from icmp6_checksum. (Johanna Amann) 2.5-beta-91 | 2016-10-20 11:40:37 -0400 * Fix istate.pybroccoli test on systems using Python 3. (Daniel Thayer) 2.5-beta-89 | 2016-10-18 21:50:51 -0400 * SSH analyzer changes: the events are now restructured a bit. There is a new event ssh_auth_attempted, that is raised each time authentication is tried. ssh_auth_failed is still only being raised once per connection. There also is an additional event ssh_auth_result giving more information about the number of times that authentication was tried and if it succeded/failed in the end. The number of authentication attemps are now part of ssh.log. Addresses BIT-1641. (Vlad Grigorescu) 2.5-beta-79 | 2016-10-13 15:58:48 -0700 * Fix MD5 problem with FreeBSD 11.0 and clang 3.8. The apparent cause is some confusion in clang when using a static char inside a static inline function that is refered to several compilation units. (Johanna Amann) * Initial TLS 1.3 support, as of draft-16. (Johanna Amann) 2.5-beta-73 | 2016-10-13 14:03:04 -0700 * Added missing README files for documentation. (Daniel Thayer) * List new log files in the log-files.rst document. (Daniel Thayer) 2.5-beta-67 | 2016-10-10 08:28:38 -0700 * Fixes for DCE_RPC analyzer. (Seth Hall) - DCE_RPC fragmentation handling returns. - Fixed some general parsing issues. - Fixed an issue with the DCE_RPC signature not working for IPv6 connections. 2.5-beta-64 | 2016-10-10 08:20:42 -0700 * Fix httpd.py test script for Py3 compatibility. (Daniel Thayer) * Tiny fix for a DCE_RPC script issue. Fixes BIT-1688. (Seth Hall) * Fix for plugins/hooks test. (Johanna Amann) * Update a TLS constants in preparation for TLS 1.3, and rename a few names that had never been formally assigned yet. (Johanna Amann) * Fixing Broxygen indexing confusion for plugins. Broxygen now indexes plugin scripts as, e.g., "Bro_Netmap/scripts/init.bro". Addresses BIT-1693. (Robin Sommer) 2.5-beta-54 | 2016-10-06 14:24:01 -0700 * Fixing documentation piece on the interesting-hostname script. (Robin Sommer) * Improve the SMB documentation. (Vlad Grigorescu) 2.5-beta-46 | 2016-10-06 14:11:03 -0700 * Fixing Broxygen indexing confusion for plugins. Scripts in plugins now get an artificial index prefix: "plugin_name/", followed by the script's relative path inside the plugin's top-level directory. For example, "/opt/bro/lib/bro/plugins/Bro_Netmap/scripts/init.bro" now turns into "Bro_Netmap/scripts/init.bro" for Broxygen purposes (whereas it used to be just "init.bro"). (Robin Sommer) * Fix a couple of tests, addressing issues of the newly introduced version.bro as well as small FreeBSD 11.0 issues. (Johanna Amann) * Update documentation license. (Johanna Amann) * Add a convenient way to access version information to Bro. @if ( Version::number >= 20500 ) or @if ( Version::at_least("2.5") ) Version::info contains detailed information about the running version of Bro, including beta flags, etc. (Johanna Amann) 2.5-beta-35 | 2016-10-02 17:38:31 -0400 * Normalize http host in seen script. (Johanna Amann) 2.5-beta-33 | 2016-10-02 14:42:22 -0400 * Handle removing non-existent intel items. (Jan Grashoefer) 2.5-beta-29 | 2016-09-28 18:18:35 -0700 * Prettifying reporter output in case no epxression is associated with runtime error. (Robin Sommer) 2.5-beta-28 | 2016-09-27 11:44:33 -0700 * Check if the number of fields in a log write are equal to the number of fields required. Addresses BIT-1683. (Johanna Amann) * Fix a small memory leak for disabled log writers. (Johanna Amann) * Fix loaded-scripts.bro to work with arbitrary indentation levels. Addresses BIT-1691 (Johanna Amann) * In interesting-hostnames.bro, move lookup_addr() outside of when statement to avoid expensive cloning of full connection record. Addresses BIT-1670. (Justin Azoff) 2.5-beta-21 | 2016-09-26 10:15:23 -0700 * Fix a debugger bug where it would not support statements like print(3). Addresses BIT-1703. (Vlad Grigorescu) 2.5-beta-19 | 2016-09-19 17:16:40 -0700 * Kerberos updates (Vlad Grigorescu): * Make PA_EncType_Info salt field optional. * Add support for parsing ENCTYPE_INFO pre-auth data (same as ENCTYPE_INFO2). 2.5-beta-17 | 2016-09-19 15:26:14 -0700 * Clarified string and fa_file documentation (Moshe Kaplan) 2.5-beta-12 | 2016-09-06 07:35:38 -0700 * Added a missing fclose in scan.l. Addresses BIT-1690. (Daniel Thayer). * Fix issue with file_extraction_limit event. (Seth Hall) * Fix a crash when a user disables DCE_RPC while enabling SMB. (Seth Hall) 2.5-beta-6 | 2016-08-19 07:50:10 -0700 * Clarify explanation of mime_entity_data event. (Moshe Kaplan) * Update NEWS, correcting typos formatting and adding more information. (Daniel Thayer) * Remove old ack_above_hole event from scripts. Fixes BIT-1673. (Johanna Amann) 2.5-beta | 2016-08-17 10:37:49 -0700 * Release 2.5-beta. * Fix sphinx build errors (Johanna Amann) * Change failure in utf16_bytestring_to_utf8_val to be a conn weird. (Seth Hall) * Fix test failure caused by uninitialized memory. (Johanna Amann) * SMB: fix rounding error due to value truncation when converting timestamps. (Johanna Amann) 2.4-947 | 2016-08-16 12:10:02 -0700 * Fix issues with handling of indermediate sumstats updates. (Justin Azoff) * Address coverity errors. (Johanna Amann) 2.4-943 | 2016-08-15 17:03:14 -0700 * Add 'bro-config' script. (Jon Siwek) * add certificate to external list for debian 8 to fix test. (Johanna Amann) * KRB: fix field value missing error for msg$client_name. (Johanna Amann) * Tiny SMB cleanup in pipe handling as well as NTLM fixes. (Seth Hall). * Fix a number of format specifier errors. (Johanna Amann) * Make several tests more stable. (Johanna Amann) 2.4-927 | 2016-08-11 21:49:06 -0700 * Make component tags generated during component initialization stable. Before, it was dependent on the order a compiler called constructors. This makes a few tests work with gcc. (Johanna Amann) * Make x509 intel seen script more robust (Johanna Amann) * Input: DisableFrontend was not called upon Init failure. (Johanna Amann) * DCE_RPC code simplification. (Seth Hall) 2.4-921 | 2016-08-10 20:29:48 -0700 * Add logging framework ext-data mechanism. It is now possible to extend logs by adding new data columns by them - either to specific ones, or globally to all logs. This can, e.g., be used to add node names to all logs. (Seth Hall) * Add unrolling separator & field name map to logging framework. One can now use logging separators other than ".", as well as change specific column names in logs. (Seth Hall) * Fix memory leak in EnumType. (Johanna Amann) * Fix configure warning when compiling with --enable-broker. (Johanna Amann) * Add netcontrol-connectors to aux directory. (Johanna Amann) * Update Mozilla CA list. (Johanna Amann) * update scripts loaded by default in local.bro. Traceroute is now disabled by default, stats and capture-loss enabled by default. (Johanna Amann) 2.4-907 | 2016-08-09 15:42:17 -0400 * Updating NEWS. 2.4-905 | 2016-08-09 08:19:37 -0700 * GSSAPI analyzer now forwards authentication blobs more correctly. (Seth Hall) * The KRB analyzer now includes support for the PA_ENCTYPE_INFO2 pre-auth data type. (Seth Hall) * Add an argument to "disable_analyzer" function to not do a reporter message by default. (Seth Hall) 2.4-902 | 2016-08-08 16:50:35 -0400 * Adding SMB analyzer. (Seth Hall, Vlad Grigorescu and many others) * NetControl: allow reasons in remove_rule calls. Addresses BIT-1655 (Johanna Amann) 2.4-893 | 2016-08-05 15:43:04 -0700 * Remove -z/--analysis option. (Johanna Amann) * Remove already defunct code for XML serialization. (Johanna Amann) 2.4-885 | 2016-08-05 15:03:59 -0700 * Reverting SMB analyzer merge. (Robin Sommer) 2.4-883 | 2016-08-05 12:57:26 -0400 * Add a new node type for logging with the cluster framework scripts by adding a new Bro node type for doing logging (this is intended to reduce the load on the manager). If a user chooses not to specify a logger node in the cluster configuration, then the manager will write logs locally as usual. (Daniel Thayer) 2.4-874 | 2016-08-05 12:43:06 -0400 * SMB analyzer (Seth Hall, Vlad Grigorescu and many others) 2.4-759 | 2016-08-05 09:32:42 -0400 * Intel framework improvements (Jan Grashoefer) * Added expiration for intelligence items. * Improved intel notices. * Added hook to allow extending the intel log. * Added support for subnets to intel-framework. 2.4-742 | 2016-08-02 15:28:31 -0700 * Fix duplicate SSH authentication failure events. Addresses BIT-1641. (Robin Sommer) * Remove OpenSSL dependency for plugins. (Robin Sommer) 2.4-737 | 2016-08-02 11:38:07 -0700 * Fix some Coverity warnings. (Robin Sommer) 2.4-735 | 2016-08-02 11:05:36 -0700 * Added string slicing examples to documentation. (Moshe Kaplan) 2.4-733 | 2016-08-01 09:09:29 -0700 * Fixing a CMake dependency issue for the pcap bifs. (Robin Sommer) 2.4-732 | 2016-08-01 08:33:00 -0700 * Removing pkg/make-*-packages scripts. BIT-1509 #closed (Robin Sommer) 2.4-731 | 2016-08-01 08:14:06 -0700 * Correct endianness of IP addresses in SNMP. Addresses BIT-1644. (Anony Mous) 2.4-729 | 2016-08-01 08:00:54 -0700 * Fix behavior of connection_pending event. It is now really only raised when Bro is terminating. Also adds a test-case that raises the event. (Johanna Amann) * Retired remove -J/-K options (set md5/hash key) from the manpage. They had already been removed from the code. (Johanna Amann) * NetControl: Add catch-and-release event when IPs are forgotten. This adds an event catch_release_forgotten() that is raised once Catch & Release ceases block management for an IP address because the IP has not been seen in traffic during the watch interval. (Johanna Amann) 2.4-723 | 2016-07-26 15:04:26 -0700 * Add error events to input framework. (Johanna Amann) This change introduces error events for Table and Event readers. Users can now specify an event that is called when an info, warning, or error is emitted by their input reader. This can, e.g., be used to raise notices in case errors occur when reading an important input stream. Example: event error_event(desc: Input::TableDescription, msg: string, level: Reporter::Level) { ... } event bro_init() { Input::add_table([$source="a", $error_ev=error_event, ...]); } Addresses BIT-1181. * Calling Error() in an input reader now automatically will disable the reader and return a failure in the Update/Heartbeat calls. (Johanna Amann) * Convert all errors in the ASCII formatter into warnings (to show that they are non-fatal. (Johanna Amann) * Enable SQLite shared cache mode. This allows all threads accessing the same database to share sqlite objects. See https://www.sqlite.org/sharedcache.html. Addresses BIT-1325. (Johanna Amann) * NetControl: Adjust default priority of ACTION_DROP hook to standad level. (Johanna Amann) * Fix types when constructing SYN_packet record. Fixes BIT-1650. (Grant Moyer). 2.4-715 | 2016-07-23 07:27:05 -0700 * SQLite writer: Remove unused string formatting function. (Johanna Amann) * Deprecated the ElasticSearch log writer. (Johanna Amann) 2.4-709 | 2016-07-15 09:05:20 -0700 * Change Bro's hashing for short inputs and Bloomfilters from H3 to Siphash, which produces much better results for HLL in particular. (Johanna Amann) * Fix a long-standing bug which truncated hash values to 32-bit on most machines. (Johanna Amann) * Fixes to HLL. Addresses BIT-1612. (Johanna Amann) * Add test checking the quality of HLL. (Johanna Amann) * Remove the -K/-J options for setting keys. (Johanna Amann) * SSL: Fix memory management problem. (Johanna Amann) 2.4-693 | 2016-07-12 11:29:17 -0700 * Change TCP analysis to process connections without the initial SYN as non-partial connections. Addresses BIT-1492. (Robin Sommer). 2.4-691 | 2016-07-12 09:58:38 -0700 * SSL: add support for signature_algorithms extension. (Johanna Amann) 2.4-688 | 2016-07-11 11:10:33 -0700 * Disable broker by default. To enable it, use --enable-broker. Addresses BIT-1645. (Daniel Thayer) 2.4-686 | 2016-07-08 19:14:43 -0700 * Added flagging of retransmission to the connection history. Addresses BIT-977. (Robin Sommer) 2.4-683 | 2016-07-08 14:55:04 -0700 * Extending connection history field to flag with '^' when Bro flips a connection's endpoints. Addresses BIT-1629. (Robin Sommer) 2.4-680 | 2016-07-06 09:18:21 -0700 * Remove ack_above_hole() event, which was a subset of content_gap and led to plenty noise. Addresses BIT-688. (Robin Sommer) 2.4-679 | 2016-07-05 16:35:53 -0700 * Fix segfault when an existing enum identifier is added again with a different value. Addresses BIT-931. (Robin Sommer) * Escape the empty indicator in logs if it occurs literally as a field's actual content. Addresses BIT-931. (Robin Sommer) 2.4-676 | 2016-06-30 17:27:54 -0700 * A larger series of NetControl updates. (Johanna Amann) * Add NetControl framework documentation to the Bro manual. * Use NetControl for ACTION_DROP of notice framework. So far, this action did nothing by default. * Rewrite of catch-and-release. * Fix several small logging issues. * find_rules_subnet() now works in cluster mode. This introduces two new events, NetControl::rule_new and NetControl::rule_destroyed, which are raised when rules are first added and then deleted from the internal state tracking. * Fix acld whitelist command. * Add rule existance as a state besides added and failure. * Suppress duplicate "plugin activated" messages. * Make new Broker plugin options accessible. * Add predicates to Broker plugin. * Tweak SMTP scripts to not to pull in the notice framework. 2.4-658 | 2016-06-30 16:55:32 -0700 * Fix a number of documentation building errors. (Johanna Amann) * Input/Logging: Make bool conversion operator explicit. (Johanna Amann) * Add new TLS ciphers from RFC 7905. (Johanna Amann) 2.4-648 | 2016-06-21 18:33:22 -0700 * Fix memory leaks. Reported by Dk Jack. (Johanna Amann) 2.4-644 | 2016-06-21 13:59:05 -0400 * Fix an off-by-one error when grabbing x-originating-ip header in email. (Seth Hall, Aashish Sharma) 2.4-642 | 2016-06-18 13:18:23 -0700 * Fix potential mismatches when ignoring duplicate weirds. (Johanna Amann) * Weird: Rewrite internals of weird logging. (Johanna Amann) - "flow weirds" now actually log information about the flow that they occur in. - weirds can now be generated by calling Weird::weird() with the info record directly, allowing more fine-granular passing of information. This is e.g. used for DNS weirds. Addresses BIT-1578 (Johanna Amann) * Exec: fix reader cleanup when using read_files, preventing file descriptors from leaking every time it was used. (Johanna Amann) * Raw Writer: Make code more c++11-y, remove raw pointers. (Johanna Amann) * Add separate section with logging changes to NEWS. (Seth Hall) 2.4-635 | 2016-06-18 01:40:17 -0400 * Add some documentation for modbus data types. Addresses BIT-1216. (Seth Hall) * Removed app-stats scripts. Addresses BIT-1171. (Seth Hall) 2.4-631 | 2016-06-16 16:45:10 -0400 * Fixed matching mail address intel and added test (Jan Grashoefer) * A new utilities script named email.bro with some utilities for parsing out email addresses from strings. (Seth Hall) * SMTP "rcptto" and "mailfrom" fields now do some minimal parsing to clean up email addresses. (Seth Hall) * Added "cc" to the SMTP log and feed it into the Intel framework with the policy/frameworks/intel/seen/smtp.bro script. (Seth Hall) 2.4-623 | 2016-06-15 17:31:12 -0700 * &default values are no longer overwritten with uninitialized by the input framework. (Jan Grashoefer) 2.4-621 | 2016-06-15 09:18:02 -0700 * Fixing memory leak in changed table expiration code. (Robin Sommer) * Fixing test portability. (Robin Sommer) * Move the HTTP "filename" field (which was never filled out anyways) to "orig_filenames" and "resp_filenames". (Seth Hall) * Add a round trip time (rtt) field to dns.log. (Seth Hall) * Add ACE archive files to the identified file types. Addresses BIT-1609. (Stephen Hosom) 2.4-613 | 2016-06-14 18:10:37 -0700 * Preventing the event processing from looping endlessly when an event reraised itself during execution of its handlers. (Robin Sommer) 2.4-612 | 2016-06-14 17:42:52 -0700 * Improved handling of 802.11 headers. (Jan Grashoefer) 2.4-609 | 2016-06-14 17:15:28 -0700 * Fixed table expiration evaluation. The expiration attribute expression is now evaluated for every use. Thus later adjustments of the value (e.g. by redefining a const) will now take effect. Values less than 0 will disable expiration. (Jan Grashoefer) 2.4-606 | 2016-06-14 16:11:07 -0700 * Fix parsing precedence of "hook" expression. Addresses BIT-1619 (Johanna Amann) * Update the "configure" usage message for --with-caf (Daniel Thayer) 2.4-602 | 2016-06-13 08:16:34 -0700 * Fixing Covertity warning (CID 1356391). (Robin Sommer) * Guarding against reading beyond packet data when accessing L2 address in Radiotap header. (Robin Sommer) 2.4-600 | 2016-06-07 15:53:19 -0700 * Fixing typo in BIF macros. Reported by Jeff Barber. (Robin Sommer) 2.4-599 | 2016-06-07 12:37:32 -0700 * Add new functions haversine_distance() and haversine_distance_ip() for calculating geographic distances. They requires that Bro be built with libgeoip. (Aashish Sharma/Daniel Thayer). 2.4-597 | 2016-06-07 11:46:45 -0700 * Fixing memory leak triggered by new MAC address logging. (Robin Sommer) 2.4-596 | 2016-06-07 11:07:29 -0700 * Don't create debug.log immediately upon startup (BIT-1616). (Daniel Thayer) 2.4-594 | 2016-06-06 18:11:16 -0700 * ASCII Input: Accept DOS/Windows newlines. Addresses BIT-1198 (Johanna Amann) * Fix BinPAC exception in RFB analyzer. (Martin van Hensbergen) * Add URL decoding for the unofficial %u00AE style of encoding. (Seth Hall) * Remove the unescaped_special_char HTTP weird. (Seth Hall) 2.4-588 | 2016-06-06 17:59:34 -0700 * Moved link-layer addresses into endpoints. The link-layer addresses are now part of the connection endpoints following the originator/responder pattern. (Jan Grashoefer) * Link-layer addresses are extracted for 802.11 plus RadioTap. (Jan Grashoefer) * Fix coverity error (uninitialized variable) (Johanna Amann) * Use ether_ntoa instead of ether_ntoa_r The latter is thread-safe, but a GNU addition which does not exist on OS-X. Since the function only is called in the main thread, it should not matter if it is or is not threadsafe. (Johanna Amann) * Fix FreeBSD/OSX compile problem due to headers (Johanna Amann) 2.4-581 | 2016-05-30 10:58:19 -0700 * Adding missing new script file mac-logging.bro. (Robin Sommer) 2.4-580 | 2016-05-29 13:41:10 -0700 * Add Ethernet MAC addresses to connection record. c$eth_src and c$eth_dst now contain the Ethernet address if available. A new script protocols/conn/mac-logging.bro adds these to conn.log when loaded. (Robin Sommer) 2.4-579 | 2016-05-29 08:54:57 -0700 * Fixing Coverity warning. Addresses CID 1356116. (Robin Sommer) * Fixing FTP cwd getting overlue long. (Robin Sommer) * Clarifying notice documentation. Addresses BIT-1405. (Robin Sommer) * Changing protocol_{confirmation,violation} events to queue like any other event. Addresses BIT-1530. (Robin Sommer) * Normalizing test baseline. (Robin Sommer) * Do not use scientific notations when printing doubles in logs. Addresses BIT-1558. (Robin Sommer) 2.4-573 | 2016-05-23 13:21:03 -0700 * Ignoring packets with negative timestamps. Addresses BIT-1562 and BIT-1443. (Robin Sommer) 2.4-572 | 2016-05-23 12:45:23 -0700 * Fix for a table refering to a expire function that's not defined. Addresses BIT-1597. (Robin Sommer) 2.4-571 | 2016-05-23 08:26:43 -0700 * Fixing a few Coverity warnings. (Robin Sommer) 2.4-569 | 2016-05-18 07:39:35 -0700 * DTLS: Use magix constant from RFC 5389 for STUN detection. (Johanna Amann) * DTLS: Fix binpac bug with DTLSv1.2 client hellos. (Johanna Amann) * DTLS: Fix interaction with STUN. Now the DTLS analyzer cleanly skips all STUN messages. (Johanna Amann) * Fix the way that child analyzers are added. (Johanna Amann) 2.4-563 | 2016-05-17 16:25:21 -0700 * Fix duplication of new_connection_contents event. Addresses BIT-1602 (Johanna Amann) * SMTP: Support SSL upgrade via X-ANONYMOUSTLS This seems to be a non-standardized microsoft extension that, besides having a different name, works pretty much the same as StartTLS. We just treat it as such. (Johanna Amann) * Fixing control framework's net_stats and peer_status commands. For the latter, this removes most of the values returned, as we don't have access to them anymore. (Robin Sommer) 2.4-555 | 2016-05-16 20:10:15 -0700 * Fix failing plugin tests on OS X 10.11. (Daniel Thayer) * Fix failing test on Debian/FreeBSD. (Johanna Amann) 2.4-552 | 2016-05-12 08:04:33 -0700 * Fix a bug in receiving remote logs via broker. (Daniel Thayer) * Fix Bro and unit tests when broker is not enabled. (Daniel Thayer) * Added interpreter error for local event variables. (Jan Grashoefer) 2.4-544 | 2016-05-07 12:19:07 -0700 * Switching all use of gmtime and localtime to use reentrant variants. (Seth Hall) 2.4-541 | 2016-05-06 17:58:45 -0700 * A set of new built-in function for gathering execution statistics: get_net_stats(), get_conn_stats(), get_proc_stats(), get_event_stats(), get_reassembler_stats(), get_dns_stats(), get_timer_stats(), get_file_analysis_stats(), get_thread_stats(), get_gap_stats(), get_matcher_stats(). net_stats() resource_usage() have been superseded by these. (Seth Hall) * New policy script misc/stats.bro that records Bro execution statistics in a standard Bro log file. (Seth Hall) * A series of documentation improvements. (Daniel Thayer) * Rudimentary XMPP StartTLS analyzer. It parses certificates out of XMPP connections using StartTLS. It aborts processing if StartTLS is not found. (Johanna Amann) 2.4-507 | 2016-05-03 11:18:16 -0700 * Fix incorrect type tags in Bro broker source code. These are just used for error reporting. (Daniel Thayer) * Update docs and tests of the fmt() function. (Daniel Thayer) 2.4-500 | 2016-05-03 11:16:50 -0700 * Updating submodule(s). 2.4-498 | 2016-04-28 11:34:52 -0700 * Rename Broker::print to Broker::send_print and Broker::event to Broker::send_event to avoid using reserved keywords as function names. (Daniel Thayer) * Add script wrapper functions for Broker BIFs. This faciliates documenting them through Broxygen. (Daniel Thayer) * Extend, update, and clean up Broker tests. (Daniel Thayer) * Intel: Allow to provide uid/fuid instead of conn/file. (Johanna Amann) * Provide file IDs for hostname matches in certificates. (Johanna Amann) * Rudimentary IMAP StartTLS analyzer. It parses certificates out of IMAP connections using StartTLS. It aborts processing if StartTLS is not found. (Johanna Amann) 2.4-478 | 2016-04-28 09:56:24 * Fix parsing of x509 pre-y2k dates. (Johanna Amann) * Fix small error in bif documentation. (Johanna Amann) * Fix unknown data link type error message. (Vitaly Repin) * Correcting spelling errors. (Jeannette Dopheide) * Minor cleanup in ARP analyzer. (Johanna Amann) * Fix parsing of pre-y2k dates in X509 certificates. (Johanna Amann) * Fix small error in get_current_packet documentation. (Johanna Amann) 2.4-471 | 2016-04-25 15:37:15 -0700 * Add DNS tests for huge TLLs and CAA. (Johanna Amann) * Add DNS "CAA" RR type and event. (Mark Taylor) * Fix DNS response parsing: TTLs are unsigned. (Mark Taylor) 2.4-466 | 2016-04-22 16:25:33 -0700 * Rename BrokerStore and BrokerComm to Broker. Also split broker main.bro into two scripts. (Daniel Thayer) * Add get_current_packet_header bif. (Jan Grashoefer) 2.4-457 | 2016-04-22 08:36:27 -0700 * Fix Intel framework not checking the CERT_HASH indicator type. (Johanna Amann) 2.4-454 | 2016-04-14 10:06:58 -0400 * Additional mime types for file identification and a few fixes. (Seth Hall) New file mime types: - .ini files - MS Registry policy files - MS Registry files - MS Registry format files (e.g. DESKTOP.DAT) - MS Outlook PST files - Apple AFPInfo files Mime type fixes: - MP3 files with ID3 tags. - JSON and XML matchers were extended * Avoid a macro name conflict on FreeBSD. (Seth Hall, Daniel Thayer) 2.4-452 | 2016-04-13 01:15:20 -0400 * Add a simple file entropy analyzer. (Seth Hall) * Analyzer and bro script for RFB/VNC protocol (Martin van Hensbergen) This analyzer parses the Remote Frame Buffer protocol, usually referred to as the 'VNC protocol'. It supports several dialects (3.3, 3.7, 3.8) and also handles the Apple Remote Desktop variant. It will log such facts as client/server versions, authentication method used, authentication result, height, width and name of the shared screen. 2.4-430 | 2016-04-07 13:36:36 -0700 * Fix regex literal in scripting documentation. (William Tom) 2.4-428 | 2016-04-07 13:33:08 -0700 * Confirm protocol in SNMP/SIP only if we saw a response SNMP/SIP packet. (Vlad Grigorescu) 2.4-424 | 2016-03-24 13:38:47 -0700 * Only load openflow/netcontrol if compiled with broker. (Johanna Amann) * Adding canonifier to test. (Robin Sommer) 2.4-422 | 2016-03-21 19:48:30 -0700 * Adapt to recent change in CAF CMake script. (Matthias Vallentin) * Deprecate --with-libcaf in favor of --with-caf, as already done in Broker. (Matthias Vallentin) 2.4-418 | 2016-03-21 12:22:15 -0700 * Add protocol confirmation to MySQL analyzer. (Vlad Grigorescu) * Check that there is only one of &read_expire, &write_expire, &create_expire. (Johanna Amann) * Fixed &read_expire for subnet-indexed tables, plus test case. (Jan Grashoefer) * Add filter_subnet_table() that works similar to matching_subnet() but returns a filtered view of the original set/table only containing the changed subnets. (Jan Grashoefer) * Fix bug in tablue values' tracking read operations. (Johanna Amann) * Update TLS constants and extensions from IANA. (Johanna Amann) 2.4-406 | 2016-03-11 14:27:47 -0800 * Add NetControl and OpenFlow frameworks. (Johanna Amann) 2.4-313 | 2016-03-08 07:47:57 -0800 * Remove old string functions in C++ code. This removes the functions: strcasecmp_n, strchr_n, and strrchr_n. (Johanna Amann) 2.4-307 | 2016-03-07 13:33:45 -0800 * Add "disable_analyzer_after_detection" and remove "skip_processing_after_detection". Addresses BIT-1545. (Aaron Eppert & Johanna Amann) * Add bad_HTTP_request_with_version weird (William Glodek) 2.4-299 | 2016-03-04 12:51:55 -0800 * More detailed installation instructions for FreeBSD 9.X. (Johanna Amann) * Update CMake OpenSSL checks. (Johanna Amann) * "SUBSCRIBE" is a valid SIP. message per RFC 3265. Addresses BIT-1529. (Johanna Amann) * Update documentation for connection log's RSTR. Addresses BIT-1535 (Johanna Amann) 2.4-284 | 2016-02-17 14:12:15 -0800 * Fix sometimes failing dump-events test. (Johanna Amann) 2.4-282 | 2016-02-13 10:48:21 -0800 * Add missing break in in StartTLS case of IRC analyzer. Found by Aaron Eppert. (Johanna Amann) 2.4-280 | 2016-02-13 10:40:16 -0800 * Fix memory leaks in stats.cc and smb.cc. (Johanna Amann) 2.4-278 | 2016-02-12 18:53:35 -0800 * Better multi-space separator handline. (Mark Taylor & Johanna Amann) 2.4-276 | 2016-02-10 21:29:33 -0800 * Allow IRC commands to not have parameters. (Mark Taylor) 2.4-272 | 2016-02-08 14:27:58 -0800 * fix memory leaks in find_all() and IRC analyzer. (Dirk Leinenbach) 2.4-270 | 2016-02-08 13:00:57 -0800 * Removed duplicate parameter for IRC "QUIT" event handler. (Mark Taylor) 2.4-267 | 2016-02-01 12:38:32 -0800 * Add testcase for CVE-2015-3194. (Johanna Amann) * Fix portability issue with use of mktemp. (Daniel Thayer) 2.4-260 | 2016-01-28 08:05:27 -0800 * Correct irc_privmsg_message event handling bug. (Mark Taylor) * Update copyright year for Sphinx. (Johanna Amann) 2.4-253 | 2016-01-20 17:41:20 -0800 * Support of RadioTap encapsulation for 802.11 (Seth Hall) Radiotap support should be fully functional with Radiotap packets that include IPv4 and IPv6. Other radiotap packets are silently ignored. 2.4-247 | 2016-01-19 10:19:48 -0800 * Fixing C++11 compiler warnings. (Seth Hall) * Updating plugin documentation building. (Johanna Amann) 2.4-238 | 2016-01-15 12:56:33 -0800 * Add HTTP version information to HTTP log file. (Aaron Eppert) * Add NOTIFY as a valid SIP message, per RFC 3265. (Aaron Eppert) * Improve HTTP parser's handling of requests that don't have a URI. (William Glodek/Robin Sommer) * Fix crash when deleting non existing record member. Addresses BIT-1519. (Johanna Amann) 2.4-228 | 2015-12-19 13:40:09 -0800 * Updating BroControl submodule. 2.4-227 | 2015-12-18 17:47:24 -0800 * Update host name in windows-version-detection.bro. (Aaron Eppert) * Update installation instructions to mention OpenSSL dependency for newer OS X version. (Johanna Amann) * Change a stale bro-ids.org to bro.org. (Johanna Amann) * StartTLS support for IRC. (Johanna Amann) * Adding usage guard to canonifier script. (Robin Sommer) 2.4-217 | 2015-12-04 16:50:46 -0800 * SIP scripts code cleanup. (Seth Hall) - Daniel Guerra pointed out a type issue for SIP request and response code length fields which is now corrected. - Some redundant code was removed. - if/else tree modified to use switch instead. 2.4-214 | 2015-12-04 16:40:15 -0800 * Delaying BinPAC initializaton until afte plugins have been activated. (Robin Sommer) 2.4-213 | 2015-12-04 15:25:48 -0800 * Use better data structure for storing BPF filters. (Robin Sommer) 2.4-211 | 2015-11-17 13:28:29 -0800 * Making cluster reconnect timeout configurable. (Robin Sommer) * Bugfix for child process' communication loop. (Robin Sommer) 2.4-209 | 2015-11-16 07:31:22 -0800 * Updating submodule(s). 2.4-207 | 2015-11-10 13:34:42 -0800 * Fix to compile with OpenSSL that has SSLv3 disalbed. (Christoph Pietsch) * Fix potential race condition when logging VLAN info to conn.log. (Daniel Thayer) 2.4-201 | 2015-10-27 16:11:15 -0700 * Updating NEWS. (Robin Sommer) 2.4-200 | 2015-10-26 16:57:39 -0700 * Adding missing file. (Robin Sommer) 2.4-199 | 2015-10-26 16:51:47 -0700 * Fix problem with the JSON Serialization code. (Aaron Eppert) 2.4-188 | 2015-10-26 14:11:21 -0700 * Extending rexmit_inconsistency() event to receive an additional parameter with the packet's TCP flags, if available. (Robin Sommer) 2.4-187 | 2015-10-26 13:43:32 -0700 * Updating NEWS for new plugins. (Robin Sommer) 2.4-186 | 2015-10-23 15:07:06 -0700 * Removing pcap options for AF_PACKET support. Addresses BIT-1363. (Robin Sommer) * Correct a typo in controller.bro documentation. (Daniel Thayer) * Extend SSL DPD signature to allow alert before server_hello. (Johanna Amann) * Make join_string_vec work with vectors containing empty elements. (Johanna Amann) * Fix support for HTTP CONNECT when server adds headers to response. (Eric Karasuda). * Load static CA list for validation tests too. (Johanna Amann) * Remove cluster certificate validation script. (Johanna Amann) * Fix a bug in diff-remove-x509-names canonifier. (Daniel Thayer) * Fix test canonifiers in scripts/policy/protocols/ssl. (Daniel Thayer) 2.4-169 | 2015-10-01 17:21:21 -0700 * Fixed parsing of V_ASN1_GENERALIZEDTIME timestamps in x509 certificates. (Yun Zheng Hu) * Improve X509 end-of-string-check code. (Johanna Amann) * Refactor X509 generalizedtime support and test. (Johanna Amann) * Fix case of offset=-1 (EOF) for RAW reader. Addresses BIT-1479. (Johanna Amann) * Improve a number of test canonifiers. (Daniel Thayer) * Remove unnecessary use of TEST_DIFF_CANONIFIER. (Daniel Thayer) * Fixed some test canonifiers to read only from stdin * Remove unused test canonifier scripts. (Daniel Thayer) * A potpourri of updates and improvements across the documentation. (Daniel Thayer) * Add configure option to disable Broker Python bindings. Also improve the configure summary output to more clearly show whether or not Broker Python bindings will be built. (Daniel Thayer) 2.4-131 | 2015-09-11 12:16:39 -0700 * Add README.rst symlink. Addresses BIT-1413 (Vlad Grigorescu) 2.4-129 | 2015-09-11 11:56:04 -0700 * hash-all-files.bro depends on base/files/hash (Richard van den Berg) * Make dns_max_queries redef-able, and bump default to 25. Addresses BIT-1460 (Vlad Grigorescu) 2.4-125 | 2015-09-03 20:10:36 -0700 * Move SIP analyzer to flowunit instead of datagram Addresses BIT-1458 (Vlad Grigorescu) 2.4-122 | 2015-08-31 14:39:41 -0700 * Add a number of out-of-bound checks to layer 2 code. Addresses BIT-1463 (Johanna Amann) * Fix error in 2.4 release notes regarding SSH events. (Robin Sommer) 2.4-118 | 2015-08-31 10:55:29 -0700 * Fix FreeBSD build errors (Johanna Amann) 2.4-117 | 2015-08-30 22:16:24 -0700 * Fix initialization of a pointer in RDP analyzer. (Daniel Thayer/Robin Sommer) 2.4-115 | 2015-08-30 21:57:35 -0700 * Enable Bro to leverage packet fanout mode on Linux. (Kris Nielander). ## Toggle whether to do packet fanout (Linux-only). const Pcap::packet_fanout_enable = F &redef; ## If packet fanout is enabled, the id to sue for it. This should be shared amongst ## worker processes processing the same socket. const Pcap::packet_fanout_id = 0 &redef; ## If packet fanout is enabled, whether packets are to be defragmented before ## fanout is applied. const Pcap::packet_fanout_defrag = T &redef; * Allow libpcap buffer size to be set via configuration. (Kris Nielander) ## Number of Mbytes to provide as buffer space when capturing from live ## interfaces. const Pcap::bufsize = 128 &redef; * Move the pcap-related script-level identifiers into the new Pcap namespace. (Robin Sommer) snaplen -> Pcap::snaplen precompile_pcap_filter() -> Pcap::precompile_pcap_filter() install_pcap_filter() -> Pcap::install_pcap_filter() pcap_error() -> Pcap::pcap_error() 2.4-108 | 2015-08-30 20:14:31 -0700 * Update Base64 decoding. (Jan Grashoefer) - A new built-in function, decode_base64_conn() for Base64 decoding. It works like decode_base64() but receives an additional connection argument that will be used for reporting decoding errors into weird.log (instead of reporter.log). - FTP, POP3, and HTTP analyzers now likewise log Base64 decoding errors to weird.log. - The built-in functions decode_base64_custom() and encode_base64_custom() are now deprecated. Their functionality is provided directly by decode_base64() and encode_base64(), which take an optional parameter to change the Base64 alphabet. * Fix potential crash if TCP header was captured incompletely. (Robin Sommer) 2.4-103 | 2015-08-29 10:51:55 -0700 * Make ASN.1 date/time parsing more robust. (Johanna Amann) * Be more permissive on what characters we accept as an unquoted multipart boundary. Addresses BIT-1459. (Johanna Amann) 2.4-99 | 2015-08-25 07:56:57 -0700 * Add ``Q`` and update ``I`` documentation for connection history field. Addresses BIT-1466. (Vlad Grigorescu) 2.4-96 | 2015-08-21 17:37:56 -0700 * Update SIP analyzer. (balintm) - Allows space on both sides of ':'. - Require CR/LF after request/reply line. 2.4-94 | 2015-08-21 17:31:32 -0700 * Add file type detection support for video/MP2T. (Mike Freemon) 2.4-93 | 2015-08-21 17:23:39 -0700 * Make plugin install honor DESTDIR= convention. (Jeff Barber) 2.4-89 | 2015-08-18 07:53:36 -0700 * Fix diff-canonifier-external to use basename of input file. (Daniel Thayer) 2.4-87 | 2015-08-14 08:34:41 -0700 * Removing the yielding_teredo_decapsulation option. (Robin Sommer) 2.4-86 | 2015-08-12 17:02:24 -0700 * Make Teredo DPD signature more precise. (Martina Balint) 2.4-84 | 2015-08-10 14:44:39 -0700 * Add hook 'HookSetupAnalyzerTree' to allow plugins access to a connection's initial analyzer tree for customization. (James Swaro) * Plugins now look for a file "__preload__.bro" in the top-level script directory. If found, they load it first, before any scripts defining BiF elements. This can be used to define types that the BiFs already depend on (like a custom type for an event argument). (Robin Sommer) 2.4-81 | 2015-08-08 07:38:42 -0700 * Fix a test that is failing very frequently. (Daniel Thayer) 2.4-78 | 2015-08-06 22:25:19 -0400 * Remove build dependency on Perl (now requiring Python instad). (Daniel Thayer) * CID 1314754: Fixing unreachable code in RSH analyzer. (Robin Sommer) * CID 1312752: Add comment to mark 'case' fallthrough as ok. (Robin Sommer) * CID 1312751: Removing redundant assignment. (Robin Sommer) 2.4-73 | 2015-07-31 08:53:49 -0700 * BIT-1429: SMTP logs now include CC: addresses. (Albert Zaharovits) 2.4-70 | 2015-07-30 07:23:44 -0700 * Updated detection of Flash and AdobeAIR. (Jan Grashoefer) * Adding tests for Flash version parsing and browser plugin detection. (Robin Sommer) 2.4-63 | 2015-07-28 12:26:37 -0700 * Updating submodule(s). 2.4-61 | 2015-07-28 12:13:39 -0700 * Renaming config.h to bro-config.h. (Robin Sommer) 2.4-58 | 2015-07-24 15:06:07 -0700 * Add script protocols/conn/vlan-logging.bro to record VLAN data in conn.log. (Aaron Brown) * Add field "vlan" and "inner_vlan" to connection record. (Aaron Brown) * Save the inner vlan in the Packet object for Q-in-Q setups. (Aaron Brown) * Increasing plugin API version for recent packet source changes. (Robin Sommer) * Slightly earlier protocol confirmation for POP3. (Johanna Amann) 2.4-46 | 2015-07-22 10:56:40 -0500 * Fix broker python bindings install location to track --prefix. (Jon Siwek) 2.4-45 | 2015-07-21 15:19:43 -0700 * Enabling Broker by default. This means CAF is now a required dependency, altjough for now at least, there's still a switch --disable-broker to turn it off. * Requiring a C++11 compiler, and turning on C++11 support. (Robin Sommer) * Tweaking the listing of hooks in "bro -NN" for consistency. (Robin Sommer) 2.4-41 | 2015-07-21 08:35:17 -0700 * Fixing compiler warning. (Robin Sommer) * Updates to IANA TLS registry. (Johanna Amann) 2.4-38 | 2015-07-20 15:30:35 -0700 * Refactor code to use a common Packet type throught. (Jeff Barber/Robin Sommer) * Extend parsing layer 2 and keeping track of layer 3 protoco. (Jeff Barber) * Add a raw_packet() event that generated for all packets and include layer 2 information. (Jeff Barber) 2.4-27 | 2015-07-15 13:31:49 -0700 * Fix race condition in intel test. (Johanna Amann) 2.4-24 | 2015-07-14 08:04:11 -0700 * Correct Perl package name on FreeBSD in documentation.(Justin Azoff) * Adding an environment variable to BTest configuration for external scripts. (Robin Sommer) 2.4-20 | 2015-07-03 10:40:21 -0700 * Adding a weird for when truncated packets lead TCP reassembly to ignore content. (Robin Sommer) 2.4-19 | 2015-07-03 09:04:54 -0700 * A set of tests exercising IP defragmentation and TCP reassembly. (Robin Sommer) 2.4-17 | 2015-06-28 13:02:41 -0700 * BIT-1314: Add detection for Quantum Insert attacks. The TCP reassembler can now keep a history of old TCP segments using the tcp_max_old_segments option. An overlapping segment with different data will then generate an rexmit_inconsistency event. The default for tcp_max_old_segments is zero, which disabled any additional buffering. (Yun Zheng Hu/Robin Sommer) 2.4-14 | 2015-06-28 12:30:12 -0700 * BIT-1400: Allow '<' and '>' in MIME multipart boundaries. The spec doesn't actually seem to permit these, but they seem to occur in the wild. (Jon Siwek) 2.4-12 | 2015-06-28 12:21:11 -0700 * BIT-1399: Trying to decompress deflated HTTP content even when zlib headers are missing. (Seth Hall) 2.4-10 | 2015-06-25 07:11:17 -0700 * Correct a name used in a header identifier (Justin Azoff) 2.4-8 | 2015-06-24 07:50:50 -0700 * Restore the --load-seeds cmd-line option and enable the short options -G/-H for --load-seeds/--save-seeds. (Daniel Thayer) 2.4-6 | 2015-06-19 16:26:40 -0700 * Generate protocol confirmations for Modbus, making it appear as a confirmed service in conn.log. (Seth Hall) * Put command line options in alphabetical order. (Daniel Thayer) * Removing dead code for no longer supported -G switch. (Robin Sommer) (Robin Sommer) 2.4 | 2015-06-09 07:30:53 -0700 * Release 2.4. * Fixing tiny thing in NEWS. (Robin Sommer) 2.4-beta-42 | 2015-06-08 09:41:39 -0700 * Fix reporter errors with GridFTP traffic. (Robin Sommer) 2.4-beta-40 | 2015-06-06 08:20:52 -0700 * PE Analyzer: Change how we calculate the rva_table size. (Vlad Grigorescu) 2.4-beta-39 | 2015-06-05 09:09:44 -0500 * Fix a unit test to check for Broker requirement. (Jon Siwek) 2.4-beta-38 | 2015-06-04 14:48:37 -0700 * Test for Broker termination. (Robin Sommer) 2.4-beta-37 | 2015-06-04 07:53:52 -0700 * BIT-1408: Improve I/O loop and Broker IOSource. (Jon Siwek) 2.4-beta-34 | 2015-06-02 10:37:22 -0700 * Add signature support for F4M files. (Seth Hall) 2.4-beta-32 | 2015-06-02 09:43:31 -0700 * A larger set of documentation updates, fixes, and extentions. (Daniel Thayer) 2.4-beta-14 | 2015-06-02 09:16:44 -0700 * Add memleak btest for attachments over SMTP. (Vlad Grigorescu) * BIT-1410: Fix flipped tx_hosts and rx_hosts in files.log. Reported by Ali Hadi. (Vlad Grigorescu) * Updating the Mozilla root certs. (Seth Hall) * Updates for the urls.bro script. Fixes BIT-1404. (Seth Hall) 2.4-beta-6 | 2015-05-28 13:20:44 -0700 * Updating submodule(s). 2.4-beta-2 | 2015-05-26 08:58:37 -0700 * Fix segfault when DNS is not available. Addresses BIT-1387. (Frank Meier and Robin Sommer) 2.4-beta | 2015-05-07 21:55:31 -0700 * Release 2.4-beta. * Update local-compat.test (Johanna Amann) 2.3-913 | 2015-05-06 09:58:00 -0700 * Add /sbin to PATH in btest.cfg and remove duplicate default_path. (Daniel Thayer) 2.3-911 | 2015-05-04 09:58:09 -0700 * Update usage output and list of command line options. (Daniel Thayer) * Fix to ssh/geo-data.bro for unset directions. (Vlad Grigorescu) * Improve SIP logging and remove reporter messages. (Seth Hall) 2.3-905 | 2015-04-29 17:01:30 -0700 * Improve SIP logging and remove reporter messages. (Seth Hall) 2.3-903 | 2015-04-27 17:27:59 -0700 * BIT-1350: Improve record coercion type checking. (Jon Siwek) 2.3-901 | 2015-04-27 17:25:27 -0700 * BIT-1384: Remove -O (optimize scripts) command-line option, which hadn't been working for a while already. (Jon Siwek) 2.3-899 | 2015-04-27 17:22:42 -0700 * Fix the -J/--set-seed cmd-line option. (Daniel Thayer) * Remove unused -l, -L, and -Z cmd-line options. (Daniel Thayer) 2.3-892 | 2015-04-27 08:22:22 -0700 * Fix typos in the Broker BIF documentation. (Daniel Thayer) * Update installation instructions and remove outdated references. (Johanna Amann) * Easier support for systems with tcmalloc_minimal installed. (Seth Hall) 2.3-884 | 2015-04-23 12:30:15 -0500 * Fix some outdated documentation unit tests. (Jon Siwek) 2.3-883 | 2015-04-23 07:10:36 -0700 * Fix -N option to work with builtin plugins as well. (Robin Sommer) 2.3-882 | 2015-04-23 06:59:40 -0700 * Add missing .pac dependencies for some binpac analyzer targets. (Jon Siwek) 2.3-879 | 2015-04-22 10:38:07 -0500 * Fix compile errors. (Jon Siwek) 2.3-878 | 2015-04-22 08:21:23 -0700 * Fix another compiler warning in DTLS. (Johanna Amann) 2.3-877 | 2015-04-21 20:14:16 -0700 * Adding missing include. (Robin Sommer) 2.3-876 | 2015-04-21 16:40:10 -0700 * Attempt at fixing a potential std::length_error exception in RDP analyzer. Addresses BIT-1337. (Robin Sommer) * Fixing compile problem caused by overeager factorization. (Robin Sommer) 2.3-874 | 2015-04-21 16:09:20 -0700 * Change details of escaping when logging/printing. (Seth Hall/Robin Sommer) - Log files now escape non-printable characters consistently as "\xXX'. Furthermore, backslashes are escaped as "\\", making the representation fully reversible. - When escaping via script-level functions (escape_string, clean), we likewise now escape consistently with "\xXX" and "\\". - There's no "alternative" output style anymore, i.e., fmt() '%A' qualifier is gone. Addresses BIT-1333. * Remove several BroString escaping methods that are no longer useful. (Seth Hall) 2.3-864 | 2015-04-21 15:24:02 -0700 * A SIP protocol analyzer. (Vlad Grigorescu) Activity gets logged into sip.log. It generates the following events: event sip_request(c: connection, method: string, original_URI: string, version: string); event sip_reply(c: connection, version: string, code: count, reason: string); event sip_header(c: connection, is_orig: bool, name: string, value: string); event sip_all_headers(c: connection, is_orig: bool, hlist: mime_header_list); event sip_begin_entity(c: connection, is_orig: bool); event sip_end_entity(c: connection, is_orig: bool); The analyzer support SIP over UDP currently. * BIT-1343: Factor common ASN.1 code from RDP, SNMP, and Kerberos analyzers. (Jon Siwek/Robin Sommer) 2.3-838 | 2015-04-21 13:40:12 -0700 * BIT-1373: Fix vector index assignment reference count bug. (Jon Siwek) 2.3-836 | 2015-04-21 13:37:31 -0700 * Fix SSH direction field being unset. Addresses BIT-1365. (Vlad Grigorescu) 2.3-835 | 2015-04-21 16:36:00 -0500 * Clarify Broker examples. (Jon Siwek) 2.3-833 | 2015-04-21 12:38:32 -0700 * A Kerberos protocol analyzer. (Vlad Grigorescu) Activity gets logged into kerberos.log. It generates the following events: event krb_as_request(c: connection, msg: KRB::KDC_Request); event krb_as_response(c: connection, msg: KRB::KDC_Response); event krb_tgs_request(c: connection, msg: KRB::KDC_Request); event krb_tgs_response(c: connection, msg: KRB::KDC_Response); event krb_ap_request(c: connection, ticket: KRB::Ticket, opts: KRB::AP_Options); event krb_priv(c: connection, is_orig: bool); event krb_safe(c: connection, is_orig: bool, msg: KRB::SAFE_Msg); event krb_cred(c: connection, is_orig: bool, tickets: KRB::Ticket_Vector); event krb_error(c: connection, msg: KRB::Error_Msg); 2.3-793 | 2015-04-20 20:51:00 -0700 * Add decoding of PROXY-AUTHORIZATION header to HTTP analyze, treating it the same as AUTHORIZATION. (Josh Liburdi) * Remove deprecated fields "hot" and "addl" from the connection record. Remove the functions append_addl() and append_addl_marker(). (Robin Sommer) * Removing the NetFlow analyzer, which hasn't been used anymore since then corresponding command-line option went away. (Robin Sommer) 2.3-787 | 2015-04-20 19:15:23 -0700 * A file analyzer for Portable Executables. (Vlad Grigorescu/Seth Hall). Activity gets logged into pe.log. It generates the following events: event pe_dos_header(f: fa_file, h: PE::DOSHeader); event pe_dos_code(f: fa_file, code: string); event pe_file_header(f: fa_file, h: PE::FileHeader); event pe_optional_header(f: fa_file, h: PE::OptionalHeader); event pe_section_header(f: fa_file, h: PE::SectionHeader); 2.3-741 | 2015-04-20 13:12:39 -0700 * API changes to file analysis mime type detection. Removed "file_mime_type" and "file_mime_types" event, replacing them with a new event called "file_metadata_inferred". Addresses BIT-1368. (Jon Siwek) * A large series of improvements for file type identification. This inludes a many signature updates (new types, cleanup, performance improvments) and splitting out signatures into subfiles. (Seth Hall) * Fix an issue with files having gaps before the bof_buffer is filled, which could lead to file type identification not working correctly. (Seth Hall) * Fix an issue with packet loss in HTTP file reporting for file type identification wasn't working correctly zero-length bodies. (Seth Hall) * X.509 certificates are now populating files.log with the mime type application/pkix-cert. (Seth Hall) * Normalized some FILE_ANALYSIS debug messages. (Seth Hall) 2.3-725 | 2015-04-20 12:54:54 -0700 * Updating submodule(s). 2.3-724 | 2015-04-20 14:11:02 -0500 * Fix uninitialized field in raw input reader. (Jon Siwek) 2.3-722 | 2015-04-20 12:59:03 -0500 * Remove unneeded documentation cross-referencing. (Jon Siwek) 2.3-721 | 2015-04-20 12:47:05 -0500 * BIT-1380: Improve Broxygen output of &default expressions. (Jon Siwek) 2.3-720 | 2015-04-17 14:18:26 -0700 * Updating NEWS. 2.3-716 | 2015-04-17 13:06:37 -0700 * Add seeking functionality to raw reader. One can now add an option "offset" to the config map. Positive offsets are interpreted to be from the beginning of the file, negative from the end of the file (-1 is end of file). Only works for raw reader in streaming or manual mode. Does not work with executables. Addresses BIT-985. (Johanna Amann) * Allow setting packet and byte thresholds for connections. (Johanna Amann) This extends the ConnSize analyzer to be able to raise events when each direction of a connection crosses a certain amount of bytes or packets. Thresholds are set using: - set_conn_bytes_threshold(c$id, [num-bytes], [direction]); - set_conn_packets_threshold(c$id, [num-packets], [direction]); They raise the events, respectively: - event conn_bytes_threshold_crossed(c: connection, threshold: count, is_orig: bool) - event conn_packets_threshold_crossed(c: connection, threshold: count, is_orig: bool) Current thresholds can be examined using get_conn_bytes_threshold() and get_conn_packets_threshold(). Only one threshold can be set per connection. * Add high-level API for packet/bytes thresholding in base/protocols/conn/thresholds.bro that holds lists of thresholds and raises an event for each threshold exactly once. (Johanna Amann) * Fix a bug where child packet analyzers of the TCP analyzer where not found using FindChild. * Update GridFTP analyzer to use connection thresholding instead of polling. (Johanna Amann) 2.3-709 | 2015-04-17 12:37:32 -0700 * Fix addressing the dreaded "internal error: unknown msg type 115 in Poll()". (Jon Siwek) This patch removes the error handling code for overload conditions in the main process that could cause trouble down the road. The "chunked_io_buffer_soft_cap" script variable can now tune when the client process begins shutting down peer connections, and the default setting is now double what it used to be. Addresses BIT-1376. 2.3-707 | 2015-04-17 10:57:59 -0500 * Add more info about Broker to NEWS. (Jon Siwek) 2.3-705 | 2015-04-16 08:16:45 -0700 * Update Mozilla CA list. (Johanna Amann) * Update tests to have them keep using older certificates where appropiate. (Johanna Amann) 2.3-699 | 2015-04-16 09:51:58 -0500 * Fix the to_count function to use strtoull versus strtoll. (Jon Siwek) 2.3-697 | 2015-04-15 09:51:15 -0700 * Removing error check verifying that an ASCII writer has been properly finished. Instead of aborting, we now just clean up in that case and proceed. Addresses BIT-1331. (Robin Sommer) 2.3-696 | 2015-04-14 15:56:36 -0700 * Update sqlite to 3.8.9 2.3-695 | 2015-04-13 10:34:42 -0500 * Fix iterator invalidation in broker::Manager dtor. (Jon Siwek) * Add paragraph to plugin documentation. (Robin Sommer) 2.3-693 | 2015-04-11 10:56:31 -0700 * BIT-1367: improve coercion of anonymous records in set constructor. (Jon Siwek) * Allow to specify ports for sftp log rotator. (Johanna Amann) 2.3-690 | 2015-04-10 21:51:10 -0700 * Make sure to always delete the remote serializer. Addresses BIT-1306 and probably also BIT-1356. (Robin Sommer) * Cleaning up --help. -D and -Y/y were still listed, even though they had no effect anymore. Removing some dead code along with -D. Addresses BIT-1372. (Robin Sommer) 2.3-688 | 2015-04-10 08:10:44 -0700 * Update SQLite to 3.8.8.3. 2.3-687 | 2015-04-10 07:32:52 -0700 * Remove stale signature benchmarking code (-L command-line option). (Jon Siwek) * BIT-844: fix UDP payload signatures to match packet-wise. (Jon Siwek) 2.3-682 | 2015-04-09 12:07:00 -0700 * Fixing input readers' component type. (Robin Sommer) * Tiny spelling correction. (Seth Hall) 2.3-680 | 2015-04-06 16:02:43 -0500 * BIT-1371: remove CMake version check from binary package scripts. (Jon Siwek) 2.3-679 | 2015-04-06 10:16:36 -0500 * Increase some unit test timeouts. (Jon Siwek) * Fix Coverity warning in RDP analyzer. (Jon Siwek) 2.3-676 | 2015-04-02 10:10:39 -0500 * BIT-1366: improve checksum offloading warning. (Frank Meier, Jon Siwek) 2.3-675 | 2015-03-30 17:05:05 -0500 * Add an RDP analyzer. (Josh Liburdi, Seth Hall, Johanna Amann) 2.3-640 | 2015-03-30 13:51:51 -0500 * BIT-1359: Limit maximum number of DTLS fragments to 30. (Johanna Amann) 2.3-637 | 2015-03-30 12:02:07 -0500 * Increase timeout duration in some broker tests. (Jon Siwek) 2.3-636 | 2015-03-30 11:26:32 -0500 * Updates related to SSH analysis. (Jon Siwek) - Some scripts used wrong SSH module/namespace scoping on events. - Fix outdated notice documentation related to SSH password guessing. - Add a unit test for SSH pasword guessing notice. 2.3-635 | 2015-03-30 11:02:45 -0500 * Fix outdated documentation unit tests. (Jon Siwek) 2.3-634 | 2015-03-30 10:22:45 -0500 * Add a canonifier to a unit test's output. (Jon Siwek) 2.3-633 | 2015-03-25 18:32:59 -0700 * Log::write in signature framework was missing timestamp. (Andrew Benson/Michel Laterman) 2.3-631 | 2015-03-25 11:03:12 -0700 * New SSH analyzer. (Vlad Grigorescu) 2.3-600 | 2015-03-25 10:23:46 -0700 * Add defensive checks in code to calculate log rotation intervals. (Pete Nelson). 2.3-597 | 2015-03-23 12:50:04 -0700 * DTLS analyzer. (Johanna Amann) * Implement correct parsing of TLS record fragmentation. (Johanna Amann) 2.3-582 | 2015-03-23 11:34:25 -0700 * BIT-1313: In debug builds, "bro -B " now supports "all" and "help" for "". "all" enables all debug streams. "help" prints a list of available debug streams. (John Donnelly/Robin Sommer). * BIT-1324: Allow logging filters to inherit default path from stream. This allows the path for the default filter to be specified explicitly through $path="..." when creating a stream. Adapted the existing Log::create_stream calls to explicitly specify a path value. (Jon Siwek) * BIT-1199: Change the way the input framework deals with values it cannot convert into BroVals, raising error messages instead of aborting execution. (Johanna Amann) * BIT-788: Use DNS QR field to better identify flow direction. (Jon Siwek) 2.3-572 | 2015-03-23 13:04:53 -0500 * BIT-1226: Fix an example in quickstart docs. (Jon siwek) 2.3-570 | 2015-03-23 09:51:20 -0500 * Correct a spelling error (Daniel Thayer) * Improvement to SSL analyzer failure mode. (Johanna Amann) 2.3-565 | 2015-03-20 16:27:41 -0500 * BIT-978: Improve documentation of 'for' loop iterator invalidation. (Jon Siwek) 2.3-564 | 2015-03-20 11:12:02 -0500 * BIT-725: Remove "unmatched_HTTP_reply" weird. (Jon Siwek) 2.3-562 | 2015-03-20 10:31:02 -0500 * BIT-1207: Add unit test to catch breaking changes to local.bro (Jon Siwek) * Fix failing sqlite leak test (Johanna Amann) 2.3-560 | 2015-03-19 13:17:39 -0500 * BIT-1255: Increase default values of "tcp_max_above_hole_without_any_acks" and "tcp_max_initial_window" from 4096 to 16384 bytes. (Jon Siwek) 2.3-559 | 2015-03-19 12:14:33 -0500 * BIT-849: turn SMTP reporter warnings into weirds, "smtp_nested_mail_transaction" and "smtp_unmatched_end_of_data". (Jon Siwek) 2.3-558 | 2015-03-18 22:50:55 -0400 * DNS: Log the type number for the DNS_RR_unknown_type weird. (Vlad Grigorescu) 2.3-555 | 2015-03-17 15:57:13 -0700 * Splitting test-all Makefile target into Bro tests and test-aux. (Robin Sommer) 2.3-554 | 2015-03-17 15:40:39 -0700 * Deprecate &rotate_interval, &rotate_size, &encrypt. Addresses BIT-1305. (Jon Siwek) 2.3-549 | 2015-03-17 09:12:18 -0700 * BIT-1077: Fix HTTP::log_server_header_names. Before, it just re-logged fields from the client side. (Jon Siwek) 2.3-547 | 2015-03-17 09:07:51 -0700 * Update certificate validation script to cache valid intermediate chains that it encounters on the wire and use those to try to validate chains that might be missing intermediate certificates. (Johanna Amann) 2.3-541 | 2015-03-13 15:44:08 -0500 * Make INSTALL a symlink to doc/install/install.rst (Jon siwek) * Fix Broxygen coverage. (Jon Siwek) 2.3-539 | 2015-03-13 14:19:27 -0500 * BIT-1335: Include timestamp in default extracted file names. And add a policy script to extract all files. (Jon Siwek) * BIT-1311: Identify GRE tunnels as Tunnel::GRE, not Tunnel::IP. (Jon Siwek) * BIT-1309: Add Connection class getter methods for flow labels. (Jon Siwek) 2.3-536 | 2015-03-12 16:16:24 -0500 * Fix Broker leak tests. (Jon Siwek) 2.3-534 | 2015-03-12 10:59:49 -0500 * Update NEWS file. (Jon Siwek) 2.3-533 | 2015-03-12 10:18:53 -0500 * Give broker python bindings default install path within --prefix. (Jon Siwek) 2.3-530 | 2015-03-10 13:22:39 -0500 * Fix broker data stores in absence of --enable-debug. (Jon Siwek) 2.3-529 | 2015-03-09 13:14:27 -0500 * Fix format specifier in SSL protocol violation. (Jon Siwek) 2.3-526 | 2015-03-06 12:48:49 -0600 * Fix build warnings, clarify broker requirements, update submodule. (Jon Siwek) * Rename comm/ directories to broker/ (Jon Siwek) * Rename broker-related namespaces. (Jon Siwek) * Improve remote logging via broker by only sending fields w/ &log. (Jon Siwek) * Disable a stream's remote logging via broker if it fails. (Jon Siwek) * Improve some broker communication unit tests. (Jon Siwek) 2.3-518 | 2015-03-04 13:13:50 -0800 * Add bytes_recvd to stats.log recording the number of bytes received, according to packet headers. (Mike Smiley) 2.3-516 | 2015-03-04 12:30:06 -0800 * Extract most specific Common Name from SSL certificates (Johanna Amann) * Send CN and SAN fields of SSL certificates to the Intel framework. (Johanna Amann) 2.3-511 | 2015-03-02 18:07:17 -0800 * Changes to plugin meta hooks for function calls. (Gilbert Clark) - Add frame argument. - Change return value to tuple unambigiously whether hook returned a result. 2.3-493 | 2015-03-02 17:17:32 -0800 * Extend the SSL weak-keys policy file to also alert when encountering SSL connections with old versions as well as unsafe cipher suites. (Johanna Amann) * Make the notice suppression handling of other SSL policy files a tad more robust. (Johanna Amann) 2.3-491 | 2015-03-02 17:12:56 -0800 * Updating docs for recent addition of local_resp. (Robin Sommer) 2.3-489 | 2015-03-02 15:29:30 -0800 * Integrate Broker, Bro's new communication library. (Jon Siwek) See aux/broker/README for more information on Broker, and doc/frameworks/comm.rst for the corresponding Bro script API. Broker support is by default off for now; it can be enabled at configure time with --enable-broker. It requires CAF (https://github.com/actor-framework/actor-framework); for now iot needs CAF's "develop" branch. Broker also requires a C++11 compiler. Broker will become a mandatory dependency in future Bro versions. * Add --enable-c++11 configure flag to compile Bro's source code in C++11 mode with a corresponding compiler. (Jon Siwek) 2.3-451 | 2015-02-24 16:37:08 -0800 * Updating submodule(s). 2.3-448 | 2015-02-23 16:58:10 -0800 * Updating NEWS. (Robin Sommer) 2.3-447 | 2015-02-23 16:28:30 -0800 * Fix potential crash in logging framework when deserializing WriterInfo from remote. where config is present. Testcase crashes on unpatched versions of Bro. (Aaron Eppert) * Fix wrong value test in WriterBackend. (Aaron Eppert) 2.3-442 | 2015-02-23 13:29:30 -0800 * Add a "local_resp" field to conn.log, along the lines of the existing "local_orig". (Mike Smiley) 2.3-440 | 2015-02-23 11:39:17 -0600 * Updating plugin docs to recent changes. (Robin Sommer) * Updating plugin tests to recent changes. (Robin Sommer) * Making plugin names case-insensitive for some internal comparisions. Makes plugin system more tolerant against spelling inconsistencies are hard to catch otherwise. (Robin Sommer) * Explicitly removing some old scripts on install that have moved into plugins to prevent them causing confusion. (Robin Sommer) * BIT-1312: Removing setting installation plugin path from bro-path-dev.sh. Also, adding to existing BRO_PLUGIN_PATH rather than replacing. (Robin Sommer) * Creating the installation directory for plugins at install time. (Robin Sommer) 2.3-427 | 2015-02-20 13:49:33 -0800 * Removing dependency on PCAP_NETMASK_UNKNOWN to compile with libpcap < 1.1.1. (Robin Sommer) 2.3-426 | 2015-02-20 12:45:51 -0800 * Add 'while' statement to Bro language. Really. (Jon Siwek) 2.3-424 | 2015-02-20 12:39:10 -0800 * Add the ability to remove surrounding braces from the JSON formatter. (Seth Hall) 2.3-419 | 2015-02-13 09:10:44 -0600 * BIT-1011: Update the SOCKS analyzer to support user/pass login. (Nicolas Retrain, Seth Hall, Jon Siwek) - Add a new field to socks.log: "password". - Two new events: "socks_login_userpass_request" and "socks_login_userpass_reply". - Two new weirds for unsupported SOCKS authentication method or version. - A new test for authenticated socks traffic. 2.3-416 | 2015-02-12 12:18:42 -0600 * Submodule update - newest sqlite version (Johanna Amann) * Fix use of deprecated gperftools headers. (Jon Siwek) 2.3-413 | 2015-02-08 18:23:05 -0800 * Fixing analyzer tag types for some Files::* functions. (Robin Sommer) * Changing load order for plugin scripts. (Robin Sommer) 2.3-411 | 2015-02-05 10:05:48 -0600 * Fix file analysis of files with total size below the bof_buffer size never delivering content to stream analyzers. (Seth Hall) * Add/fix log fields in x509 diff canonifier. (Jon Siwek) * "id" not defined for debug code when using -DPROFILE_BRO_FUNCTIONS (Mike Smiley) 2.3-406 | 2015-02-03 17:02:45 -0600 * Add x509 canonifier to a unit test. (Jon Siwek) 2.3-405 | 2015-02-02 11:14:24 -0600 * Fix memory leak in new split_string* functions. (Jon Siwek) 2.3-404 | 2015-01-30 14:23:27 -0800 * Update documentation (broken links, outdated tests). (Jon Siwek) * Deprecate split* family of BIFs. (Jon Siwek) These functions are now deprecated in favor of alternative versions that return a vector of strings rather than a table of strings. Deprecated functions: - split: use split_string instead. - split1: use split_string1 instead. - split_all: use split_string_all instead. - split_n: use split_string_n instead. - cat_string_array: see join_string_vec instead. - cat_string_array_n: see join_string_vec instead. - join_string_array: see join_string_vec instead. - sort_string_array: use sort instead instead. - find_ip_addresses: use extract_ip_addresses instead. Changed functions: - has_valid_octets: uses a string_vec parameter instead of string_array. Addresses BIT-924. * Add a new attribute: &deprecated. While scripts are parsed, a warning is raised for each usage of an identifier marked as &deprecated. This also works for BIFs. Addresses BIT-924, BIT-757. (Jon Siwek) 2.3-397 | 2015-01-27 10:13:10 -0600 * Handle guess_lexer exceptions in pygments reST directive (Jon Siwek) 2.3-396 | 2015-01-23 10:49:15 -0600 * DNP3: fix reachable assertion and buffer over-read/overflow. CVE number pending. (Travis Emmert, Jon Siwek) * Update binpac: Fix potential out-of-bounds memory reads in generated code. CVE-2014-9586. (John Villamil and Chris Rohlf - Yahoo Paranoids, Jon Siwek) * Fixing (harmless) Coverity warning. (Robin Sommer) 2.3-392 | 2015-01-15 09:44:15 -0800 * Small changes to EC curve names in a newer draft. (Johanna Amann) 2.3-390 | 2015-01-14 13:27:34 -0800 * Updating MySQL analyses. (Vlad Grigorescu) - Use a boolean success instead of a result string. - Change the affected_rows response detail string to a "rows" count. - Fix the state tracking to log incomplete command. * Extend DNP3 to support communication over UDP. (Hui Lin) * Fix a bug in DNP3 determining the length of an object in some cases. (Hui Lin) 2.3-376 | 2015-01-12 09:38:10 -0600 * Improve documentation for connection_established event. (Jon Siwek) 2.3-375 | 2015-01-08 13:10:09 -0600 * Increase minimum required CMake version to 2.8. (Jon Siwek) 2.3-374 | 2015-01-07 10:03:17 -0600 * Improve documentation of the Intelligence Framework. (Daniel Thayer) 2.3-371 | 2015-01-06 09:58:09 -0600 * Update/improve file mime type identification. (Seth Hall) - Change to the default BOF buffer size to 3000 (was 1024). - Reorganized MS signatures into a separate file. - Remove all of the x-c detections. Nearly all false positives. - Improve TAR detections, removing old, back up TAR detections. - Remove one of the x-elc detections that was too loose and caused many false positives. - Improved lots of the signatures and added new ones. (Seth Hall) * Add support for file reassembly in the file analysis framework (Seth Hall, Jon Siwek). - The reassembly behavior can be modified per-file by enabling or disabling the reassembler and/or modifying the size of the reassembly buffer. - Changed the file extraction analyzer to use stream-wise input to avoid issues with the chunk-wise approach not immediately triggering the file_new event due to mime-type detection delay. Before, early chunks frequently ended up lost. Extraction also will now explicitly NUL-fill gaps in the file instead of implicitly relying on pwrite to do it. 2.3-349 | 2015-01-05 15:21:13 -0600 * Fix race condition in unified2 file analyzer startup. (Jon siwek) 2.3-348 | 2014-12-31 09:19:34 -0800 * Changing Makefile's test-all to run test-all for broctl, which now executes trace-summary tests as well. (Robin Sommer) 2.3-345 | 2014-12-31 09:06:15 -0800 * Correct a typo in the Notice framework doc. (Daniel Thayer) 2.3-343 | 2014-12-12 12:43:46 -0800 * Fix PIA packet replay to deliver copy of IP header. This prevented one from writing a packet-wise analyzer that needs access to IP headers and can be attached to a connection via signature match. Addresses BIT-1298 (Jon Siwek) 2.3-338 | 2014-12-08 13:56:19 -0800 * Add man page for Bro. (Raúl Benencia) * Updating doc baselines. (Robin Sommer) 2.3-334 | 2014-12-03 14:22:07 -0800 * Fix compound assignment to require proper L-value. Addresses BIT-1295. (Jon Siwek) 2.3-332 | 2014-12-03 14:14:11 -0800 * Make using local IDs in @if directives an error. Addresses BIT-1296. (Jon Siwek) 2.3-330 | 2014-12-03 14:10:39 -0800 * Fix some "make doc" warnings and update some doc tests. (Daniel Thayer) 2.3-328 | 2014-12-02 08:13:10 -0500 * Update windows-version-detection.bro to add support for Windows 10. (Michal Purzynski) 2.3-326 | 2014-12-01 12:10:27 -0600 * BIFScanner: fix invalid characters in generated preprocessor macros. (Hilko Bengen) * BIT-1294: fix exec.bro from mutating Input::end_of_data event parameters. (Johanna Amann) * Add/invoke "distclean" for testing directories. (Raúl Benencia) * Delete prebuilt python bytecode files from git. (Jon Siwek) * Add Windows detection based on CryptoAPI HTTP traffic as a software framework policy script. (Vlad Grigorescu) 2.3-316 | 2014-11-25 17:35:06 -0800 * Make the SSL analyzer skip further processing once encountering situations which are very probably non-recoverable. (Johanna Amann) 2.3-313 | 2014-11-25 14:27:07 -0800 * Make SSL v2 protocol tests more strict. In its former state they triggered on http traffic over port 443 sometimes. Found by Michał Purzyński. (Johanna Amann) * Fix X509 analyzer to correctly return ECDSA as the key_type for ECDSA certs. Bug found by Michał Purzyński. (Johanna Amann) 2.3-310 | 2014-11-19 10:56:59 -0600 * Disable verbose bison output. (Jon Siwek) 2.3-309 | 2014-11-18 12:17:53 -0800 * New decompose_uri() function in base/utils/urls that splits a URI into its pieces. (Anthony Kasza). 2.3-305 | 2014-11-18 11:09:04 -0800 * Improve coercion of &default expressions. Addresses BIT-1288. (Jon Siwek) 2.3-303 | 2014-11-18 10:53:04 -0800 * For DH key exchanges, use p as the parameter for weak key exchanges. (Johanna Amann) 2.3-301 | 2014-11-11 13:47:27 -0800 * Add builtin function enum_to_int() that converts an enum into a integer. (Christian Struck) 2.3-297 | 2014-11-11 11:50:47 -0800 * Removing method from SSL analyzer that's no longer used. (Robin Sommer) 2.3-296 | 2014-11-11 11:42:38 -0800 * A new analyzer parsing the MySQL wire protocol. Activity gets logged into mysql.log. Supports protocol versions 9 and 10. (Vlad Grigorescu) 2.3-280 | 2014-11-05 09:46:33 -0500 * Add Windows detection based on CryptoAPI HTTP traffic as a software framework policy script. (Vlad Grigorescu) 2.3-278 | 2014-11-03 18:55:18 -0800 * Add new curves from draft-ietf-tls-negotiated-ff-dhe to SSL analysis. (Johanna Amann) 2.3-274 | 2014-10-31 17:45:25 -0700 * Adding call to new binpac::init() function. (Robin Sommer) 2.3-272 | 2014-10-31 16:29:42 -0700 * Fix segfault if when statement's RHS is unitialized. Addresses BIT-1176. (Jon Siwek) * Fix checking vector indices via "in". Addresses BIT-1280. (Jon Siwek) 2.3-268 | 2014-10-31 12:12:22 -0500 * BIT-1283: Fix crash when using &encrypt. (Jon Siwek) 2.3-267 | 2014-10-31 10:35:02 -0500 * BIT-1284: Allow arbitrary when statement timeout expressions (Jon Siwek) 2.3-266 | 2014-10-31 09:21:28 -0500 * BIT-1166: Add configure options to fine tune local state dirs used by BroControl. (Jon Siwek) 2.3-264 | 2014-10-30 13:25:57 -0500 * Fix some minor Coverity Scan complaints. (Jon Siwek) 2.3-263 | 2014-10-28 15:09:10 -0500 * Fix checking of fwrite return values (Johanna Amann) 2.3-260 | 2014-10-27 12:54:17 -0500 * Fix errors/warnings when compiling with -std=c++11 (Jon Siwek) 2.3-259 | 2014-10-27 10:04:04 -0500 * Documentation fixes. (Vicente Jimenez Aguilar and Stefano Azzalini) 2.3-256 | 2014-10-24 15:33:45 -0700 * Adding missing test baseline. (Robin Sommer) 2.3-255 | 2014-10-24 13:39:44 -0700 * Fixing unstable active-http test. (Robin Sommer) 2.3-254 | 2014-10-24 11:40:51 -0700 * Fix active-http.bro to deal reliably with empty server responses, which will now be passed back as empty files. (Christian Struck) 2.3-248 | 2014-10-23 14:20:59 -0700 * Change order in which a plugin's scripts are loaded at startup. (Robin Sommer) 2.3-247 | 2014-10-21 13:42:38 -0700 * Updates to the SSL analyzer. (Johanna Amann) * Mark everything below 2048 bit as a weak key. * Fix notice suppression. * Add information about server-chosen protocol to ssl.log, if provided by application_layer_next_protocol. * Add boolean flag to ssl.log signaling if a session was resumed. Remove the (usually not really that useful) session ID that the client sent. 2.3-240 | 2014-10-21 13:36:33 -0700 * Fix Coverity-reported issues in DNP3 analyzer. (Seth Hall) 2.3-238 | 2014-10-16 06:51:49 -0700 * Fix multipart HTTP/MIME entity file analysis so that (1) singular CR or LF characters in multipart body content are no longer converted to a full CRLF (thus corrupting the file) and (2) it also no longer considers the CRLF before the multipart boundary as part of the content. Addresses BIT-1235. (Jon Siwek) 2.3-235 | 2014-10-15 10:20:47 -0500 * BIT-1273: Add error message for bad enum declaration syntax. (Jon Siwek) 2.3-234 | 2014-10-14 14:42:09 -0500 * Documentation fixes. (Steve Smoot) 2.3-233 | 2014-10-09 16:00:27 -0500 * Change find-bro-logs unit test to follow symlinks. (Jon Siwek) * Add error checks and messages to a test script (Daniel Thayer) 2.3-230 | 2014-10-08 08:15:17 -0700 * Further baseline normalization for plugin test portability. (Robin Sommer) 2.3-229 | 2014-10-07 20:18:11 -0700 * Fix for test portability. (Robin Sommer) 2.3-228 | 2014-10-07 15:32:37 -0700 * Include plugin unit tests into the top-level btest configuration. (Robin Sommer) * Switching the prefix separator for packet source/dumper plugins once more, now to "::". Addresses BIT-1267. (Robin Sommer) * Fix for allowing a packet source/dumper plugin to support multiple prefixes with a colon. (Robin Sommer) 2.3-225 | 2014-10-07 15:13:35 -0700 * Updating plugin documentation. (Robin Sommer) 2.3-224 | 2014-10-07 14:32:17 -0700 * Improved the log file reference documentation. (Jeannette Dopheide and Daniel Thayer) * Improves shockwave flash file signatures. (Seth Hall) - This moves the signatures out of the libmagic imported signatures and into our own general.sig. - Expand the detection to LZMA compressed flash files. * Add new script language reference documentation on operators, statements, and directives. Also improved the documentation on types and attributes by splitting them into two docs, and providing more examples and adding a chart on the top of each page with links to each type and attribute for easier access to the information. (Daniel Thayer) * Split the types and attributes reference doc into two docs. (Daniel Thayer) 2.3-208 | 2014-10-03 09:38:52 -0500 * BIT-1268: Fix uninitialized router_list argument in dhcp_offer/dhcp_ack. (Jon Siwek) 2.3-207 | 2014-10-02 16:39:17 -0700 * Updating plugin docs. (Robin Sommer) * Fix packet sources being treated as idle when a packet is available. Addresses BIT-1266. (Jon Siwek) * Fix regression causing the main loop to spin more frequently. Addresses BIT-1266. (Jon Siwek) 2.3-203 | 2014-09-29 20:06:54 -0700 * Fix to use length parameter in DNP3 time conversion correctly now. (Robin Sommer) 2.3-202 | 2014-09-29 17:05:18 -0700 * New SSL extension type from IANA and a few other SSL const changes. (Johanna Amann) * Make unexpected pipe errors fatal as precaution. Addresses BIT-1260. (Jon Siwek) * Adding a function for DNP3 to translate the timestamp format. (Hui Lin) 2.3-197 | 2014-09-29 10:42:01 -0500 * Fix possible seg fault in TCP reassembler. (Jon Siwek) 2.3-196 | 2014-09-25 17:53:27 -0700 * Changing prefix for packet sources/dumper from ':' to '%'. Addresses BIT-1249. (Robin Sommer) * Remove timeouts from remote communication loop. The select() now blocks until there's work to do instead of relying on a small timeout value which can cause unproductive use of cpu cycles. (Jon Siwek) * Improve error message when failing to activate a plugin. Also fix a unit test helper script that checks plugin availability. (Jon Siwek) 2.3-183 | 2014-09-24 10:08:04 -0500 * Add a "node" field to Intel::Seen struture and intel.log to indicate which node discovered a hit on an intel item. (Seth Hall) * BIT-1261: Fixes to plugin quick start doc. (Jon Siwek) 2.3-180 | 2014-09-22 12:52:41 -0500 * BIT-1259: Fix issue w/ duplicate TCP reassembly deliveries. (Jon Siwek) 2.3-178 | 2014-09-18 14:29:46 -0500 * BIT-1256: Fix file analysis events from coming after bro_done(). (Jon Siwek) 2.3-177 | 2014-09-17 09:41:27 -0500 * Documentation fixes. (Chris Mavrakis) 2.3-174 | 2014-09-17 09:37:09 -0500 * Fixed some "make doc" warnings caused by reST formatting (Daniel Thayer). 2.3-172 | 2014-09-15 13:38:52 -0500 * Remove unneeded allocations for HTTP messages. (Jon Siwek) 2.3-171 | 2014-09-15 11:14:57 -0500 * Fix a compile error on systems without pcap-int.h. (Jon Siwek) 2.3-170 | 2014-09-12 19:28:01 -0700 * Fix incorrect data delivery skips after gap in HTTP Content-Range. Addresses BIT-1247. (Jon Siwek) * Fix file analysis placement of data after gap in HTTP Content-Range. Addresses BIT-1248. (Jon Siwek) * Fix issue w/ TCP reassembler not delivering some segments. Addresses BIT-1246. (Jon Siwek) * Fix MIME entity file data/gap ordering and raise http_entity_data in line with data arrival. Addresses BIT-1240. (Jon Siwek) * Implement file ID caching for MIME_Mail. (Jon Siwek) * Fix a compile error. (Jon Siwek) 2.3-161 | 2014-09-09 12:35:38 -0500 * Bugfixes and test updates/additions. (Robin Sommer) * Interface tweaks and docs for PktSrc/PktDumper. (Robin Sommer) * Moving PCAP-related bifs to iosource/pcap.bif. (Robin Sommer) * Moving some of the BPF filtering code into base class. This will allow packet sources that don't support BPF natively to emulate the filtering via libpcap. (Robin Sommer) * Removing FlowSrc. (Robin Sommer) * Removing remaining pieces of the 2ndary path, and left-over files of packet sorter. (Robin Sommer) * A bunch of infrastructure work to move IOSource, IOSourceRegistry (now iosource::Manager) and PktSrc/PktDumper code into iosource/, and over to a plugin structure. (Robin Sommer) 2.3-137 | 2014-09-08 19:01:13 -0500 * Fix Broxygen's rendering of opaque types. (Jon Siwek) 2.3-136 | 2014-09-07 20:50:46 -0700 * Change more http links to https. (Johanna Amann) 2.3-134 | 2014-09-04 16:16:36 -0700 * Fixed a number of issues with OCSP reply validation. Addresses BIT-1212. (Johanna Amann) * Fix null pointer dereference in OCSP verification code in case no certificate is sent as part as the ocsp reply. Addresses BIT-1212. (Johanna Amann) 2.3-131 | 2014-09-04 16:10:32 -0700 * Make links in documentation templates protocol relative. (Johanna Amann) 2.3-129 | 2014-09-02 17:21:21 -0700 * Simplify a conditional with equivalent branches. (Jon Siwek) * Change EDNS parsing code to use rdlength more cautiously. (Jon Siwek) * Fix a memory leak when bind() fails due to EADDRINUSE. (Jon Siwek) * Fix possible buffer over-read in DNS TSIG parsing. (Jon Siwek) 2.3-124 | 2014-08-26 09:24:19 -0500 * Better documentation for sub_bytes (Jimmy Jones) * BIT-1234: Fix build on systems that already have ntohll/htonll (Jon Siwek) 2.3-121 | 2014-08-22 15:22:15 -0700 * Detect functions that try to bind variables from an outer scope and raise an error saying that's not supported. Addresses BIT-1233. (Jon Siwek) 2.3-116 | 2014-08-21 16:04:13 -0500 * Adding plugin testing to Makefile's test-all. (Robin Sommer) * Converting log writers and input readers to plugins. DataSeries and ElasticSearch plugins have moved to the new bro-plugins repository, which is now a git submodule in the aux/plugins directory. (Robin Sommer) 2.3-98 | 2014-08-19 11:03:46 -0500 * Silence some doc-related warnings when using `bro -e`. Closes BIT-1232. (Jon Siwek) * Fix possible null ptr derefs reported by Coverity. (Jon Siwek) 2.3-96 | 2014-08-01 14:35:01 -0700 * Small change to DHCP documentation. In server->client messages the host name may differ from the one requested by the client. (Johanna Amann) * Split DHCP log writing from record creation. This allows users to customize dhcp.log by changing the record in their own dhcp_ack event. (Johanna Amann) * Update PATH so that documentation btests can find bro-cut. (Daniel Thayer) * Remove gawk from list of optional packages in documentation. (Daniel Thayer) * Fix for redefining built-in constants. (Robin Sommer) 2.3-86 | 2014-07-31 14:19:58 -0700 * Fix for redefining built-in constants. (Robin Sommer) * Adding missing check that a plugin's API version matches what Bro defines. (Robin Sommer) * Adding NEWS entry for plugins. (Robin Sommer) 2.3-83 | 2014-07-30 16:26:11 -0500 * Minor adjustments to plugin code/docs. (Jon Siwek) * Dynamic plugin support. (Rpbin Sommer) Bro now supports extending core functionality, like protocol and file analysis, dynamically with external plugins in the form of shared libraries. See doc/devel/plugins.rst for an overview of the main functionality. Changes coming with this: - Replacing the old Plugin macro magic with a new API. - The plugin API changed to generally use std::strings instead of const char*. - There are a number of invocations of PLUGIN_HOOK_ {VOID,WITH_RESULT} across the code base, which allow plugins to hook into the processing at those locations. - A few new accessor methods to various classes to allow plugins to get to that information. - network_time cannot be just assigned to anymore, there's now function net_update_time() for that. - Redoing how builtin variables are initialized, so that it works for plugins as well. No more init_net_var(), but instead bifcl-generated code that registers them. - Various changes for adjusting to the now dynamic generation of analyzer instances. - same_type() gets an optional extra argument allowing record type comparision to ignore if field names don't match. (Robin Sommer) - Further unify file analysis API with the protocol analyzer API (assigning IDs to analyzers; adding Init()/Done() methods; adding subtypes). (Robin Sommer) - A new command line option -Q that prints some basic execution time stats. (Robin Sommer) - Add support to the file analysis for activating analyzers by MIME type. (Robin Sommer) - File::register_for_mime_type(tag: Analyzer::Tag, mt: string): Associates a file analyzer with a MIME type. - File::add_analyzers_for_mime_type(f: fa_file, mtype: string): Activates all analyzers registered for a MIME type for the file. - The default file_new() handler calls File::add_analyzers_for_mime_type() with the file's MIME type. 2.3-20 | 2014-07-22 17:41:02 -0700 * Updating submodule(s). 2.3-19 | 2014-07-22 17:29:19 -0700 * Implement bytestring_to_coils() in Modbus analyzer so that coils gets passed to the corresponding events. (Hui Lin) * Add length field to ModbusHeaders. (Hui Lin) 2.3-12 | 2014-07-10 19:17:37 -0500 * Include yield of vectors in Broxygen's type descriptions. Addresses BIT-1217. (Jon Siwek) 2.3-11 | 2014-07-10 14:49:27 -0700 * Fixing DataSeries output. It was using a now illegal value as its default compression level. (Robin Sommer) 2.3-7 | 2014-06-26 17:35:18 -0700 * Extending "make test-all" to include aux/bro-aux. (Robin Sommer) 2.3-6 | 2014-06-26 17:24:10 -0700 * DataSeries compilation issue fixed. (mlaterman) * Fix a reference counting bug in ListVal ctor. (Jon Siwek) 2.3-3 | 2014-06-26 15:41:04 -0500 * Support tilde expansion when Bro tries to find its own path. (Jon Siwek) 2.3-2 | 2014-06-23 16:54:15 -0500 * Remove references to line numbers in tutorial text. (Daniel Thayer) 2.3 | 2014-06-16 09:48:25 -0500 * Release 2.3. 2.3-beta-33 | 2014-06-12 11:59:28 -0500 * Documentation improvements/fixes. (Daniel Thayer) 2.3-beta-24 | 2014-06-11 15:35:31 -0500 * Fix SMTP state tracking when server response is missing. (Robin Sommer) 2.3-beta-22 | 2014-06-11 12:31:38 -0500 * Fix doc/test that broke due to a Bro script change. (Jon Siwek) * Remove unused --with-libmagic configure option. (Jon Siwek) 2.3-beta-20 | 2014-06-10 18:16:51 -0700 * Fix use-after-free in some cases of reassigning a table index. Addresses BIT-1202. (Jon Siwek) 2.3-beta-18 | 2014-06-06 13:11:50 -0700 * Add two more SSL events, one triggered for each handshake message and one triggered for the tls change cipherspec message. (Johanna Amann) * Small SSL bug fix. In case SSL::disable_analyzer_after_detection was set to false, the ssl_established event would fire after each data packet once the session is established. (Johanna Amann) 2.3-beta-16 | 2014-06-06 13:05:44 -0700 * Re-activate notice suppression for expiring certificates. (Johanna Amann) 2.3-beta-14 | 2014-06-05 14:43:33 -0700 * Add new TLS extension type numbers from IANA (Johanna Amann) * Switch to double hashing for Bloomfilters for better performance. (Matthias Vallentin) * Bugfix to use full digest length instead of just one byte for Bloomfilter's universal hash function. Addresses BIT-1140. (Matthias Vallentin) * Make buffer for X509 certificate subjects larger. Addresses BIT-1195 (Johanna Amann) 2.3-beta-5 | 2014-05-29 15:34:42 -0500 * Fix misc/load-balancing.bro's reference to PacketFilter::sampling_filter (Jon Siwek) 2.3-beta-4 | 2014-05-28 14:55:24 -0500 * Fix potential mem leak in remote function/event unserialization. (Jon Siwek) * Fix reference counting bug in table coercion expressions (Jon Siwek) * Fix an "unused value" warning. (Jon Siwek) * Remove a duplicate unit test baseline dir. (Jon Siwek) 2.3-beta | 2014-05-19 16:36:50 -0500 * Release 2.3-beta * Clean up OpenSSL data structures on exit. (Johanna Amann) * Fixes for OCSP & x509 analysis memory leak issues. (Johanna Amann) * Remove remaining references to BROMAGIC (Daniel Thayer) * Fix typos and formatting in event and BiF documentation (Daniel Thayer) * Update intel framework plugin for ssl server_name extension API changes. (Johanna Amann, Justin Azoff) * Fix expression errors in SSL/x509 scripts when unparseable data is in certificate chain. (Johanna Amann) 2.2-478 | 2014-05-19 15:31:33 -0500 * Change record ctors to only allow record-field-assignment expressions. (Jon Siwek) 2.2-477 | 2014-05-19 14:13:00 -0500 * Fix X509::Result record's "result" field to be set internally as type int instead of type count. (Johanna Amann) * Fix a couple of doc build warnings (Daniel Thayer) 2.2-470 | 2014-05-16 15:16:32 -0700 * Add a new section "Cluster Configuration" to the docs that is intended as a how-to for configuring a Bro cluster. Most of this content was moved here from the BroControl doc (which is now intended as more of a reference guide for more experienced users) and the load balancing FAQ on the website. (Daniel Thayer) * Update some doc tests and line numbers (Daniel Thayer) 2.2-457 | 2014-05-16 14:38:31 -0700 * New script policy/protocols/ssl/validate-ocsp.bro that adds OSCP validation to ssl.log. The work is done by a new bif x509_ocsp_verify(). (Johanna Amann) * STARTTLS support for POP3 and SMTP. The SSL analyzer takes over when seen. smtp.log now logs when a connection switches to SSL. (Johanna Amann) * Replace errors when parsing x509 certs with weirds. (Johanna Amann) * Improved Heartbleed attack/scan detection. (Johanna Amann) * Let TLS analyzer fail better when no longer in sync with the data stream. (Johanna Amann) 2.2-444 | 2014-05-16 14:10:32 -0500 * Disable all default AppStat plugins except facebook. (Jon Siwek) * Update for the active http test to force it to use ipv4. (Seth Hall) 2.2-441 | 2014-05-15 11:29:56 -0700 * A new RADIUS analyzer. (Vlad Grigorescu) It produces a radius.log and generates two events: event radius_message(c: connection, result: RADIUS::Message); event radius_attribute(c: connection, attr_type: count, value: string); 2.2-427 | 2014-05-15 13:37:23 -0400 * Fix dynamic SumStats update on clusters (Johanna Amann) 2.2-425 | 2014-05-08 16:34:44 -0700 * Fix reassembly of data w/ sizes beyond 32-bit capacities. (Jon Siwek) Reassembly code (e.g. for TCP) now uses int64/uint64 (signedness is situational) data types in place of int types in order to support delivering data to analyzers that pass 2GB thresholds. There's also changes in logic that accompany the change in data types, e.g. to fix TCP sequence space arithmetic inconsistencies. Another significant change is in the Analyzer API: the *Packet and *Undelivered methods now use a uint64 in place of an int for the relative sequence space offset parameter. Addresses BIT-348. * Fixing compiler warnings. (Robin Sommer) * Update SNMP analyzer's DeliverPacket method signature. (Jon Siwek) 2.2-417 | 2014-05-07 10:59:22 -0500 * Change handling of atypical OpenSSL error case in x509 verification. (Jon Siwek) * Fix memory leaks in X509 certificate parsing/verification. (Jon Siwek) * Fix new []/delete mismatch in input::reader::Raw::DoClose(). (Jon Siwek) * Fix buffer over-reads in file_analysis::Manager::Terminate() (Jon Siwek) * Fix buffer overlows in IP address masking logic. (Jon Siwek) That could occur either in taking a zero-length mask on an IPv6 address (e.g. [fe80::]/0) or a reverse mask of length 128 on any address (e.g. via the remask_addr BuiltIn Function). * Fix new []/delete mismatch in ~Base64Converter. (Jon Siwek) 2.2-410 | 2014-05-02 12:49:53 -0500 * Replace an unneeded OPENSSL_malloc call. (Jon Siwek) 2.2-409 | 2014-05-02 12:09:06 -0500 * Clean up and documentation for base SNMP script. (Jon Siwek) * Update base SNMP script to now produce a snmp.log. (Seth Hall) * Add DH support to SSL analyzer. When using DHE or DH-Anon, sever key parameters are now available in scriptland. Also add script to alert on weak certificate keys or weak dh-params. (Johanna Amann) * Add a few more ciphers Bro did not know at all so far. (Johanna Amann) * Log chosen curve when using ec cipher suite in TLS. (Johanna Amann) 2.2-397 | 2014-05-01 20:29:20 -0700 * Fix reference counting for lookup_ID() usages. (Jon Siwek) 2.2-395 | 2014-05-01 20:25:48 -0700 * Fix missing "irc-dcc-data" service field from IRC DCC connections. (Jon Siwek) * Correct a notice for heartbleed. The notice is thrown correctly, just the message conteined wrong values. (Johanna Amann) * Improve/standardize some malloc/realloc return value checks. (Jon Siwek) * Improve file analysis manager shutdown/cleanup. (Jon Siwek) 2.2-388 | 2014-04-24 18:38:07 -0700 * Fix decoding of MIME quoted-printable. (Mareq) 2.2-386 | 2014-04-24 18:22:29 -0700 * Do a Intel::ADDR lookup for host field if we find an IP address there. (jshlbrd) 2.2-381 | 2014-04-24 17:08:45 -0700 * Add Java version to software framework. (Brian Little) 2.2-379 | 2014-04-24 17:06:21 -0700 * Remove unused Val::attribs member. (Jon Siwek) 2.2-377 | 2014-04-24 16:57:54 -0700 * A larger set of SSL improvements and extensions. Addresses BIT-1178. (Johanna Amann) - Fixes TLS protocol version detection. It also should bail-out correctly on non-tls-connections now - Adds support for a few TLS extensions, including server_name, alpn, and ec-curves. - Adds support for the heartbeat events. - Add Heartbleed detector script. - Adds basic support for OCSP stapling. * Fix parsing of DNS TXT RRs w/ multiple character-strings. Addresses BIT-1156. (Jon Siwek) 2.2-353 | 2014-04-24 16:12:30 -0700 * Adapt HTTP partial content to cache file analysis IDs. (Jon Siwek) * Adapt SSL analyzer to generate file analysis handles itself. (Jon Siwek) * Adapt more of HTTP analyzer to use cached file analysis IDs. (Jon Siwek) * Adapt IRC/FTP analyzers to cache file analysis IDs. (Jon Siwek) * Refactor regex/signature AcceptingSet data structure and usages. (Jon Siwek) * Enforce data size limit when checking files for MIME matches. (Jon Siwek) * Refactor file analysis file ID lookup. (Jon Siwek) 2.2-344 | 2014-04-22 20:13:30 -0700 * Refactor various hex escaping code. (Jon Siwek) 2.2-341 | 2014-04-17 18:01:41 -0500 * Fix duplicate DNS log entries. (Robin Sommer) 2.2-341 | 2014-04-17 18:01:01 -0500 * Refactor initialization of ASCII log writer options. (Jon Siwek) * Fix a memory leak in ASCII log writer. (Jon Siwek) 2.2-338 | 2014-04-17 17:48:17 -0500 * Disable input/logging threads setting their names on every heartbeat. (Jon Siwek) * Fix bug when clearing Bloom filter contents. Reported by @colonelxc. (Matthias Vallentin) 2.2-335 | 2014-04-10 15:04:57 -0700 * Small logic fix for main SSL script. (Johanna Amann) * Update DPD signatures for detecting TLS 1.2. (Johanna Amann) * Remove unused data member of SMTP_Analyzer to silence a Coverity warning. (Jon Siwek) * Fix missing @load dependencies in some scripts. Also update the unit test which is supposed to catch such errors. (Jon Siwek) 2.2-326 | 2014-04-08 15:21:51 -0700 * Add SNMP datagram parsing support.This supports parsing of SNMPv1 (RFC 1157), SNMPv2 (RFC 1901/3416), and SNMPv2 (RFC 3412). An event is raised for each SNMP PDU type, though there's not currently any event handlers for them and not a default snmp.log either. However, simple presence of SNMP is currently visible now in conn.log service field and known_services.log. (Jon Siwek) 2.2-319 | 2014-04-03 15:53:25 -0700 * Improve __load__.bro creation for .bif.bro stubs. (Jon Siwek) 2.2-317 | 2014-04-03 10:51:31 -0400 * Add a uid field to the signatures.log. Addresses BIT-1171 (Anthony Verez) 2.2-315 | 2014-04-01 16:50:01 -0700 * Change logging's "#types" description of sets to "set". Addresses BIT-1163 (Johanna Amann) 2.2-313 | 2014-04-01 16:40:19 -0700 * Fix a couple nits reported by Coverity.(Jon Siwek) * Fix potential memory leak in IP frag reassembly reported by Coverity. (Jon Siwek) 2.2-310 | 2014-03-31 18:52:22 -0700 * Fix memory leak and unchecked dynamic cast reported by Coverity. (Jon Siwek) * Fix potential memory leak in x509 parser reported by Coverity. (Johanna Amann) 2.2-304 | 2014-03-30 23:05:54 +0200 * Replace libmagic w/ Bro signatures for file MIME type identification. Addresses BIT-1143. (Jon Siwek) Includes: - libmagic is no longer used at all. All MIME type detection is done through new Bro signatures, and there's no longer a means to get verbose file type descriptions. The majority of the default file magic signatures are derived from the default magic database of libmagic ~5.17. - File magic signatures consist of two new constructs in the signature rule parsing grammar: "file-magic" gives a regular expression to match against, and "file-mime" gives the MIME type string of content that matches the magic and an optional strength value for the match. - Modified signature/rule syntax for identifiers: they can no longer start with a '-', which made for ambiguous syntax when doing negative strength values in "file-mime". Also brought syntax for Bro script identifiers in line with reality (they can't start with numbers or include '-' at all). - A new built-in function, "file_magic", can be used to get all file magic matches and their corresponding strength against a given chunk of data. - The second parameter of the "identify_data" built-in function can no longer be used to get verbose file type descriptions, though it can still be used to get the strongest matching file magic signature. - The "file_transferred" event's "descr" parameter no longer contains verbose file type descriptions. - The BROMAGIC environment variable no longer changes any behavior in Bro as magic databases are no longer used/installed. - Removed "binary" and "octet-stream" mime type detections. They don' provide any more information than an uninitialized mime_type field which implicitly means no magic signature matches and so the media type is unknown to Bro. - The "fa_file" record now contains a "mime_types" field that contains all magic signatures that matched the file content (where the "mime_type" field is just a shortcut for the strongest match). - Reverted back to minimum requirement of CMake 2.6.3 from 2.8.0. * The logic for adding file ids to {orig,resp}_fuids fields of the http.log incorrectly depended on the state of {orig,resp}_mime_types fields, so sometimes not all file ids associated w/ the session were logged. (Jon Siwek) * Fix MHR script's use of fa_file$mime_type before checking if it's initialized. (Jon Siwek) 2.2-294 | 2014-03-30 22:08:25 +0200 * Rework and move X509 certificate processing from the SSL protocol analyzer to a dedicated file analyzer. This will allow us to examine X509 certificates from sources other than SSL in the future. Furthermore, Bro now parses more fields and extensions from the certificates (e.g. elliptic curve information, subject alternative names, basic constraints). Certificate validation also was improved, should be easier to use and exposes information like the full verified certificate chain. (Johanna Amann) This update changes the format of ssl.log and adds a new x509.log with certificate information. Furthermore all x509 events and handling functions have changed. 2.2-271 | 2014-03-30 20:25:17 +0200 * Add unit tests covering vector/set/table ctors/inits. (Jon Siwek) * Fix parsing of "local" named table constructors. (Jon Siwek) * Improve type checking of records. Addresses BIT-1159. (Jon Siwek) 2.2-267 | 2014-03-30 20:21:43 +0200 * Improve documentation of Bro clusters. Addresses BIT-1160. (Daniel Thayer) 2.2-263 | 2014-03-30 20:19:05 +0200 * Don't include locations into serialization when cloning values. (Robin Sommer) 2.2-262 | 2014-03-30 20:12:47 +0200 * Refactor SerializationFormat::EndWrite and ChunkedIO::Chunk memory management. (Jon Siwek) * Improve SerializationFormat's write buffer growth strategy. (Jon Siwek) * Add --parse-only option to exit after parsing scripts. May be useful for syntax-checking tools. (Jon Siwek) 2.2-256 | 2014-03-30 19:57:28 +0200 * For the summary statistics framewirk, change all &create_expire attributes to &read_expire in the cluster part. (Johanna Amann) 2.2-254 | 2014-03-30 19:55:22 +0200 * Update instructions on how to build Bro docs. (Daniel Thayer) 2.2-251 | 2014-03-28 08:37:37 -0400 * Quick fix to the ElasticSearch writer. (Seth Hall) 2.2-250 | 2014-03-19 17:20:55 -0400 * Improve performance of MHR script by reducing cloned Vals in a "when" scope. (Jon Siwek) 2.2-248 | 2014-03-19 14:47:40 -0400 * Make SumStats work incrementally and non-blocking in non-cluster mode, but force it to operate by blocking if Bro is shutting down. (Seth Hall) 2.2-244 | 2014-03-17 08:24:17 -0700 * Fix compile errror on FreeBSD caused by wrong include file order. (Johanna Amann) 2.2-240 | 2014-03-14 10:23:54 -0700 * Derive results of DNS lookups from from input when in BRO_DNS_FAKE mode. Addresses BIT-1134. (Jon Siwek) * Fixing a few cases of undefined behaviour introduced by recent formatter work. * Fixing compiler error. (Robin Sommer) * Fixing (very unlikely) double delete in HTTP analyzer when decapsulating CONNECTs. (Robin Sommer) 2.2-235 | 2014-03-13 16:21:19 -0700 * The Ascii writer has a new option LogAscii::use_json for writing out logs as JSON. (Seth Hall) * Ascii input reader now supports all config options as per-input stream "config" values. (Seth Hall) * Refactored formatters and updated the the writers a bit. (Seth Hall) 2.2-229 | 2014-03-13 14:58:30 -0700 * Refactoring analyzer manager code to reuse ApplyScheduledAnalyzers(). (Robin Sommer) 2.2-228 | 2014-03-13 14:25:53 -0700 * Teach async DNS lookup builtin-functions about BRO_DNS_FAKE. Addresses BIT-1134. (Jon Siwek) * Enable fake DNS mode for test suites. * Improve analysis of TCP SYN/SYN-ACK reversal situations. (Jon Siwek) - Since it's just the handshake packets out of order, they're no longer treated as partial connections, which some protocol analyzers immediately refuse to look at. - The TCP_Reassembler "is_orig" state failed to change, which led to protocol analyzers sometimes using the wrong value for that. - Add a unit test which exercises the Connection::FlipRoles() code path (i.e. the SYN/SYN-ACK reversal situation). Addresses BIT-1148. * Fix bug in Connection::FlipRoles. It didn't swap address values right and also didn't consider that analyzers might be scheduled for the new connection tuple. Reported by Kevin McMahon. Addresses BIT-1148. (Jon Siwek) 2.2-221 | 2014-03-12 17:23:18 -0700 * Teach configure script --enable-jemalloc, --with-jemalloc. Addresses BIT-1128. (Jon Siwek) 2.2-218 | 2014-03-12 17:19:45 -0700 * Improve DBG_LOG macro (perf. improvement for --enable-debug mode). (Jon Siwek) * Silences some documentation warnings from Sphinx. (Jon Siwek) 2.2-215 | 2014-03-10 11:10:15 -0700 * Fix non-deterministic logging of unmatched DNS msgs. Addresses BIT-1153 (Jon Siwek) 2.2-213 | 2014-03-09 08:57:37 -0700 * No longer accidentally attempting to parse NBSTAT RRs as SRV RRs in DNS analyzer. (Seth Hall) * Fix DNS SRV responses and a small issue with NBNS queries and label length. (Seth Hall) - DNS SRV responses never had the code written to actually generate the dns_SRV_reply event. Adding this required extending the event a bit to add extra information. SRV responses now appear in the dns.log file correctly. - Fixed an issue where some Microsoft NetBIOS Name Service lookups would exceed the max label length for DNS and cause an incorrect "DNS_label_too_long" weird. 2.2-210 | 2014-03-06 22:52:36 -0500 * Improve SSL logging so that connections are logged even when the ssl_established event is not generated as well as other small SSL fixes. (Johanna Amann) 2.2-206 | 2014-03-03 16:52:28 -0800 * HTTP CONNECT proxy support. The HTTP analyzer now supports handling HTTP CONNECT proxies. (Seth Hall) * Expanding the HTTP methods used in the DPD signature to detect HTTP traffic. (Seth Hall) * Fixing removal of support analyzers. (Robin Sommer) 2.2-199 | 2014-03-03 16:34:20 -0800 * Allow iterating over bif functions with result type vector of any. This changes the internal type that is used to signal that a vector is unspecified from any to void. Addresses BIT-1144 (Johanna Amann) 2.2-197 | 2014-02-28 15:36:58 -0800 * Remove test code. (Robin Sommer) 2.2-194 | 2014-02-28 14:50:53 -0800 * Remove packet sorter. Addresses BIT-700. (Johanna Amann) 2.2-192 | 2014-02-28 09:46:43 -0800 * Update Mozilla root bundle. (Johanna Amann) 2.2-190 | 2014-02-27 07:34:44 -0800 * Adjust timings of a few leak tests. (Johanna Amann) 2.2-187 | 2014-02-25 07:24:42 -0800 * More Google TLS extensions that are being actively used. Johanna( Amann) * Remove unused, and potentially unsafe, function ListVal::IncludedInString. (Johanna Amann) 2.2-184 | 2014-02-24 07:28:18 -0800 * New TLS constants from https://tools.ietf.org/html/draft-bmoeller-tls-downgrade-scsv-01. (Johanna Amann) 2.2-180 | 2014-02-20 17:29:14 -0800 * New SSL alert descriptions from https://tools.ietf.org/html/draft-ietf-tls-applayerprotoneg-04. (Johanna Amann) * Update SQLite. (Johanna Amann) 2.2-177 | 2014-02-20 17:27:46 -0800 * Update to libmagic version 5.17. Addresses BIT-1136. (Jon Siwek) 2.2-174 | 2014-02-14 12:07:04 -0800 * Support for MPLS over VLAN. (Chris Kanich) 2.2-173 | 2014-02-14 10:50:15 -0800 * Fix misidentification of SOCKS traffic that in particiular seemed to happen a lot with DCE/RPC traffic. (Vlad Grigorescu) 2.2-170 | 2014-02-13 16:42:07 -0800 * Refactor DNS script's state management to improve performance. (Jon Siwek) * Revert "Expanding the HTTP methods used in the signature to detect HTTP traffic." (Robin Sommer) 2.2-167 | 2014-02-12 20:17:39 -0800 * Increase timeouts of some unit tests. (Jon Siwek) * Fix memory leak in modbus analyzer. Would happen if there's a 'modbus_read_fifo_queue_response' event handler. (Jon Siwek) * Add channel_id TLS extension number. This number is not IANA defined, but we see it being actively used. (Johanna Amann) * Test baseline updates for DNS change. (Robin Sommer) 2.2-158 | 2014-02-09 23:45:39 -0500 * Change dns.log to include only standard DNS queries. (Jon Siwek) * Improve DNS analysis. (Jon Siwek) - Fix parsing of empty question sections (when QDCOUNT == 0). In this case, the DNS parser would extract two 2-byte fields for use in either "dns_query_reply" or "dns_rejected" events (dependent on value of RCODE) as qclass and qtype parameters. This is not correct, because such fields don't actually exist in the DNS message format when QDCOUNT is 0. As a result, these events are no longer raised when there's an empty question section. Scripts that depends on checking for an empty question section can do that in the "dns_message" event. - Add a new "dns_unknown_reply" event, for when Bro does not know how to fully parse a particular resource record type. This helps fix a problem in the default DNS scripts where the logic to complete request-reply pair matching doesn't work because it's waiting on more RR events to complete the reply. i.e. it expects ANCOUNT number of dns_*_reply events and will wait until it gets that many before completing a request-reply pair and logging it to dns.log. This could cause bogus replies to match a previous request if they happen to share a DNS transaction ID. (Jon Siwek) - The previous method of matching queries with replies was still unreliable in cases where the reply contains no answers. The new code also takes extra measures to avoid pending state growing too large in cases where the condition to match a query with a corresponding reply is never met, but yet DNS messages continue to be exchanged over the same connection 5-tuple (preventing cleanup of the pending state). (Jon Siwek) * Updates to httpmonitor and mimestats documentation. (Jeannette Dopheide) * Updates to Logs and Cluster documentation (Jeannette Dopheide) 2.2-147 | 2014-02-07 08:06:53 -0800 * Fix x509-extension test sometimes failing. (Johanna Amann) 2.2-144 | 2014-02-06 20:31:18 -0800 * Fixing bug in POP3 analyzer. With certain input the analyzer could end up trying to write to non-writable memory. (Robin Sommer) 2.2-140 | 2014-02-06 17:58:04 -0800 * Fixing memory leaks in input framework. (Robin Sommer) * Add script to detect filtered TCP traces. Addresses BIT-1119. (Jon Siwek) 2.2-137 | 2014-02-04 09:09:55 -0800 * Minor unified2 script documentation fix. (Jon Siwek) 2.2-135 | 2014-01-31 11:09:36 -0800 * Added some grammar and spelling corrections to Installation and Quick Start Guide. (Jeannette Dopheide) 2.2-131 | 2014-01-30 16:11:11 -0800 * Extend file analysis API to allow file ID caching. This allows an analyzer to either provide file IDs associated with some file content or to cache a file ID that was already determined by script-layer logic so that subsequent calls to the file analysis interface can bypass costly detours through script-layer. This can yield a decent performance improvement for analyzers that are able to take advantage of it and deal with streaming content (like HTTP, which has been adapted accordingly). (Jon Siwek) 2.2-128 | 2014-01-30 15:58:47 -0800 * Add leak test for Exec module. (Johanna Amann) * Fix file_over_new_connection event to trigger when entire file is missed. (Jon Siwek) * Improve TCP connection size reporting for half-open connections. (Jon Siwek) * Improve gap reporting in TCP connections that never see data. We no longer accomodate SYN/FIN/RST-filtered traces by not reporting missing data. The behavior can be reverted by redef'ing "detect_filtered_trace". (Jon Siwek) * Improve TCP FIN retransmission handling. (Jon Siwek) 2.2-120 | 2014-01-28 10:25:23 -0800 * Fix and extend x509_extension() event, which now actually returns the extension. (Johanna Amann) New event signauture: event x509_extension(c: connection, is_orig: bool, cert:X509, extension: X509_extension_info) 2.2-117 | 2014-01-23 14:18:19 -0800 * Fixing initialization context in anonymous functions. (Robin Sommer) 2.2-115 | 2014-01-22 12:11:18 -0800 * Add unit tests for new Bro Manual docs. (Jon Siwek) * New content for the "Using Bro" section of the manual. (Rafael Bonilla/Jon Siwek) 2.2-105 | 2014-01-20 12:16:48 -0800 * Support GRE tunnel decapsulation, including enhanced GRE headers. GRE tunnels are treated just like IP-in-IP tunnels by parsing past the GRE header in between the delivery and payload IP packets. Addresses BIT-867. (Jon Siwek) * Simplify FragReassembler memory management. (Jon Siwek) 2.2-102 | 2014-01-20 12:00:29 -0800 * Include file information (MIME type and description) into notice emails if available. (Justin Azoff) 2.2-100 | 2014-01-20 11:54:58 -0800 * Fix caching of recently validated SSL certifcates. (Justin Azoff) 2.2-98 | 2014-01-20 11:50:32 -0800 * For notice suppresion, instead of storing the entire notice in Notice::suppressing, just store the time the notice should be suppressed until. This saves significant memory but can no longer raise end_suppression, which has been removed. (Justin Azoff) 2.2-96 | 2014-01-20 11:41:07 -0800 * Integrate libmagic 5.16. Bro now now always relies on builtin/shipped magic library/database. (Jon Siwek) * Bro now requires a CMake 2.8.x, but no longer a pre-installed libmagic. (Jon Siwek) 2.2-93 | 2014-01-13 09:16:51 -0800 * Fixing compile problems with some versions of libc++. Reported by Craig Leres. (Robin Sommer) 2.2-91 | 2014-01-13 01:33:28 -0800 * Improve GeoIP City database support. When trying to open a city database, it now considers both the "REV0" and "REV1" versions of the city database instead of just the former. (Jon Siwek) * Broxygen init fixes. Addresses BIT-1110. (Jon Siwek) - Don't check mtime of bro binary if BRO_DISABLE_BROXYGEN env var set. - Fix failure to locate bro binary if invoking from a relative path and '.' isn't in PATH. * Fix for packet writing to make it use the global snap length. (Seth Hall) * Fix for traffic with TCP segmentation offloading with IP header len field being set to zero. (Seth Hall) * Canonify output of a unit test. (Jon Siwek) * A set of documentation updates. (Daniel Thayer) - Fix typo in Bro 2.2 NEWS on string indexing. - Fix typo in the Quick Start Guide, and clarified the instructions about modifying crontab. - Add/fix documentation for missing/misnamed event parameters. - Fix typos in BIF documentation of hexstr_to_bytestring. - Update the documentation of types and attributes. - Documented the new substring extraction functionality. - Clarified the description of "&priority" and "void". 2.2-75 | 2013-12-18 08:36:50 -0800 * Fixing segfault with mismatching set &default in record fields. (Robin Sommer) 2.2-74 | 2013-12-16 08:49:55 -0800 * Improve warnings emitted from raw/execute input reader. (Jon Siwek) * Further improve core.when-interpreter-exceptions unit test. (Jon Siwek) 2.2-72 | 2013-12-12 07:12:47 -0800 * Improve the core.when-interpreter-exceptions unit test to prevent it from occasionally timing out. (Jon Siwek) 2.2-70 | 2013-12-10 15:02:50 -0800 * Fix (harmless) uninitialized field in basename/dirname util wrapper. (Jon Siwek) 2.2-68 | 2013-12-09 15:19:37 -0800 * Several improvements to input framework error handling for more robustness and more helpful error messages. Includes tests for many cases. (Johanna Amann) 2.2-66 | 2013-12-09 13:54:16 -0800 * Fix table &default reference counting for record ctor expressions. (Jon Siwek) * Close signature files after done parsing. (Jon Siwek) * Fix unlikely null ptr deref in broxygen::Manager. (Jon Siwek) * FreeBSD build fix addendum: unintended variable shadowing. (Jon Siwek) * Fix build on FreeBSD. basename(3)/dirname(3) const-ness may vary w/ platform. (Jon Siwek) * Updated software framework to support parsing IE11 user-agent strings. (Seth Hall) * Fix the irc_reply event for several server message types. (Seth Hall) * Fix memory leak in input framework. If the input framework was used to read event streams and those streams contained records with more than one field, not all elements of the threading Values were cleaned up. Addresses BIT-1103. (Johanna Amann) * Minor Broxygen improvements. Addresses BIT-1098. (Jon Siwek) 2.2-51 | 2013-12-05 07:53:37 -0800 * Improve a unit test involving 'when' conditionals. (Jon Siwek) 2.2-48 | 2013-12-04 13:45:47 -0800 * Support omission of string slice low/high indices, BIT-1097. Omission of the low index defaults to 0: s = "12345"; s[:3] == "123" Omission of the high index defaults to length of the string: s = "12345"; s[3:] == "45" (Jon Siwek) * Tweak to SMTP script to adjust for new string slicing behaviour. (Robin Sommer) * Test updates. (Robin Sommer) 2.2-44 | 2013-12-04 12:41:51 -0800 * Fix string slice notation. Addresses BIT-1097. (Jon Siwek) Slice ranges were not correctly determined for negative indices and also off by one in general (included one more element at the end of the substring than what actually matched the index range). It's now equivalent to Python slice notation. Accessing a string at a single index is also the same as Python except that an out-of-range index returns an empty string instead of throwing an expection. 2.2-41 | 2013-12-04 12:40:51 -0800 * Updating tests. (Robin Sommer) 2.2-40 | 2013-12-04 12:16:38 -0800 * ssl_client_hello() now receives a vector of ciphers, instead of a set, to preserve their order. (Johanna Amann) 2.2-38 | 2013-12-04 12:10:54 -0800 * New script misc/dump-events.bro, along with core support, that dumps events Bro is raising in an easily readable form for debugging. (Robin Sommer) * Prettyfing Describe() for record types. If a record type has a name and ODesc is set to short, we now print the name instead of the full field list. (Robin Sommer) 2.2-35 | 2013-12-04 10:10:32 -0800 * Rework the automated script-reference documentation generation process, broxygen. Addresses BIT-701 and BIT-751. (Jon Siwek) Highlights: - Remove --doc-scripts and -Z options to toggle documentation mode. The parser is now always instrumented to gather documentation from comments of the form "##", "##!", or "##<". - Raw comments are available at runtime through several BIF functions: get_*_comments; - Add --broxygen and -X options to toggle generating reST-format documentation output, driven by a config file argument. - Add a "broxygen" Sphinx extension domain, allowing certain pieces of documentation to be generated on-the-fly via invoking a Bro process. Re-organized/cleaned up the Sphinx source tree in doc/ to use this in some places. 2.2-11 | 2013-12-03 10:56:28 -0800 * Unit test for broccoli vector support. (Jon Siwek) * Changed ordering of Bro type tag enum, which was out of sync. (Jon Siwek) 2.2-9 | 2013-11-18 14:03:21 -0800 * Update local.bro for Bro >= 2.2. The commented out Notice::policy example didn't work anymore. (Daniel Thayer) 2.2-6 | 2013-11-15 07:05:15 -0800 * Make "install-example-configs" target use DESTDIR. (Jon Siwek) 2.2-5 | 2013-11-11 13:47:54 -0800 * Fix the irc_reply event for certain server message types. (Seth Hall) * Fixed Segmentation fault in SQLite Writer. (Jon Crussell) 2.2 | 2013-11-07 10:25:50 -0800 * Release 2.2. * Removing location information from ssh.log in external tests. (Robin Sommer) 2.2-beta-199 | 2013-11-07 00:36:46 -0800 * Fixing warnings during doc build. (Robin Sommer) 2.2-beta-198 | 2013-11-06 22:54:30 -0800 * Update docs and tests for a recent change to detect-MHR.bro (Daniel Thayer) * Update tests and baselines for sumstats docs. (Daniel Thayer) 2.2-beta-194 | 2013-11-06 14:39:50 -0500 * Remove resp_size from the ssh log. Refactor when we write out to the log a bit. Geodata now works reliably. (Vlad Grigorescu) * Update VirusTotal URL to work with changes to their website and changed it to a redef. (Vlad Grigorescu) * Added a document for the SumStats framework. (Seth Hall) 2.2-beta-184 | 2013-11-03 22:53:42 -0800 * Remove swig-ruby from required packages section of install doc. (Daniel Thayer) 2.2-beta-182 | 2013-11-01 05:26:05 -0700 * Adding source and original copyright statement to Mozilla cert list. (Robin Sommer) * Canonfying an intel test to not depend on output order. (Robin Sommer) 2.2-beta-177 | 2013-10-30 04:54:54 -0700 * Fix thread processing/termination conditions. (Jon Siwek) 2.2-beta-175 | 2013-10-29 09:30:09 -0700 * Return the Dir module to file name tracking instead of inode tracking to avoid missing files that reuse a formerly seen inode. (Seth Hall) * Deprecate Broccoli Ruby bindings and no longer build them by default; use --enable-ruby to do so. (Jon Siwek) 2.2-beta-167 | 2013-10-29 06:02:38 -0700 * Change percent_lost in capture-loss from a string to a double. (Vlad Grigorescu) * New version of the threading queue deadlock fix. (Robin Sommer) * Updating README with download/git information. (Robin Sommer) 2.2-beta-161 | 2013-10-25 15:48:15 -0700 * Add curl to list of optional dependencies. It's used by the active-http.bro script. (Daniel Thayer) * Update test and baseline for a recent doc test fix. (Daniel Thayer) 2.2-beta-158 | 2013-10-25 15:05:08 -0700 * Updating README with download/git information. (Robin Sommer) 2.2-beta-157 | 2013-10-25 11:11:17 -0700 * Extend the documentation of the SQLite reader/writer framework. (Johanna Amann) * Fix inclusion of wrong example file in scripting tutorial. Reported by Michael Auger @LM4K. (Johanna Amann) * Alternative fix for the thrading deadlock issue to avoid potential performance impact. (Johanna Amann) 2.2-beta-152 | 2013-10-24 18:16:49 -0700 * Fix for input readers occasionally dead-locking. (Robin Sommer) 2.2-beta-151 | 2013-10-24 16:52:26 -0700 * Updating submodule(s). 2.2-beta-150 | 2013-10-24 16:32:14 -0700 * Change temporary ASCII reader workaround for getline() on Mavericks to permanent fix. (Johanna Amann) 2.2-beta-148 | 2013-10-24 14:34:35 -0700 * Add gawk to list of optional packages. (Daniel Thayer) * Add more script package README files. (Daniel Thayer) * Add NEWS about new features of BroControl and upgrade info. (Daniel Thayer) * Intel framework notes added to NEWS. (Seth Hall) * Temporary OSX Mavericks libc++ issue workaround for getline() problem in ASCII reader. (Johanna Amann) * Change test of identify_data BIF to ignore charset as it may vary with libmagic version. (Jon Siwek) * Ensure that the starting BPF filter is logged on clusters. (Seth Hall) * Add UDP support to the checksum offload detection script. (Seth Hall) 2.2-beta-133 | 2013-10-23 09:50:16 -0700 * Fix record coercion tolerance of optional fields. (Jon Siwek) * Add NEWS about incompatible local.bro changes, addresses BIT-1047. (Jon Siwek) * Fix minor formatting problem in NEWS. (Jon Siwek) 2.2-beta-129 | 2013-10-23 09:47:29 -0700 * Another batch of documentation fixes and updates. (Daniel Thayer) 2.2-beta-114 | 2013-10-18 14:17:57 -0700 * Moving the SQLite examples into separate Bro files to turn them into sphinx-btest tests. (Robin Sommer) 2.2-beta-112 | 2013-10-18 13:47:13 -0700 * A larger chunk of documentation fixes and cleanup. (Daniel Thayer) Apart from many smaller improves this includes in particular: * Add README files for most Bro frameworks and base/protocols. * Add README files for base/protocols. * Update installation instructions. * Improvements to file analysis docs and conversion to using btest sphinx. 2.2-beta-80 | 2013-10-18 13:18:05 -0700 * SQLite reader/writer documentation. (Johanna Amann) * Check that the SQLite reader is only used in MANUAL reading mode. (Johanna Amann) * Rename the SQLite writer "dbname" configuration option to "tablename". (Johanna Amann) * Remove the "dbname" configuration option from the SQLite reader as it wasn't used there. (Johanna Amann) 2.2-beta-73 | 2013-10-14 14:28:25 -0700 * Fix misc. Coverity-reported issues (leaks, potential null pointer deref, dead code, uninitialized values, time-of-check-time-of-use). (Jon Siwek) * Add check for sqlite3 command to tests that require it. (Daniel Thayer) 2.2-beta-68 | 2013-10-14 09:26:09 -0700 * Add check for curl command to active-http.test. (Daniel Thayer) 2.2-beta-64 | 2013-10-14 09:20:04 -0700 * Review usage of Reporter::InternalError, addresses BIT-1045. Replaced some with InternalWarning or AnalyzerError, the later being a new method which signals the analyzer to not process further input. (Jon Siwek) * Add new event for TCP content file write failures: "contents_file_write_failure". (Jon Siwek) 2.2-beta-57 | 2013-10-11 17:23:25 -0700 * Improve Broxygen end-of-sentence detection. (Jon Siwek) 2.2-beta-55 | 2013-10-10 13:36:38 -0700 * A couple of new TLS extension numbers. (Johanna Amann) * Suport for three more new TLS ciphers. (Johanna Amann) * Removing ICSI notary from default site config. (Robin Sommer) 2.2-beta-51 | 2013-10-07 17:33:56 -0700 * Polishing the reference and scripting sections of the manual. (Robin Sommer) * Fixing the historical CHANGES record. (Robin Sommer) * Updating copyright notice. (Robin Sommer) 2.2-beta-38 | 2013-10-02 11:03:29 -0700 * Fix uninitialized (or unused) fields. (Jon Siwek) * Remove logically dead code. (Jon Siwek) * Remove dead/unfinished code in unary not expression. (Jon Siwek) * Fix logic for failed DNS TXT lookups. (Jon Siwek) * A couple null ptr checks. (Jon Siwek) * Improve return value checking and error handling. (Jon Siwek) * Remove unused variable assignments. (Jon Siwek) * Prevent division/modulo by zero in scripts. (Jon Siwek) * Fix unintentional always-false condition. (Jon Siwek) * Fix invalidated iterator usage. (Jon Siwek) * Fix DNS_Mgr iterator mismatch. (Jon Siwek) * Set safe umask when creating script profiler tmp files. (Jon Siwek) * Fix nesting/indent level whitespace mismatch. (Jon Siwek) * Add checks to avoid improper negative values use. (Jon Siwek) 2.2-beta-18 | 2013-10-02 10:28:17 -0700 * Add support for further TLS cipher suites. (Johanna Amann) 2.2-beta-13 | 2013-10-01 11:31:55 -0700 * Updating bifcl usage message. (Robin Sommer) * Fix bifcl getopt() usage. (Jon Siwek) 2.2-beta-8 | 2013-09-28 11:16:29 -0700 * Fix a "make doc" warning. (Daniel Thayer) 2.2-beta-4 | 2013-09-24 13:23:30 -0700 * Fix for setting REPO in Makefile. (Robin Sommer) * Whitespace fix. (Robin Sommer) * Removing :doc: roles so that we can render this with docutils directly. (Robin Sommer) 2.2-beta | 2013-09-23 20:57:48 -0700 * Update 'make dist' target. (Jon Siwek) 2.1-1387 | 2013-09-23 11:54:48 -0700 * Change submodules to fixed URL. (Jon Siwek) * Updating NEWS. (Robin Sommer) * Fixing an always false condition. (Robin Sommer) * Fix required for compiling with clang 3.3. (Robin Sommer) 2.1-1377 | 2013-09-20 14:38:15 -0700 * Updates to the scripting introduction. (Scott Runnels) * Kill raw input reader's child by process group to reliably clean it up. (Jon Siwek) 2.1-1368 | 2013-09-19 20:07:57 -0700 * Add more links in the GeoLocation document (Daniel Thayer) 2.1-1364 | 2013-09-19 15:12:08 -0700 * Add links to Intelligence Framework documentation. (Daniel Thayer) * Update Mozilla root CA list. (Johanna Amann, Jon Siwek) * Update documentation of required packages. (Daniel Thayer) 2.1-1359 | 2013-09-18 15:01:50 -0700 * Make client and server random available on script-level. Addresses BIT-950. (Eric Wustrow) 2.1-1357 | 2013-09-18 14:58:52 -0700 * Update HLL API and its documentation. (Johanna Amann) * Fix case in HLL where hll_error_margin could be undefined. (Johanna Amann) 2.1-1352 | 2013-09-18 14:42:28 -0700 * Fix a number of compiler warnings. (Daniel Thayer) * Fix cmake warning about ENABLE_PERFTOOLS not being used. (Daniel Thayer) 2.1-1344 | 2013-09-16 16:20:55 -0500 * Refactor Analyzer::AddChildAnalyzer and usages. (Jon Siwek) * Minor refactor to SSL BinPAC grammer. (Jon Siwek) * Minor refactor to Broxygen enum comments. (Jon Siwek) * Fix possible (unlikely) use of uninitialized value. (Jon Siwek) * Fix/improve dereference-before-null-checks. (Jon Siwek) * Fix out-of-bounds memory accesses, and remove a variable-length-array usage. (Jon Siwek) * Fix potential mem leak. (Jon Siwek) * Fix double-free and deallocator mismatch. (Jon Siwek) * Fix another function val reference counting bug. (Jon Siwek) 2.1-1335 | 2013-09-12 16:13:53 -0500 * Documentation fixes (Daniel Thayer, Jon Siwek) * Fix various potential memory leaks. (Jon Siwek) * Fix significant memory leak in function unserialization. (Jon Siwek) * Fix use-after-free and invalid/mismatch deallocator bugs. (Jon Siwek) * Fixed an issue with the HLL_UNIQUE SumStats plugin that caused a reporter error. (Seth Hall) * Make the notice $actions field have a default empty set to avoid having to check for it's presence. (Seth Hall) * Fix signatures that use identifiers of type table. (Jon Siwek) * Fix memory leak if a DNS request fails to be made. (Jon Siwek) * Fix memory leak in DNS TXT lookups. (Jon Siwek) * Fix raw execution input reader's signal blocking which resulted in lingering processes. (Jon Siwek) 2.1-1306 | 2013-08-31 16:06:05 -0700 * Reorganized and signifcantly extended documentation. This includes two new chapters contributed by Scott Runnels. 2.1-1216 | 2013-08-31 10:39:40 -0700 * Support for probabilistic set cardinality, using the HyperLogLog algorithm. (Johanna Amann, Soumya Basu) Bro now provides the following BiFs: hll_cardinality_init(err: double, confidence: double): opaque of cardinality hll_cardinality_add(handle: opaque of cardinality, elem: any): bool hll_cardinality_merge_into(handle1: opaque of cardinality, handle2: opaque of cardinality): bool hll_cardinality_estimate(handle: opaque of cardinality): double hll_cardinality_copy(handle: opaque of cardinality): opaque of cardinality 2.1-1154 | 2013-08-30 08:27:45 -0700 * Fix global opaque val segfault. Addresses BIT-1071. (Jon Siwek) * Fix malloc/delete mismatch. (Jon Siwek) * Fix invalid pointer dereference in AsciiFormatter. (Jon Siwek) 2.1-1150 | 2013-08-29 13:43:01 -0700 * Fix input framework memory leaks. (Jon Siwek) * Fix memory leak in SOCKS analyzer for bad addr types. (Jon Siwek) * Fix Bloom filter memory leaks. (Jon Siwek) 2.1-1144 | 2013-08-28 18:51:06 -0700 * Add bits_per_uid unit test. Addresses BIT-1016. (Jon Siwek) * UID optimizations. Addresses BIT-1016. (Jon Siwek) * Added a $unique_max field to Reducers for the SumStats::UNIQUE calculation, and using the new option in scan.bro and the FTP bruteforce detection. (Seth Hall) 2.1-1137 | 2013-08-27 13:26:44 -0700 * Add BiF hexstr_to_bytestring() that does exactly the opposite of bytestring_to_hexstr(). (Johanna Amann) 2.1-1135 | 2013-08-27 12:16:26 -0700 * More SumStats fixes. (Seth Hall) * Increase UIDs to 96 bits. (Jon Siwek) - The bit-length is adjustable via redef'ing bits_per_uid. - Prefix 'C' is added to connection UIDS (including IP tunnels) and 'F' to files. Addresses BIT-1016. 2.1-1128 | 2013-08-24 10:27:29 -0700 * Remove code relict in input framework. (Jon Siwek) * Fix documentation for mkdir BIF. (Jon Siwek) * File extraction tweaks. (Jon Siwek) - Default extraction limit of 100MB now provided via a tuning script loaded in local.bro so that command-line Bro is unlimited by default. - Extraction directory is now created on request of file extraction rather than unconditionally in bro_init(). (Jon Siwek) 2.1-1124 | 2013-08-23 16:33:52 -0700 * Fixed a number of object bugs DNP3 analyzer. (Hui Lin) 2.1-1122 | 2013-08-22 16:52:27 -0700 * Use macros to create file analyzer plugin classes. (Jon Siwek) * Add options to limit extracted file sizes w/ 100MB default. (Jon Siwek) 2.1-1117 | 2013-08-22 08:44:12 -0700 * A number of input framework fixes and corresponding test stability improvements. (Jon Siwek) * Make memory leak tests able to time out. (Jon Siwek) * Fix a compiler warning regarding strncat misuse. (Jon Siwek) 2.1-1103 | 2013-08-21 19:11:34 -0400 * A number of sumstats fixes. (Seth Hall, Vlad Grigorescu) * Fix memory leak w/ when statements. Addresses BIT-1058. (Jon Siwek) * Switching to relative submodule paths (Robin Sommer) 2.1-1089 | 2013-08-19 11:25:11 -0700 * Fix bloom filters' dependence on size_t. (Jon Siwek, Matthias Vallentin). 2.1-1081 | 2013-08-19 11:19:33 -0700 * New BiF levenshtein_distance() to compute the Levenshtein distance between two strings. (Anthony Kasza) 2.1-1078 | 2013-08-19 09:29:30 -0700 * Moving sqlite code into new external 3rdparty submodule. Johanna( Amann) 2.1-1074 | 2013-08-14 10:29:54 -0700 * Fix timer type enum and timer name array mismatch. (Jon Siwek) 2.1-1072 | 2013-08-14 10:28:51 -0700 * Adding the unified2 analyzer that reads unified2 files from disk, turning them into events. (Seth Hall) * Fixing intel framework tests. (Seth Hall) 2.1-1059 | 2013-08-13 23:52:41 -0400 * Add file name support to intel framework. (Seth Hall) * Add file support to intel framework and slightly restructure intel http handling. (Seth Hall) 2.1-1052 | 2013-08-12 14:38:14 -0700 * Fixing bug in DNP3 analyzer flagged by compiler warning. (Robin Sommer) 2.1-1050 | 2013-08-12 11:37:44 -0700 * Experimental DNP3 analyzer. This includes only very basic script-level support at the moment, but quite a number of events are provided. (Hui Lin, Robin Sommer) 2.1-1041 | 2013-08-09 15:32:22 -0700 * Update coverage baselines for canonical load order of scripts. (Jon Siwek) 2.1-1039 | 2013-08-09 15:30:15 -0700 * Fix mem leak in DHCP analyzer. (Jon Siwek) * Fix a unit test outdated by recent sumstats changes. (Jon Siwek) 2.1-1036 | 2013-08-05 17:29:11 -0400 * Fix the SSL infinite loop I just created. (Seth Hall) 2.1-1035 | 2013-08-05 16:44:50 -0400 * Change to SSL log delay to cause the log to write even if delay times out. (Seth Hall) 2.1-1034 | 2013-08-03 20:27:43 -0700 * A set of DHCP extensions. (Vlad Grigorescu) - Leases are logged to dhcp.log as they are seen. - scripts/policy/protocols/dhcp/known-devices-and-hostnames.bro - Added DPD sig. 2.1-1027 | 2013-08-03 01:57:37 -0400 * Fix a major memory issue in the SumStats framework. 2.1-1026 | 2013-08-02 22:35:09 -0400 * Fix the SumStats top-k plugin and test. (Seth Hall) * Rework of SumStats API to reduce high instantaneous memory use on clusters. (Seth Hall) * Large update for the SumStats framework. - On-demand access to sumstats results through "return from" functions named SumStats::request and Sumstats::request_key. Both functions are tested in standalone and clustered modes. - $name field has returned to SumStats which simplifies cluster code and makes the on-demand access stuff possible. - Clustered results can only be collected for 1 minute from their time of creation now instead of time of last read. - Thresholds use doubles instead of counts everywhere now. - Calculation dependency resolution occurs at start up time now instead of doing it at observation time which provide a minor cpu performance improvement. A new plugin registration mechanism was created to support this change. - AppStats now has a minimal doc string and is broken into hook-based plugins. - AppStats and traceroute detection added to local.bro (Seth Hall) 2.1-1009 | 2013-08-02 17:19:08 -0700 * A number of exec module and raw input reader fixes. (Jon Siwek) 2.1-1007 | 2013-08-01 15:41:54 -0700 * More function documentation. (Johanna Amann) 2.1-1004 | 2013-08-01 14:37:43 -0700 * Adding a probabilistic data structure for computing "top k" elements. (Johanna Amann) The corresponding functions are: topk_init(size: count): opaque of topk topk_add(handle: opaque of topk, value: any) topk_get_top(handle: opaque of topk, k: count) topk_count(handle: opaque of topk, value: any): count topk_epsilon(handle: opaque of topk, value: any): count topk_size(handle: opaque of topk): count topk_sum(handle: opaque of topk): count topk_merge(handle1: opaque of topk, handle2: opaque of topk) topk_merge_prune(handle1: opaque of topk, handle2: opaque of topk) 2.1-971 | 2013-08-01 13:28:32 -0700 * Fix some build errors. (Jon Siwek) * Internal refactoring of how plugin components are tagged/managed. (Jon Siwek) * Fix various documentation, mostly related to file analysis. (Jon Siwek) * Changing the Bloom filter hashing so that it's independent of CompositeHash. (Robin Sommer) 2.1-951 | 2013-08-01 11:19:23 -0400 * Small fix to deal with a bug in the SSL log delay mechanism. 2.1-948 | 2013-07-31 20:08:28 -0700 * Fix segfault caused by merging an empty bloom-filter with a bloom-filter already containing values. (Johanna Amann) 2.1-945 | 2013-07-30 10:05:10 -0700 * Make hashers serializable. (Matthias Vallentin) * Add docs and use default value for hasher names. (Matthias Vallentin) 2.1-939 | 2013-07-29 15:42:38 -0700 * Added Exec, Dir, and ActiveHTTP modules. (Seth Hall) base/utils/exec.bro provides a module to start external processes asynchronously and retrieve their output on termination. base/utils/dir.bro uses it to monitor a directory for changes, and base/utils/active-http.bro for providing an interface for querying remote web servers. 2.1-930 | 2013-07-29 15:06:07 -0700 * Major file analysis overhaul in naming and appearance, along with fixes and test updates. (Seth Hall and Jon Siwek) Includes: * Added protocol description functions that provide a super compressed log representation. (Seth Hall) * Added mime types to http.log (Seth Hall) * Add jar files to the default MHR lookups. (Seth Hall) * Adding CAB files for MHR checking. (Seth Hall) * Improve malware hash registry script. - Include a link to a virustotal search in the notice sub message field. - Give all information returned from Team Cymru in the notice message. - Add more file types to match on to the default set. * Make the custom libmagic database a git submodule. * Add an is_orig parameter to file_over_new_connection event. * Recorrected the module name to Files. * Added Files::analyzer_name to get a more readable name for a file analyzer. * Improved and just overall better handled multipart mime transfers in HTTP and SMTP. HTTP now has orig_fuids and resp_fuids log fields since multiple "files" can be transferred with multipart mime in a single request/response pair. SMTP has an fuids field which has file unique IDs for all parts transferred. FTP and IRC have a log field named fuid added because only a single file can be transferred per irc and ftp log line. 2.1-895 | 2013-07-29 14:07:35 -0700 * Adding a test for a DNSKEY RR. (Robin Sommer) 2.1-894 | 2013-07-29 16:44:41 -0400 * Updates for the Intel Framework. (Seth Hall) - policy/frameworks/intel/seen is the new location for the scripts that push data into the intel framework for checking. - The new policy/frameworks/intel/do_notice script adds an example mechanism for data driven notices. - Remove the Intel insertion after heuristically detecting SSH bruteforcing. - Intel importing format has changed (refer to docs). - All string matching is now case insensitive. - SMTP intel script has been updated to extract email addresses correctly. - Small fix sneaking into the smtp base script to actually extract individual email addresses in the To: field correctly. 2.1-888 | 2013-07-25 12:02:41 -0700 * Protection about broken traces with empty pcap headers. (Matt Thompson) 2.1-887 | 2013-07-25 11:33:27 -0700 * Support for Bloom filter. (Matthias Vallentin) Bro now provides the following BiFs: bloomfilter_basic_init(fp: double, capacity: count, name: string &default=""): opaque of bloomfilter bloomfilter_counting_init(k: count, cells: count, max: count, name: string &default=""): opaque of bloomfilter bloomfilter_add(bf: opaque of bloomfilter, x: any) bloomfilter_lookup(bf: opaque of bloomfilter, x: any): count bloomfilter_merge(bf1: opaque of bloomfilter, bf2: opaque of bloomfilter): opaque of bloomfilter bloomfilter_clear(bf: opaque of bloomfilter) Note that currently Bloom filters from separate Bro instances (e.g., from different cluster nodes) cannot be merged. 2.1-826 | 2013-07-25 10:12:26 -0700 * bif files declared with bif_target() are now automatically compiled in. No more manual includes to pull them in. (Robin Sommer) * Covenience make target in testing/btest to update the three coverage tests that usually need tweaking when scripts get added/removed. (Robin Sommer) 2.1-824 | 2013-07-22 14:25:14 -0400 * Fixed a scriptland state issue that manifested especially badly on proxies. (Seth Hall) * Another test fix. (Robin Sommer) * Canonyfying the output of core.print-bpf-filters. (Robin Sommer) 2.1-820 | 2013-07-18 12:30:04 -0700 * Extending external canonifier to remove fractional values from capture_loss.log. (Robin Sommer) * Canonifying internal order for plugins and their components to make it deterministic. (Robin Sommer) * Small raw reader tweaks that got left our earlier. (Robin Sommer) 2.1-814 | 2013-07-15 18:18:20 -0700 * Fixing raw reader crash when accessing nonexistant file, and memory leak when reading from file. Addresses #1038. (Johanna Amann) 2.1-811 | 2013-07-14 08:01:54 -0700 * Bump sqlite to 3.7.17. (Johanna Amann) * Small test fixes. (Seth Hall) * Fix a bug where the same analyzer tag was reused for two different analyzers. (Seth Hall) * Moved DPD signatures into script specific directories. Left out the BitTorrent signatures pending further updates to that analyzer. (Seth Hall) 2.1-802 | 2013-07-10 10:55:14 -0700 * Const adjustment for methods. (Jon Siwek) 2.1-798 | 2013-07-08 13:05:37 -0700 * Rewrite of the packet filter framework. (Seth Hall) This includes: - Plugin interface for adding filtering mechanisms. - Integrated the packet filter framework with the analyzer framework to retrieve well-known ports from there. - Support for BPF-based load balancing (IPv4 and IPv6). This will tie in with upcoming BroControl support for configuring this. - Support for BPF-based connection sampling. - Support for "shunting" traffic with BPF filters. - Replaced PacketFilter::all_packets with PacketFilter::enable_auto_protocol_capture_filters. 2.1-784 | 2013-07-04 22:28:48 -0400 * Add a call to lookup_connection in SSH scripts to update connval. (Seth Hall) * Updating submodule(s). (Robin Sommer) 2.1-782 | 2013-07-03 17:00:39 -0700 * Remove the SSL log queueing mechanism that was included with the log delay mechanism. (Seth Hall) 2.1-780 | 2013-07-03 16:46:26 -0700 * Rewrite of the RAW input reader for improved robustness and new features. (Johanna Amann) This includes: - Send "end_of_data" event for all kind of streams. - Send "process_finished" event with exit code of child process at process termination. - Expose name of input stream to readers. - Better error handling. - New "force_kill" option which SIGKILLs processes on reader termination. - Supports reading from stdout and stderr simultaneously. - Support sending data to stdin of child process. - Streaming reads from external commands work without blocking. 2.1-762 | 2013-07-03 16:33:22 -0700 * Fix to correct support for TLS 1.2. Addresses #1020. (Seth Hall, with help from Rafal Lesniak). 2.1-760 | 2013-07-03 16:31:36 -0700 * Teach broxygen to generate protocol analyzer plugin reference. (Jon Siwek) * Adding 'const' to a number of C++ methods. (Jon Siwek) 2.1-757 | 2013-07-03 16:28:10 -0700 * Fix redef of table index from clearing table. `redef foo["x"] = 1` now acts like `redef foo += { ["x"] = 1 }` instead of `redef foo = { ["x"] = 1 }`. Addresses #1013. (Jon Siwek) 2.1-755 | 2013-07-03 16:22:43 -0700 * Add a general file analysis overview/how-to document. (Jon Siwek) * Improve file analysis doxygen comments. (Jon Siwek) * Improve tracking of HTTP file extraction. http.log now has files taken from request and response bodies in different fields for each, and can now track multiple files per body. That is, the "extraction_file" field is now "extracted_request_files" and "extracted_response_files". Addresses #988. (Jon Siwek) * Fix HTTP multipart body file analysis. Each part now gets assigned a different file handle/id. (Jon Siwek) * Remove logging of analyzers field of FileAnalysis::Info. (Jon Siwek) * Remove extraction counter in default file extraction scripts. (Jon Siwek) * Remove FileAnalysis::postpone_timeout. FileAnalysis::set_timeout_interval can now perform same function. (Jon Siwek) * Make default get_file_handle handlers &priority=5 so they're easier to override. (Jon Siwek) * Add input interface to forward data for file analysis. The new Input::add_analysis function is used to automatically forward input data on to the file analysis framework. (Jon Siwek) * File analysis framework interface simplifications. (Jon Siwek) - Remove script-layer data input interface (will be managed directly by input framework later). - Only track files internally by file id hash. Chance of collision too small to justify also tracking unique file string. 2.1-741 | 2013-06-07 17:28:50 -0700 * Fixing typo that could cause an assertion to falsely trigger. (Robin Sommer) 2.1-740 | 2013-06-07 16:37:32 -0700 * Fix for CMake 2.6.x. (Robin Sommer) 2.1-738 | 2013-06-07 08:38:13 -0700 * Remove invalid free on non-allocated pointer in hash function object. Addresses #1018. (Matthias Vallentin) 2.1-736 | 2013-06-06 10:05:20 -0700 * New "magic constants" @DIR and @FILENAME that expand to the directory path of the current script and just the script file name without path, respectively. (Jon Siwek) 2.1-731 | 2013-06-04 21:19:08 -0700 * Reorginization of internal protocol analyzer code. We're moving them to a modularized structure, based on a plugin model. Along with this change comes generic plugin infrastructure that we'll later extend to other Bro component as well. For now all plugins are compiled in statically, but in the future we plan to also enable dynamic loading at run time. (Robin Sommer) * Ignoring file ids in external tests. (Robin Sommer) 2.1-675 | 2013-06-02 20:03:19 -0700 * Fix a compiler warning. (Robin Sommer) * Allow named vector/set/table/record constructors. Addresses #983. (Jon Siwek) * Adding Makefile target test-all that also runs the BroControl test suite. (Robin Sommer) 2.1-664 | 2013-05-28 21:37:46 -0700 * Dangling pointer fix. Addresses #1004. (Jon Siwek) 2.1-659 | 2013-05-24 17:24:18 -0700 * Fix broken/missing documentation. (Jon Siwek) * Fixing test that would fail without ES/curl support. (Robin Sommer) 2.1-656 | 2013-05-17 15:58:07 -0700 * Fix mutex lock problem for writers. (Johanna Amann) 2.1-654 | 2013-05-17 13:49:52 -0700 * Tweaks to sqlite3 configuration to address threading issues. (Johanna Amann) 2.1-651 | 2013-05-17 13:37:16 -0700 * Fix uninitialized DPM member. (Jon Siwek) * Fix issue with transaction ID reuse in a single DNS connection. (Seth Hall) * New function added to the queue.bro script to support peeking at the new gettable item in the queue without removing it. (Seth Hall) 2.1-647 | 2013-05-17 07:47:14 -0700 * Fixing Broxygen generation to have BROMAGIC set. (Robin Sommer) * Fix for 'fchmod undeclared here' on FreeBSD. (Robin Sommer) * CMake policy fix to avoid errors with older versions. (Robin Sommer) 2.1-641 | 2013-05-15 18:15:09 -0700 * Test update. (Robin Sommer) 2.1-640 | 2013-05-15 17:24:09 -0700 * Support for cleaning up threads that have terminated. (Johanna Amann and Robin Sommer). Includes: - Both logging and input frameworks now clean up threads once they aren't further needed anymnore. - New function Log::remove_stream() that removes a logging stream, stopping all writer threads that are associated with it. Note, however, that removing a *filter* from a stream still doesn't clean up any threads. The problem is that because of the output paths potentially being created dynamically it's unclear if the writer thread will still be needed in the future. 2.1-626 | 2013-05-15 16:09:31 -0700 * Add "reservoir" sampler for SumStats framework. This maintains a set of N uniquely distributed random samples. (Johanna Amann) 2.1-619 | 2013-05-15 16:01:42 -0700 * SQLite reader and writer combo. This allows to read/write persistent data from on disk SQLite databases. The current interface is quite low-level, we'll add higher-level abstractions in the future. (Johanna Amann) 2.1-576 | 2013-05-15 14:29:09 -0700 * Initial version of new file analysis framework. This moves most of the processing of file content from script-land into the core, where it belongs. Much of this is an internal change, and at this point the new code has essentially feature-equality with the old one. More script-level changes to come. (Jon Siwek) 2.1-502 | 2013-05-10 19:29:37 -0700 * Allow default function/hook/event parameters. Addresses #972. (Jon Siwek) * Change the endianness parameter of bytestring_to_count() BIF to default to false (big endian). (Jon Siwek) 2.1-500 | 2013-05-10 19:22:24 -0700 * Fix to prevent merge-hook of SumStat's unique plugin from damaging source data. (Johanna Amann) 2.1-498 | 2013-05-03 17:44:08 -0700 * Table lookups return copy of non-const &default vals. This prevents unintentional modifications to the &default value itself. Addresses #981. (Jon Siwek) 2.1-496 | 2013-05-03 15:54:47 -0700 * Fix memory leak and unnecessary allocations in OpaqueVal. Addresses #986. (Matthias Vallentin) 2.1-492 | 2013-05-02 12:46:26 -0700 * Work-around for sumstats framework not propagating updates after intermediate check in cluster environments. (Johanna Amann) * Always apply tcp_connection_attempt. Before this change it was only applied when a connection_attempt() event handler was defined. (Robin Sommer) * Fixing coverage.bare-mode-errors test. (Robin Sommer) 2.1-487 | 2013-05-01 18:03:22 -0700 * Always apply tcp_connection_attempt timer, even if no connection_attempt() event handler is defined. (Robin Sommer) 2.1-486 | 2013-05-01 15:28:45 -0700 * New framework for computing summary statistics in base/framework/sumstats. This replaces the metrics frameworks, and comes with a number of applications build on top, see NEWS. More documentation to follow. (Seth Hall) 2.1-397 | 2013-04-29 21:19:00 -0700 * Fixing memory leaks in CompHash implementation. Addresses #987. (Robin Sommer) 2.1-394 | 2013-04-27 15:02:31 -0700 * Fixed a bug in the vulnerable software script and added a test. (Seth Hall) * Fix schedule statements used outside event handlers. Addresses #974. (Jon Siwek) * Fix record coercion for default inner record fields. Addresses #973. (Jon Siwek) * Add bytestring_to_count function to bro.bif. Addresses #968. (Yun Zheng Hu) 2.1-386 | 2013-03-22 12:41:50 -0700 * Added reverse() function to strings.bif. (Yun Zheng Hu) 2.1-384 | 2013-03-22 12:10:14 -0700 * Fix record constructors in table initializer indices. Addresses #660. (Jon Siwek) 2.1-382 | 2013-03-22 12:01:34 -0700 * Add support for 802.1ah (Q-in-Q). Addresses #641. (Seth Hall) 2.1-380 | 2013-03-18 12:18:10 -0700 * Fix gcc compile warnings in base64 encoder and benchmark reader. (Johanna Amann) 2.1-377 | 2013-03-17 17:36:09 -0700 * Fixing potential leak in DNS error case. (Vlad Grigorescu) 2.1-375 | 2013-03-17 13:14:26 -0700 * Add base64 encoding functionality, including new BiFs encode_base64() and encode_base64_custom(). (Johanna Amann) * Replace call to external "openssl" in extract-certs-pem.bro with that encode_base64(). (Johanna Amann) * Adding a test for extract-certs-pem.pem. (Robin Sommer) * Renaming Base64Decoder to Base64Converter. (Robin Sommer) 2.1-366 | 2013-03-17 12:35:59 -0700 * Correctly handle DNS lookups for software version ranges. (Seth Hall) * Improvements to vulnerable software detection. (Seth Hall) - Add a DNS based updating method. This needs to be tested still. - Vulnerable version ranges are used now instead of only single versions. This can deal with software with multiple stable major versions. * Update software version parsing and comparison to account for a third numeric subversion. Also, $addl is now compared numerically if the value is actually numeric. (Seth Hall) 2.1-361 | 2013-03-13 07:18:22 -0700 * Add check for truncated link frames. Addresses #962. (Jacob Baines) * Fix large memory allocation in IP fragment reassembly. Addresses #961. (Jacob Baines) 2.1-357 | 2013-03-08 09:18:35 -0800 * Fix race-condition in table-event test. (Johanna Amann) * s/bro-ids.org/bro.org/g. (Robin Sommer) 2.1-353 | 2013-03-07 13:31:37 -0800 * Fix function type-equivalence requiring same parameter names. Addresses #957. (Jon Siwek) 2.1-351 | 2013-03-07 13:27:29 -0800 * Fix new/delete mismatch. Addresses #958. (Jacob Baines) * Fix compiler warnings. (Jon Siwek) 2.1-347 | 2013-03-06 16:48:44 -0800 * Remove unused parameter from vector assignment method. (Johanna Amann) * Remove the byte_len() and length() bifs. (Johanna Amann) 2.1-342 | 2013-03-06 15:42:52 -0800 * Moved the Notice::notice event and Notice::policy table to both be hooks. See documentation and NEWS for information. (Seth Hall). 2.1-338 | 2013-03-06 15:10:43 -0800 * Fix init of local sets/vectors via curly brace initializer lists. (Jon Siwek) 2.1-336 | 2013-03-06 15:08:06 -0800 * Fix memory leaks resulting from 'when' and 'return when' statements. Addresses #946. (Jon Siwek) * Fix three bugs with 'when' and 'return when' statements. Addresses #946. (Jon Siwek) 2.1-333 | 2013-03-06 14:59:47 -0800 * Add parsing for GTPv1 extension headers and control messages. (Jon Siwek) This includes: - A new generic gtpv1_message() event generated for any GTP message type. - Specific events for the create/update/delete PDP context request/response messages. Addresses #934. 2.1-331 | 2013-03-06 14:54:33 -0800 * Fix possible null pointer dereference in identify_data BIF. Also centralized libmagic calls for consistent error handling/output. (Jon Siwek) * Fix build on OpenBSD 5.2. (Jon Siwek) 2.1-328 | 2013-02-05 01:34:29 -0500 * New script to query the ICSI Certificate Notary (http://notary.icsi.berkeley.edu/) over DNS and add information to the SSL log at runtime. (Matthias Vallentin) * Add delayed logging to SSL base scripts. (Matthias Vallentin) 2.1-319 | 2013-02-04 09:45:34 -0800 * Update input tests to use exit_only_after_terminate. (Johanna Amann) * New option exit_only_after_terminate to prevent Bro from exiting. If set, the main loop won't terminate before somebody calls terminate(). (Robin Sommer) 2.1-311 | 2013-02-01 08:03:01 -0800 * Updating submodule(s). 2.1-310 | 2013-01-30 20:09:27 -0800 * Add an error for record coercions that would orphan a field. (Jon Siwek) * Fixing several scripts where a field in an inlined record was never removed after a code refactor. (Jon Siwek) 2.1-307 | 2013-01-25 13:50:57 -0800 * Fix runaway reference counting bug in record coercion. (Jon Siwek) * Fix memory leak in some reporter messaging cases. (Jon Siwek) 2.1-304 | 2013-01-23 19:43:27 -0800 * Making a test portable. (Robin Sommer) 2.1-302 | 2013-01-23 16:17:29 -0800 * Refactoring ASCII formatting/parsing from loggers/readers into a separate AsciiFormatter class. (Johanna Amann) * Fix uninitialized locals in event/hook handlers from having a value. Addresses #932. (Jon Siwek) * Add a null value check in CompositeHash::ComputeHash. Addresses #930. (Jon Siwek) * Change reporter messages to more reliably print to stderr. Addressed #930 (and revisits #836). (Jon Siwek) * Changing test=suite's btest call to use "-j" instead of "-j 5". (Robin Sommer) * Require "case" blocks to end with either "break", "return", or a new "fallthrough" statement that passes control on to the subsequent case. This gives us the best mix of safety, readability, and flexibility. Addresses #754. (Jon Siwek) 2.1-279 | 2013-01-18 17:18:22 -0800 * Revert "Trick for parallelizing input framework unit tests." The old way of doing the tests seems more reliable for now. (Jon Siwek) * Fixing variable size issues with http response code in ElasticSearch writer. (Gilbert Clark) * Removing unused class member. (Robin Sommer) * Add opaque type-ignoring for the accept_unsupported_types input framework option. (Johanna Amann) 2.1-271 | 2013-01-08 10:18:57 -0800 * Change substring index notation to use a colon. String slice notation is now written as `s[1:2]`. Addresses #422. (Jon Siwek) 2.1-268 | 2013-01-07 09:43:44 -0800 * Fix memory leak in OpaqueType::DoUnserialize. (Jon Siwek) 2.1-265 | 2012-12-20 17:38:42 -0800 * Add array-style index accessor for strings. Addresses #422. (Jon Siwek) The index expression can take up to two indices for the start and end index of the substring to return (e.g. "mystring[1,3]"). Negative indices are allowed, with -1 representing the last character in the string. The indexing is not cyclic -- if the starting index is >= the length of the string an empty string is returned, and if the ending index is >= the length of the string then it's interpreted as the last index of the string. Assigning to substrings accessed like this isn't allowed. 2.1-263 | 2012-12-20 16:22:09 -0800 * Bro's language now has a new set of types "opaque of X". (Matthias Vallentin) Opaque values can be passed around like other values but they can only be manipulated with BiF functions, not with other operators. Currently, the following opaque types are supported: - opaque of md5 - opaque of sha1 - opaque of sha256 - opaquey of entropy. They go along with the corrsponding BiF functions md5_*, sha1_*, sha256_*, and entropy_*, respectively. Note that these functions have changed their signatures to work with opaques types rather than global state as it was before. 2.1-240 | 2012-12-20 15:21:07 -0800 * Improve error for invalid use of types as values. Addresses #923. (Jon Siwek) 2.1-238 | 2012-12-20 15:11:25 -0800 * Finish implementation of script-layer switch statement. Addresses #754. (Jon Siwek) They behave like C-style switches except case labels can be comprised of multiple literal constants delimited by commas. Only atomic types are allowed for now. Case label bodies that don't execute a "return" or "break" statement will fall through to subsequent cases. A default case label is allowed. * Fix a case where c$resp$size is misrepresented. Addresses #730. (Jon Siwek) 2.1-234 | 2012-12-20 12:12:19 -0800 * Fix return value of hook calls that have no handlers. For this case, the return value is always true. (Jon Siwek) * Fix to_port() BIF for port strings with a port number of zero. (Jon Siwek) 2.1-231 | 2012-12-14 14:51:35 -0800 * Make const variables actually constant. Both local and global variables declared with "const" could be modified, but now expressions that would modify them generate an error message at parse-time. Addresses #922. (Jon Siwek) 2.1-229 | 2012-12-14 14:46:12 -0800 * Fix memory leak in ASCII reader when encoutering errors in input. (Johanna Amann) * Improvements for the "bad checksums" detector to make it detect bad TCP checksums. (Seth Hall) 2.1-223 | 2012-12-12 14:25:15 -0800 * Trick for parallelizing input framework unit tests. Instead of loading listen.bro to block until files are read, just read a pcap file in pseudo-realtime. (Jon Siwek) * Fix reliability of a unit test that relies on when statements. (Jon Siwek) * Remove unused attributes. (Daniel Thayer) - Removed attributes &postprocessor and &match from documentation and source code. - Removed undocumented attribute &attr from source code. - Removed internal attribute "(&tracked)" from documentation. 2.1-218 | 2012-12-10 14:45:04 -0800 * Add GPRS Tunnelling Protocol (GTPv1) decapsulation. This currently supports automatic decapsulation of GTP-U packets on UDP port 2152. The GTPv1 headers for such tunnels can be inspected by handling the "gtpv1_g_pdu_packet" event, which has a parameter of type "gtpv1_hdr". Addresses #690. (Jon Siwek; derived from patch by Carsten Langer) * Change BinPAC exceptions in AYIYA/GTP analyzers to do "protocol_violation". (Jon Siwek) 2.1-212 | 2012-12-07 19:42:03 -0800 * Changing the HTTP parser to accept request methods in alignment with the RFC. (Robin Sommer) 2.1-209 | 2012-12-05 16:44:04 -0800 * Adapting the HTTP request line parsing to only accept methods consisting of letters [A-Za-z]. (Robin Sommer) 2.1-207 | 2012-12-05 15:47:32 -0800 * Reporting warnings if kill/waitpid fail in communication system. (Bill Parker) * Replace() bzero with memset(). (Bill Parker) * Merge remote-tracking branch 'vlad/topic/vladg/http-verbs' * vlad/topic/vladg/http-verbs: A test for HTTP methods, including some horribly illegal requests. Remove hardcoded HTTP verbs from the analyzer (#741) I added a "bad_HTTP_request" weird for HTTP request lines that don't have more than a single word. Closes #741. (Robin Sommer) * A test for HTTP methods, including some horribly illegal requests. (Vlad Grigorescu) * Remove hardcoded HTTP verbs from the analyzer (#741) (Vlad Grigorescu) 2.1-203 | 2012-12-05 14:36:56 -0800 * Fix segfault: Synchronization of state between connecting peers now skips over identifiers that aren't initialized with a value yet. Addresses #66. (Jon Siwek) * Fix segfault: Delete correct entry in error case in input framework. (Johanna Amann) * Bad record constructor initializers now give an error. Addresses #34. (Jon Siwek) * Invalid vector indices now generate error message. Addresses #24. (Jon Siwek) * Bump CPack RPM package requirement to Python >= 2.6.0. (Jon Siwek) * Interpreter exceptions occurring in "when" blocks are now handled. Addresses #779 (Jon Siwek) 2.1-195 | 2012-12-03 14:50:33 -0800 * Catching out-of-memory in patricia tree code. (Bill Parker) 2.1-194 | 2012-12-03 14:36:26 -0800 * Renaming ASCII writer filter option 'only_single_header_row' to 'tsv'. Also clarifying usage. Closes #912. (Robin Sommer) 2.1-193 | 2012-12-03 14:11:14 -0800 * Fix a set of bugs with table/set attributes. (Jon Siwek) - Identifiers that are initialized with set()/table() constructor expressions now inherit attributes from the expression. Before, statements like const i: set[string] = set() &redef; associated the attribute with the set() constructor, but not the "i" identifier, preventing redefinition. Addresses #866. - Allow &default attribute to apply to tables initialized as empty (via either "{ }" or "table()") or if the expression supplied to it can evaluate to a type that's promotable to the same yield type as the table. 2.1-191 | 2012-12-03 14:08:56 -0800 * Add test of record() constructor to table initializer unit test. (Jon Siwek) * Fix table(), set(), vector() constructors in table initializer lists. Also adds type checking of yield values to table() constructor and fixes the type checking of yield values in vector() constructor. Addresses #5. (Jon Siwek) 2.1-188 | 2012-12-03 14:04:29 -0800 * Hook functions now callable with "hook" expression (i.e., hook is no longer a statement). The return value of the call is an implicit boolean value of T if all hook handlers ran, or F if one hook handler exited as a result of a break statement and potentially prevented other handlers from running. Scripts don't need to declare hooks with an explicit return type of bool (internally, that's assumed), and any values given to (optional) return statements in handler definitions are just ignored. Addresses #918. (Jon Siwek) * Clarification in hook documentation. (Jon Siwek) 2.1-184 | 2012-12-03 13:59:50 -0800 * Slightly fix up file name extraction from Content-Disposition headers. (Seth Hall) * Adding -b flag to bro in unit tests so they run faster. * Fixed a DNS attribute issue. Reported by Matt Thompson. (Seth Hall) * Adding NEWS placeholder for hooks and CSV mode. (Robin Sommer) 2.1-178 | 2012-11-23 19:35:32 -0800 * The ASCII writer now supports a new filter config option "only_single_header_row" that turns the output into CSV format when set to "T". (Carsten Langer) * Add new function flavor called a "hook". This new flavor of function behaves like a "synchronous event". See doc/scripts/builtins.rst more details on usage. (Jon Siwek) * Improve auto-generated enum documentation. The names of enum types are tracked so that variables holding a value of a given enum type can generate a reference to it instead of just listing the type as a generic "enum". (Jon Siwek) 2.1-171 | 2012-11-23 18:24:15 -0800 * Fix ambiguity between composite table index and record ctor expressions. If a table type is "global t = table[conn_id, bool] of count", then checking membership like "[c$id, is_orig] in t" now works. Addresses #80. (Jon Siwek) 2.1-169 | 2012-11-23 18:21:32 -0800 * Fix some warnings from sphinx when building docs. (Jon Siwek) 2.1-167 | 2012-11-14 13:19:17 -0800 * Add a new BIF "bytestring_to_double" for converting from a binary representation of a double. Addresses #908. (Carsten Langer/Daniel Thayer) 2.1-162 | 2012-11-13 17:29:00 -0800 * Fix modbus register array parsing. (Jon Siwek) * Adjustments to modbus test cases. (Jon Siwek) 2.1-157 | 2012-11-08 16:22:00 -0800 * Fix for lookup_hostname BIF. (Jon Siwek) * Fix for modbus test portability. (Robin Sommer) 2.1-152 | 2012-11-05 16:52:34 -0800 * Initial version of a completely reworked intelligence framework. See doc/intel.rst for more information. (Seth Hall) * Experimental Modbus analyzer. See policy/protocols/modbus/* for example policies. (Dina Hadziosmanovic, Seth Hall) 2.1-112 | 2012-11-05 13:58:20 -0800 * New base script for detecting cases of checksum offloading. Reporter messages will now tell if one has bad checksums. (Seth Hall) * Clarifying ownership rules for BroString constructors. (Robin Sommer) 2.1-109 | 2012-11-05 13:39:34 -0800 * Add detection rate threshold for MHR. (Vlad Grigorescu) * lookup_hostname_txt fixes. (Vlad Grigorescu) 2.1-104 | 2012-11-01 10:37:50 -0700 * A new built-in function lookup_hostname_txt() provides support for DNS TXT queries. (Vlad Grigorescu) 2.1-101 | 2012-10-31 14:30:26 -0700 * Documentation reorg: The install info has been consolidated into a single document (INSTALL), the upgrade info has been moved from the FAQ to a section in the install doc, and the "upgrading from 1.5 to 2.0" document has been updated (and renamed) to also include 2.0 to 2.1 upgrade info. (Daniel Thayer) 2.1-96 | 2012-10-31 14:23:50 -0700 * Renaming option defining the frequency of alarm summary mails to 'Logging::default_alarm_mail_interval'. (Daniel Thayer) 2.1-91 | 2012-10-24 16:04:47 -0700 * Adding PPPoE support to Bro. (Seth Hall) 2.1-87 | 2012-10-24 15:40:06 -0700 * Adding missing &redef for some TCP options. Addresses #905, #906, #907. (Carsten Langer) 2.1-86 | 2012-10-24 15:37:11 -0700 * Add parsing rules for IPv4/IPv6 subnet literal constants. Addresses #888. (Jon Siwek) 2.1-84 | 2012-10-19 15:12:56 -0700 * Added a BiF strptime() to wrap the corresponding C function. (Seth Hall) 2.1-82 | 2012-10-19 15:05:40 -0700 * Add IPv6 support to signature header conditions. (Jon Siwek) - "src-ip" and "dst-ip" conditions can now use IPv6 addresses/subnets. They must be written in colon-hexadecimal representation and enclosed in square brackets (e.g. [fe80::1]). Addresses #774. - "icmp6" is now a valid protocol for use with "ip-proto" and "header" conditions. This allows signatures to be written that can match against ICMPv6 payloads. Addresses #880. - "ip6" is now a valid protocol for use with the "header" condition. (also the "ip-proto" condition, but it results in a no-op in that case since signatures apply only to the inner-most IP packet when packets are tunneled). This allows signatures to match specifically against IPv6 packets (whereas "ip" only matches against IPv4 packets). - "ip-proto" conditions can now match against IPv6 packets. Before, IPv6 packets were just silently ignored which meant DPD based on signatures did not function for IPv6 -- protocol analyzers would only get attached to a connection over IPv6 based on the well-known ports set in the "dpd_config" table. 2.1-80 | 2012-10-19 14:48:42 -0700 * Change how "gridftp" gets added to service field of connection records. In addition to checking for a finished SSL handshake over an FTP connection, it now also requires that the SSL handshake occurs after the FTP client requested AUTH GSSAPI, more specifically identifying the characteristics of GridFTP control channels. Addresses #891. (Jon Siwek) * Allow faster rebuilds in certain cases. Previously, when rebuilding with a different "--prefix" or "--scriptdir", all Bro source files were recompiled. With this change, only util.cc is recompiled. (Daniel Thayer) 2.1-76 | 2012-10-12 10:32:39 -0700 * Add support for recognizing GridFTP connections as an extension to the standard FTP analyzer. (Jon Siwek) This is enabled by default and includes: - An analyzer for GSI mechanism of GSSAPI FTP AUTH method. GSI authentication involves an encoded TLS/SSL handshake over the FTP control session. For FTP sessions that attempt GSI authentication, the *service* field of the connection log will include "gridftp" (as well as also "ftp" and "ssl"). - Add an example of a GridFTP data channel detection script. It relies on the heuristics of GridFTP data channels commonly default to SSL mutual authentication with a NULL bulk cipher and that they usually transfer large datasets (default threshold of script is 1 GB). The script also defaults to skip_further_processing() after detection to try to save cycles analyzing the large, benign connection. For identified GridFTP data channels, the *services* fields of the connection log will include "gridftp-data". * Add *client_subject* and *client_issuer_subject* as &log'd fields to SSL::Info record. Also add *client_cert* and *client_cert_chain* fields to track client cert chain. (Jon Siwek) * Add a script in base/protocols/conn/polling that generalizes the process of polling a connection for interesting features. The GridFTP data channel detection script depends on it to monitor bytes transferred. (Jon Siwek) 2.1-68 | 2012-10-12 09:46:41 -0700 * Rename the Input Framework's update_finished event to end_of_data. It will now not only fire after table-reads have been completed, but also after the last event of a whole-file-read (or whole-db-read, etc.). (Johanna Amann) * Fix for DNS log problem when a DNS response is seen with 0 RRs. (Seth Hall) 2.1-64 | 2012-10-12 09:36:41 -0700 * Teach --disable-dataseries/--disable-elasticsearch to ./configure. Addresses #877. (Jon Siwek) * Add --with-curl option to ./configure. Addresses #877. (Jon Siwek) 2.1-61 | 2012-10-12 09:32:48 -0700 * Fix bug in the input framework: the config table did not work. (Johanna Amann) 2.1-58 | 2012-10-08 10:10:09 -0700 * Fix a problem with non-manager cluster nodes applying Notice::policy. This could, for example, result in duplicate emails being sent if Notice::emailed_types is redef'd in local.bro (or any script that gets loaded on all cluster nodes). (Jon Siwek) 2.1-56 | 2012-10-03 16:04:52 -0700 * Add general FAQ entry about upgrading Bro. (Jon Siwek) 2.1-53 | 2012-10-03 16:00:40 -0700 * Add new Tunnel::delay_teredo_confirmation option that indicates that the Teredo analyzer should wait until it sees both sides of a connection using a valid Teredo encapsulation before issuing a protocol_confirmation. Default is on. Addresses #890. (Jon Siwek) 2.1-50 | 2012-10-02 12:06:08 -0700 * Fix a typing issue that prevented the ElasticSearch timeout to work. (Matthias Vallentin) * Use second granularity for ElasticSearch timeouts. (Matthias Vallentin) * Fix compile issues with older versions of libcurl, which don't offer *_MS timeout constants. (Matthias Vallentin) 2.1-47 | 2012-10-02 11:59:29 -0700 * Fix for the input framework: BroStrings were constructed without a final \0, which makes them unusable by basically all internal functions (like to_count). (Johanna Amann) * Remove deprecated script functionality (see NEWS for details). (Daniel Thayer) 2.1-39 | 2012-09-29 14:09:16 -0700 * Reliability adjustments to istate tests with network communication. (Jon Siwek) 2.1-37 | 2012-09-25 14:21:37 -0700 * Reenable some tests that previously would cause Bro to exit with an error. (Daniel Thayer) * Fix parsing of large integers on 32-bit systems. (Daniel Thayer) * Serialize language.when unit test with the "comm" group. (Jon Siwek) 2.1-32 | 2012-09-24 16:24:34 -0700 * Fix race condition in language/when.bro test. (Daniel Thayer) 2.1-26 | 2012-09-23 08:46:03 -0700 * Add an item to FAQ page about broctl options. (Daniel Thayer) * Add more language tests. We now have tests of all built-in Bro data types (including different representations of constant values, and max./min. values), keywords, and operators (including special properties of certain operators, such as short-circuit evaluation and associativity). (Daniel Thayer) * Fix construction of ip6_ah (Authentication Header) record values. Authentication Headers with a Payload Len field set to zero would cause a crash due to invalid memory allocation because the previous code assumed Payload Len would always be great enough to contain all mandatory fields of the header. (Jon Siwek) * Update compile/dependency docs for OS X. (Jon Siwek) * Adjusting Mac binary packaging script. Setting CMAKE_PREFIX_PATH helps link against standard system libs instead of ones that come from other package manager (e.g. MacPorts). (Jon Siwek) * Adjusting some unit tests that do cluster communication. (Jon Siwek) * Small change to non-blocking DNS initialization. (Jon Siwek) * Reorder a few statements in scan.l to make 1.5msecs etc work. Adresses #872. (Johanna Amann) 2.1-6 | 2012-09-06 23:23:14 -0700 * Fixed a bug where "a -= b" (both operands are intervals) was not allowed in Bro scripts (although "a = a - b" is allowed). (Daniel Thayer) * Fixed a bug where the "!=" operator with subnet operands was treated the same as the "==" operator. (Daniel Thayer) * Add sleeps to configuration_update test for better reliability. (Jon Siwek) * Fix a segfault when iterating over a set when using malformed index. (Daniel Thayer) 2.1 | 2012-08-28 16:46:42 -0700 * Make bif.identify_magic robust against FreeBSD's libmagic config. (Robin Sommer) * Remove automatic use of gperftools on non-Linux systems. --enable-perftools must now explicity be supplied to ./configure on non-Linux systems to link against the tcmalloc library. * Fix uninitialized value for 'is_partial' in TCP analyzer. (Jon Siwek) * Parse 64-bit consts in Bro scripts correctly. (Johanna Amann) * Output 64-bit counts correctly on 32-bit machines (Johanna Amann) * Input framework fixes, including: (Johanna Amann) - One of the change events got the wrong parameters. - Escape commas in sets and vectors that were unescaped before tokenization. - Handling of zero-length-strings as last element in a set was broken (sets ending with a ,). - Hashing of lines just containing zero-length-strings was broken. - Make set_separators different from , work for input framework. - Input framework was not handling counts and ints out of 32-bit-range correctly. - Errors in single lines do not kill processing, but simply ignore the line, log it, and continue. * Update documentation for builtin types. (Daniel Thayer) - Add missing description of interval "msec" unit. - Improved description of pattern by clarifying the issue of operand order and difference between exact and embedded matching. * Documentation fixes for signature 'eval' conditions. (Jon Siwek) * Remove orphaned 1.5 unit tests. (Jon Siwek) * Add type checking for signature 'eval' condition functions. (Jon Siwek) * Adding an identifier to the SMTP blocklist notices for duplicate suppression. (Seth Hall) 2.1-beta-45 | 2012-08-22 16:11:10 -0700 * Add an option to the input framework that allows the user to chose to not die upon encountering files/functions. (Johanna Amann) 2.1-beta-41 | 2012-08-22 16:05:21 -0700 * Add test serialization to "leak" unit tests that use communication. (Jon Siwek) * Change to metrics/basic-cluster unit test for reliability. (Jon Siwek) * Fixed ack tracking which could overflow quickly in some situations. (Seth Hall) * Minor tweak to coverage.bare-mode-errors unit test to work with a symlinked 'scripts' dir. (Jon Siwek) 2.1-beta-35 | 2012-08-22 08:44:52 -0700 * Add testcase for input framework reading sets (rather than tables). (Johanna Amann) 2.1-beta-31 | 2012-08-21 15:46:05 -0700 * Tweak to rotate-custom.bro unit test. (Jon Siwek) * Ignore small mem leak every rotation interval for dataseries logs. (Jon Siwek) 2.1-beta-28 | 2012-08-21 08:32:42 -0700 * Linking ES docs into logging document. (Robin Sommer) 2.1-beta-27 | 2012-08-20 20:06:20 -0700 * Add the Stream record to Log:active_streams to make more dynamic logging possible. (Seth Hall) * Fix portability of printing to files returned by open("/dev/stderr"). (Jon Siwek) * Fix mime type diff canonifier to also skip mime_desc columns. (Jon Siwek) * Unit test tweaks/fixes. (Jon Siwek) - Some baselines for tests in "leaks" group were outdated. - Changed a few of the cluster/communication tests to terminate more explicitly instead of relying on btest-bg-wait to kill processes. This makes the tests finish faster in the success case and makes the reason for failing clearer in the that case. * Fix memory leak of serialized IDs when compiled with --enable-debug. (Jon Siwek) 2.1-beta-21 | 2012-08-16 11:48:56 -0700 * Installing a handler for running out of memory in "new". Bro will now print an error message in that case rather than abort with an uncaught exception. (Robin Sommer) 2.1-beta-20 | 2012-08-16 11:43:31 -0700 * Fixed potential problems with ElasticSearch output plugin. (Seth Hall) 2.1-beta-13 | 2012-08-10 12:28:04 -0700 * Reporter warnings and error now print to stderr by default. New options Reporter::warnings_to_stderr and Reporter::errors_to_stderr to disable. (Seth Hall) 2.1-beta-9 | 2012-08-10 12:24:29 -0700 * Add more BIF tests. (Daniel Thayer) 2.1-beta-6 | 2012-08-10 12:22:52 -0700 * Fix bug in input framework with an edge case. (Johanna Amann) * Fix small bug in input framework test script. (Johanna Amann) 2.1-beta-3 | 2012-08-03 10:46:49 -0700 * Merge branch 'master' of ssh://git.bro-ids.org/bro (Robin Sommer) * Fix configure script to exit with non-zero status on error (Jon Siwek) * Improve ASCII output performance. (Robin Sommer) 2.1-beta | 2012-07-30 11:59:53 -0700 * Improve log filter compatibility with remote logging. Addresses #842. (Jon Siwek) 2.0-907 | 2012-07-30 09:13:36 -0700 * Add missing breaks to switch cases in ElasticSearch::HTTPReceive(). (Jon Siwek) 2.0-905 | 2012-07-28 16:24:34 -0700 * Fix log manager hanging on waiting for pending file rotations, plus writer API tweak for failed rotations. Addresses #860. (Jon Siwek and Robin Sommer) * Tweaking logs-to-elasticsearch.bro so that it doesn't do anything if ES server is unset. (Robin Sommer) 2.0-902 | 2012-07-27 12:42:13 -0700 * New variable in logging framework Log::active_streams to indicate Log:ID enums which are currently active. (Seth Hall) * Reworked how the logs-to-elasticsearch scripts works to stop abusing the logging framework. (Seth Hall) * Fix input test for recent default change on fastpath. (Robin Sommer) 2.0-898 | 2012-07-27 12:22:03 -0700 * Small (potential performance) improvement for logging framework. (Seth Hall) * Script-level rotation postprocessor fix. This fixes a problem with writers that don't have a postprocessor. (Seth Hall) * Update input framework documentation to reflect want_record change. (Johanna Amann) * Fix crash when encountering an InterpreterException in a predicate in logging or input Framework. (Johanna Amann) * Input framework: Make want_record=T the default for events (Johanna Amann) * Changing the start/end markers in logs to open/close now reflecting wall clock. (Robin Sommer) 2.0-891 | 2012-07-26 17:15:10 -0700 * Reader/writer API: preventing plugins from receiving further messages after a failure. (Robin Sommer) * New test for input framework that fails to find a file. (Robin Sommer) * Improving error handling for threads. (Robin Sommer) * Tweaking the custom-rotate test to produce stable output. (Robin Sommer) 2.0-884 | 2012-07-26 14:33:21 -0700 * Add comprehensive error handling for close() calls. (Jon Siwek) * Add more test cases for input framework. (Johanna Amann) * Input framework: make error output for non-matching event types much more verbose. (Johanna Amann) 2.0-877 | 2012-07-25 17:20:34 -0700 * Fix double close() in FilerSerializer class. (Jon Siwek) * Fix build warnings. (Daniel Thayer) * Fixes to ElasticSearch plugin to make libcurl handle http responses correctly. (Seth Hall) * Fixing FreeBSD compiler error. (Robin Sommer) * Silencing compiler warnings. (Robin Sommer) 2.0-871 | 2012-07-25 13:08:00 -0700 * Fix complaint from valgrind about uninitialized memory usage. (Jon Siwek) * Fix differing log filters of streams from writing to same writer/path (which now produces a warning, but is otherwise skipped for the second). Addresses #842. (Jon Siwek) * Fix tests and error message for to_double BIF. (Daniel Thayer) * Compile fix. (Robin Sommer) 2.0-866 | 2012-07-24 16:02:07 -0700 * Correct a typo in usage message. (Daniel Thayer) * Fix file permissions of log files (which were created with execute permissions after a recent change). (Daniel Thayer) 2.0-862 | 2012-07-24 15:22:52 -0700 * Fix initialization problem in logging class. (Jon Siwek) * Input framework now accepts escaped ASCII values as input (\x##), and unescapes appropiately. (Johanna Amann) * Make reading ASCII logfiles work when the input separator is different from \t. (Johanna Amann) * A number of smaller fixes for input framework. (Johanna Amann) 2.0-851 | 2012-07-24 15:04:14 -0700 * New built-in function to_double(s: string). (Scott Campbell) 2.0-849 | 2012-07-24 11:06:16 -0700 * Adding missing include needed on some systems. (Robin Sommer) 2.0-846 | 2012-07-23 16:36:37 -0700 * Fix WriterBackend::WriterInfo serialization, reenable ascii start/end tags. (Jon Siwek) 2.0-844 | 2012-07-23 16:20:59 -0700 * Reworking parts of the internal threading/logging/input APIs for thread-safety. (Robin Sommer) * Bugfix for SSL version check. (Johanna Amann) * Changing a HTTP DPD from port 3138 to 3128. Addresses #857. (Robin Sommer) * ElasticSearch logging writer. See logging-elasticsearch.rst for more information. (Vlad Grigorescu and Seth Hall). * Give configure a --disable-perftools option to disable Perftools support even if found. (Robin Sommer) * The ASCII log writer now includes "#start " and "#end lines in the each file. (Robin Sommer) * Renamed ASCII logger "header" options to "meta". (Robin Sommer) * ASCII logs now escape '#' at the beginning of log lines. Addresses #763. (Robin Sommer) * Fix bug, where in dns.log rcode always was set to 0/NOERROR when no reply package was seen. (Johanna Amann) * Updating to Mozilla's current certificate bundle. (Seth Hall) 2.0-769 | 2012-07-13 16:17:33 -0700 * Fix some Info:Record field documentation. (Vlad Grigorescu) * Fix overrides of TCP_ApplicationAnalyzer::EndpointEOF. (Jon Siwek) * Fix segfault when incrementing whole vector values. Also removed RefExpr::Eval(Val*) method since it was never called. (Jon Siwek) * Remove baselines for some leak-detecting unit tests. (Jon Siwek) * Unblock SIGFPE, SIGILL, SIGSEGV and SIGBUS for threads, so that they now propagate to the main thread. Adresses #848. (Johanna Amann) 2.0-761 | 2012-07-12 08:14:38 -0700 * Some small fixes to further reduce SOCKS false positive logs. (Seth Hall) * Calls to pthread_mutex_unlock now log the reason for failures. (Johanna Amann) 2.0-757 | 2012-07-11 08:30:19 -0700 * Fixing memory leak. (Seth Hall) 2.0-755 | 2012-07-10 16:25:16 -0700 * Add sorting canonifier to rotate-custom unit test. Addresses #846. (Jon Siwek) * Fix many compiler warnings. (Daniel Thayer) * Fix segfault when there's an error/timeout resolving DNS requests. Addresses #846. (Jon Siwek) * Remove a non-portable test case. (Daniel Thayer) * Fix typos in input framework doc. (Daniel Thayer) * Fix typos in DataSeries documentation. (Daniel Thayer) * Bugfix making custom rotate functions work again. (Robin Sommer) * Tiny bugfix for returning writer name. (Robin Sommer) * Moving make target update-doc-sources from top-level Makefile to btest Makefile. (Robin Sommer) 2.0-733 | 2012-07-02 15:31:24 -0700 * Extending the input reader DoInit() API. (Johanna Amann). It now provides a Info struct similar to what we introduced for log writers, including a corresponding "config" key/value table. * Fix to make writer-info work when debugging is enabled. (Johanna Amann) 2.0-726 | 2012-07-02 15:19:15 -0700 * Extending the log writer DoInit() API. (Robin Sommer) We now pass in a Info struct that contains: - the path name (as before) - the rotation interval - the log_rotate_base_time in seconds - a table of key/value pairs with further configuration options. To fill the table, log filters have a new field "config: table[string] of strings". This gives a way to pass arbitrary values from script-land to writers. Interpretation is left up to the writer. * Split calc_next_rotate() into two functions, one of which is thread-safe and can be used with the log_rotate_base_time value from DoInit(). * Updates to the None writer. (Robin Sommer) - It gets its own script writers/none.bro. - New bool option LogNone::debug to enable debug output. It then prints out all the values passed to DoInit(). - Fixed a bug that prevented Bro from terminating. 2.0-723 | 2012-07-02 15:02:56 -0700 * Extract ICMPv6 NDP options and include in ICMP events. This adds a new parameter of type "icmp6_nd_options" to the ICMPv6 neighbor discovery events. Addresses #833. (Jon Siwek) * Set input frontend type before starting the thread. This means that the thread type will be output correctly in the error message. (Johanna Amann) 2.0-719 | 2012-07-02 14:49:03 -0700 * Fix inconsistencies in random number generation. The srand()/rand() interface was being intermixed with the srandom()/random() one. The later is now used throughout. (Jon Siwek) * Changed the srand() and rand() BIFs to work deterministically if Bro was given a seed file. Addresses #825. (Jon Siwek) * Updating input framework unit tests to make them more reliable and execute quicker. (Jon Siwek) * Fixed race condition in writer and reader initializations. (Jon Siwek) * Small tweak to make test complete quicker. (Jon Siwek) * Drain events before terminating log/thread managers. (Jon Siwek) * Fix strict-aliasing warning in RemoteSerializer.cc. Addresses #834. (Jon Siwek) * Fix typos in event documentation. (Daniel Thayer) * Fix typos in NEWS for Bro 2.1 beta. (Daniel Thayer) 2.0-709 | 2012-06-21 10:14:24 -0700 * Fix exceptions thrown in event handlers preventing others from running. (Jon Siwek) * Add another SOCKS command. (Seth Hall) * Fixed some problems with the SOCKS analyzer and tests. (Seth Hall) * Updating NEWS in preparation for beta. (Robin Sommer) * Accepting different AF_INET6 values for loopback link headers. (Robin Sommer) 2.0-698 | 2012-06-20 14:30:40 -0700 * Updates for the SOCKS analyzer (Seth Hall). - A SOCKS log! - Now supports SOCKSv5 in the analyzer and the DPD sigs. - Added protocol violations. * Updates to the tunnels framework. (Seth Hall) - Make the uid field optional since it's conceptually incorrect for proxies being treated as tunnels to have it. - Reordered two fields in the log. - Reduced the default tunnel expiration interface to something more reasonable (1 hour). * Make Teredo bubble packet parsing more lenient. (Jon Siwek) * Fix a crash in NetSessions::ParseIPPacket(). (Jon Siwek) 2.0-690 | 2012-06-18 16:01:33 -0700 * Support for decapsulating tunnels via the new tunnel framework in base/frameworks/tunnels. Bro currently supports Teredo, AYIYA, IP-in-IP (both IPv4 and IPv6), and SOCKS. For all these, it logs the outher tunnel connections in both conn.log and tunnel.log, and proceeds to analyze the inner payload as if it were not tunneled, including also logging it in conn.log (with a new tunnel_parents column pointing back to the outer connection(s)). (Jon Siwek, Seth Hall, Gregor Maier) * The options "tunnel_port" and "parse_udp_tunnels" have been removed. (Jon Siwek) 2.0-623 | 2012-06-15 16:24:52 -0700 * Changing an error in the input framework to a warning. (Robin Sommer) 2.0-622 | 2012-06-15 15:38:43 -0700 * Input framework updates. (Johanna Amann) - Disable streaming reads from executed commands. This lead to hanging Bros because pclose apparently can wait for eternity if things go wrong. - Automatically delete disabled input streams. - Documentation. 2.0-614 | 2012-06-15 15:19:49 -0700 * Remove an old, unused diff canonifier. (Jon Siwek) * Improve an error message in ICMP analyzer. (Jon Siwek) * Fix a warning message when building docs. (Daniel Thayer) * Fix many errors in the event documentation. (Daniel Thayer) 2.0-608 | 2012-06-11 15:59:00 -0700 * Add more error handling code to logging of enum vals. Addresses #829. (Jon Siwek) 2.0-606 | 2012-06-11 15:55:56 -0700 * Fix summary lines for BIF documentation and corrected the description of "fmt" and "floor" BIFs. (Daniel Thayer) * Fix val_size BIF tests and improve docs. (Daniel Thayer) 2.0-602 | 2012-06-07 15:06:19 -0700 * Include header for usleep(), caused compile failure on Archlinux. (Jon Siwek) * Revert "Fixed a bug with the MIME analyzer not removing whitespace on wrapped headers." Needs discussion. (Robin Sommer) 2.0-598 | 2012-06-06 11:47:00 -0700 * Add @load-sigs directive for loading signature files (addresses #551). This can be used to load signatures relative to the current scripts (e.g., "@load-sigs ./foo.sig"). (Jon Siwek) 2.0-596 | 2012-06-06 11:41:00 -0700 * Fixes for some BiFs and their documentation. (Daniel Thayer) * Many new unit tests for BiFs. (Daniel Thayer) 2.0-579 | 2012-06-06 11:04:46 -0700 * Memory leak fixes for bad usages of VectorVal ctor. (Jon Siwek) * Fixed a bug with the MIME analyzer not removing whitespace on wrapped headers. (Seth Hall) * Change Input::update_finished lookup to happen at init time. (Jon Siwek) * Fix going through the internal_handler() function which will now set the event as "used" (i.e. it's marked as being raised somewhere). Addresses #823. (Jon Siwek) * Fix format specifier on RemoteSerializer::Connect. This caused 32-bit systems to show a warning at compile-time, and fail when connecting to peers. (Jon Siwek) * Fixes for running tests in parallel. (Robin Sommer) 2.0-571 | 2012-05-30 19:12:43 -0700 * Updating submodule(s). 2.0-570 | 2012-05-30 19:08:18 -0700 * A new input framework enables scripts to read in external data dynamically on the fly as Bro is processing network traffic. (Johanna Amann) Currently, the framework supports reading ASCII input that's structured similar as Bro's log files as well as raw blobs of data. Other formats will come in the future. See doc/input.rst for more information (this will be extended further soon). 2.0-395 | 2012-05-30 17:03:31 -0700 * Remove unnecessary assert in ICMP analyzer which could lead to aborts. Addresses #822. * Improve script debugger backtrace and print commands. (Jon Siwek) * Switching default DS compression to gzip. (Robin Sommer) * Improve availability of IPv6 flow label in connection records. This adds a "flow_label" field to the "endpoint" record type, which is used for both the "orig" and "resp" fields of "connection" records. The new "connection_flow_label_changed" event also allows tracking of changes in flow labels: it's raised each time one direction of the connection starts using a different label. (Jon Siwek) * Add unit tests for Broccoli SSL and Broccoli IPv6 connectivity. (Jon Siwek) * Remove AI_ADDRCONFIG getaddrinfo hints flag for listening sockets. (Jon Siwek) * Undo unnecessary communication protocol version bump. (Jon Siwek) * Add support to Bro for connecting with peers over IPv6. (Jon Siwek) - Communication::listen_ipv6 needs to be redef'd to true in order for IPv6 listening sockets to be opened. - Added Communication::listen_retry option as an interval at which to retry binding to socket addresses that were already in use. - Added some explicit baselines to check in the istate.events and istate.events-ssl tests -- the SSL test was incorrectly passing because it compared two empty files. (The files being empty because "http/base" was given as an argument to Bro which it couldn't handle because that script doesn't exist anymore). - Support for communication over non-global IPv6 addresses. This usually requires specifying an additional zone identifier (see RFC 4007). The connect() and listen() BIFs have been changed to accept this zone identifier as an argument. 2.0-377 | 2012-05-24 16:46:06 -0700 * Documentation fixes. (Jon Siwek and Daniel Thayer) 2.0-372 | 2012-05-17 13:59:45 -0700 * Fix compile errors. (Jon Siwek) * Linking in the DS docs. (Robin Sommer) * Fix mobility checksums unit test. (Jon Siwek) 2.0-367 | 2012-05-17 12:42:30 -0700 * Adding support for binary output via DataSeries. See logging-dataseries.rst for more information. (Gilbert Clark and Robin Sommer) * Adding target update-doc-sources to top-level Makefile that runs genDocSourcesList.sh. (Robin Sommer) * Moving trace for rotation test into traces directory. (Robin Sommer) * Fixing a rotation race condition at termination. (Robin Sommer) * Extending log post-processor call to include the name of the writer. (Robin Sommer) * In threads, an internal error now immediately aborts. Otherwise, the error won't make it back to the main thread for a while and subsequent code in the thread would still execute. (Robin Sommer) * DataSeries cleanup. (Robin Sommer) * Fixing threads' DoFinish() method. It wasn't called reliably. Now, it's always called before the thread is destroyed (assuming processing has went normally so far). (Robin Sommer) 2.0-341 | 2012-05-17 09:54:30 -0700 * Add a comment to explain the ICMPv6 error message types. (Daniel Thayer) * Quieting external test output somehwat. (Robin Sommer) 2.0-336 | 2012-05-14 17:15:44 -0700 * Don't print the various "weird" events to stderr. Address #805. (Daniel Thayer) * Generate icmp_error_message event for ICMPv6 error msgs. Previously, icmp_sent was being generated, but icmp_error_message contains more info. * Improved documentation comments for icmp-related events. (Daniel Thayer) 2.0-330 | 2012-05-14 17:05:56 -0700 * Add `addr_to_uri` script-level function that adds brackets to an address if it's IPv6 and will be included in a URI or when a ":" needs to be appended to it. (Jon Siwek) * Also add a test case for content extraction. (Jon Siwek) * Fix typos and improve INSTALL document. (Daniel Thayer) * Switching to new btest command TEST-SERIALIZE for communication tests. (Robin Sommer) 2.0-323 | 2012-05-04 21:04:34 -0700 * Add SHA1 and SHA256 hashing BIFs. Addresses #542. * Refactor all internal MD5 stuff to use OpenSSL's. (Jon Siwek) * Changes to open-file caching limits and uncached file unserialization. (Jon Siwek) - Unserializing files that were previously kicked out of the open-file cache would cause them to be fopen'd with the original access permissions which is usually 'w' and causes truncation. They are now opened in 'a' mode. (addresses #780) - Add 'max_files_in_cache' script option to manually set the maximum amount of opened files to keep cached. Mainly this just helped to create a simple test case for the above change. - Remove unused NO_HAVE_SETRLIMIT preprocessor switch. - On systems that don't enforce a limit on number of files opened for the process, raise default max size of open-file cache from 32 to 512. 2.0-319 | 2012-05-03 13:24:44 -0700 * SSL bugfixes and cleanup. (Seth Hall) - SSL related files and classes renamed to remove the "binpac" term. - A small fix for DPD scripts to make the DPD log more helpful if there are multiple continued failures. - Fixed the SSL analyzer to make it stop doing repeated violation messages for some handshake failures. - Added a $issuer_subject to the SSL log. - Created a basic test for SSL. - Fixed parsing of TLS server extensions. (Seth Hall) 2.0-315 | 2012-05-03 11:44:17 -0700 * Add two more TLS extension values that we see in live traffic. (Johanna Amann) * Fixed IPv6 link local unicast CIDR and added IPv6 loopback to private address space. (Seth Hall) * Fixed a problem where cluster workers were still processing notices in some cases. (Seth Hall) * Added a configure option to specify the 'etc' directory. Addresses #801. (Daniel Thayer) 2.0-306 | 2012-04-24 14:37:00 -0700 * Add further TLS extension values "extended_random" and "heartbeat". (Seth Hall) * Fix problem with extracting FTP passwords and add "ftpuser" as another anonymous username. (Seth Hall, discovered by Patrik Lundin). 2.0-303 | 2012-04-19 10:01:06 -0700 * Changes related to ICMPv6 Neighbor Discovery messages. (Jon Siwek) - The 'icmp_conn' record now contains an 'hlim' field since hop limit in the IP header is an interesting field for at least these ND messages. - Fixed and extended 'icmp_router_advertisement' event parameters. - Changed 'icmp_neighbor_advertisement' event parameters to add more of the known boolean flags. 2.0-301 | 2012-04-17 17:58:55 -0700 * Bro now support ICMPv6. (Matti Mantere, Jon Siwek, Robin Sommer, Daniel Thayer). Overall, Bro now raises the following ICMP events for v4 and v6 as appropiate: event icmp_sent(c: connection, icmp: icmp_conn); event icmp_echo_request(c: connection, icmp: icmp_conn, id: count, seq: count, payload: string); event icmp_echo_reply(c: connection, icmp: icmp_conn, id: count, seq: count, payload: string); event icmp_error_message(c: connection, icmp: icmp_conn, code: count, context: icmp_context); event icmp_unreachable(c: connection, icmp: icmp_conn, code: count, context: icmp_context); event icmp_packet_too_big(c: connection, icmp: icmp_conn, code: count, context: icmp_context); event icmp_time_exceeded(c: connection, icmp: icmp_conn, code: count, context: icmp_context); event icmp_parameter_problem(c: connection, icmp: icmp_conn, code: count, context: icmp_context); event icmp_router_solicitation(c: connection, icmp: icmp_conn); event icmp_router_advertisement(c: connection, icmp: icmp_conn, hop_limit: count, managed: bool, router_lifetime: count, reachable_time: interval, retrans_timer: interval); event icmp_neighbor_solicitation(c: connection, icmp: icmp_conn, tgt:addr); event icmp_neighbor_advertisement(c: connection, icmp: icmp_conn, tgt:addr); event icmp_redirect(c: connection, icmp: icmp_conn, tgt: addr, dest: addr); The `icmp_conn` record got a new boolean field 'v6' that indicates whether the ICMP message is v4 or v6. This change also includes further low-level work on existing IP and ICMP code, including a reorganization of how ICMPv4 is handled. 2.0-281 | 2012-04-17 17:40:39 -0700 * Small updates for the bittorrent analyzer to support 64bit types in binpac. (Seth Hall) * Removed the attempt at bittorrent resynchronization. (Seth Hall) 2.0-276 | 2012-04-17 17:35:56 -0700 * Add more support for 's that lack some structure definitions. (Jon Siwek) 2.0-273 | 2012-04-16 18:08:56 -0700 * Removing QR flag from DNS log in response, which should not have been there in the first place. (Seth Hall) * Sync up patricia.c/h with pysubnettree repo. (Daniel Thayer) * Adding missing leak groups to a couple tests. Also activating leak checking for proxy in basic-cluster test. (Robin Sommer) 2.0-267 | 2012-04-09 17:47:28 -0700 * Add support for mobile IPv6 Mobility Header (RFC 6275). (Jon Siwek) - Enabled through a new --enable-mobile-ipv6 configure-time option. If not enabled, the mobility header (routing type 2) and Home Address Destination option are ignored. - Accessible at script-layer through 'mobile_ipv6_message' event. * Refactor IP_Hdr routing header handling, add MobileIPv6 Home Address handling. Packets that use the Home Address Destination option use that option's address as the connection's originator. (Jon Siwek) * Revert TCP checksumming to cache common data, like it did before. (Jon Siwek) * Improve handling of IPv6 routing type 0 extension headers. (Jon Siwek) - flow_weird event with name argument value of "routing0_hdr" is raised for packets containing an IPv6 routing type 0 header because this type of header is now deprecated according to RFC 5095. - Packets with a routing type 0 header and non-zero segments left now use the last address in that header in order to associate with a connection/flow and for calculating TCP/UDP checksums. - Added a set of IPv4/IPv6 TCP/UDP checksum unit tests (Jon Siwek) * Fix table expiry for values assigned in bro_init() when reading live. (Jon Siwek) 2.0-257 | 2012-04-05 15:32:43 -0700 * Fix CMake from warning about unused ENABLE_PERFTOOLS_DEBUG variable. (Jon Siwek) * Fix handling of IPv6 atomic fragments. (Jon Siwek) * Fix that prevents Bro processes that do neither local logging nor request remote logs from spawning threads. (Robin Sommer) * Fixing perftools-debug support. (Robin Sommer) * Reverting SocketComm change tuning I/O behaviour. (Robin Sommer) * Adding notice_policy.log canonification for external tests. (Robin Sommer) 2.0-245 | 2012-04-04 17:25:20 -0700 * Internal restructuring of the logging framework: we now spawn threads doing the I/O. From a user's perspective not much should change, except that the OS may now show a bunch of Bro threads. (Gilbert Clark and Robin Sommer). * When building Bro, we now always link in tcmalloc if it's found at configure time. If it's installed but not picked up, --with-perftools may help. (Robin Sommer) * Renaming the configure option --enable-perftools to --enable-perftool-debug to indicate that the switch is only relevant for debugging the heap. It's not needed to pick up tcmalloc for better performance. (Robin Sommer) 2.0-184 | 2012-03-28 15:11:11 -0700 * Improve handling of IPv6 Routing Type 0 headers. (Jon Siwek) - For RH0 headers with non-zero segments left, a "routing0_segleft" flow_weird event is raised (with a destination indicating the last address in the routing header), and an "rh0_segleft" event can also be handled if the other contents of the packet header are of interest. No further analysis is done as the complexity required to correctly identify destination endpoints of connections doesn't seem worth it as RH0 has been deprecated by RFC 5095. - For RH0 headers without any segments left, a "routing0_header" flow_weird event is raised, but further analysis still occurs as normal. 2.0-182 | 2012-03-28 15:01:57 -0700 * Remove dead tcp_checksum function from net_util. (Jon Siwek) * Change routing0_data_to_addrs BIF to return vector of addresses. The order of addresses in type 0 routing headers is interesting/important. (Jon Siwek) 2.0-179 | 2012-03-23 17:43:31 -0700 * Remove the default "tcp or udp or icmp" filter. In default mode, Bro would load the packet filter script framework which installs a filter that allows all packets, but in bare mode (the -b option), this old filter would not follow IPv6 protocol chains and thus filter out packets with extension headers. (Jon Siwek) * Update PacketFilter/Discarder code for IP version independence. (Jon Siwek) * Fix some IPv6 header related bugs. (Jon Siwek) * Add IPv6 fragment reassembly. (Jon Siwek) * Add handling for IPv6 extension header chains. Addresses #531. (Jon Siwek) - The script-layer 'pkt_hdr' type is extended with a new 'ip6' field representing the full IPv6 header chain. - The 'new_packet' event is now raised for IPv6 packets. Addresses #523. - A new event called 'ipv6_ext_header' is raised for any IPv6 packet containing extension headers. - A new event called 'esp_packet' is raised for any packets using ESP ('new_packet' and 'ipv6_ext_header' events provide connection info, but that info can't be provided here since the upper-layer payload is encrypted). - The 'unknown_protocol' weird is now raised more reliably when Bro sees a transport protocol or IPv6 extension header it can't handle. Addresses #522. * Add unit tests for IPv6 fragment reassembly, ipv6_ext_headers and esp_packet events. (Jon Siwek) * Adapt FreeBSD's inet_ntop implementation for internal use. Now we get consistent text representations of IPv6 addresses across platforms. (Jon Siwek) * Update documentation for new syntax of IPv6 literals. (Jon Siwek) 2.0-150 | 2012-03-13 16:16:22 -0700 * Changing the regular expression to allow Site::local_nets in signatures. (Julien Sentier) * Removing a line of dead code. Found by . Closes #786. (Julien Sentier) 2.0-146 | 2012-03-13 15:39:38 -0700 * Change IPv6 literal constant syntax to require encasing square brackets. (Jon Siwek) 2.0-145 | 2012-03-09 15:10:35 -0800 * Remove the match expression. 'match' and 'using' are no longer keywords. Addressed #753. (Jon Siwek) 2.0-143 | 2012-03-09 15:07:42 -0800 * Fix a BRO_PROFILER_FILE/mkstemp portability issue. Addresses #794. (Jon Siwek) 2.0-139 | 2012-03-02 09:33:04 -0800 * Changes to how script coverage integrates with test suites. (Jon Siwek) - BRO_PROFILER_FILE now passes .X* templated filenames to mkstemp for generating unique coverage state files. - Rearranging Makefile targets. The general rule is that if the all/brief target fails out due to a test failure, then the dependent coverage target won't run, but can still be invoked directly later. (e.g. make brief || make coverage) * Standardized on the &default function for SSL constants. (Seth Hall) * Adding btest group "leaks" to leak tests. (Robin Sommer) * Adding btest group "comm" to communication tests for parallelizing execution with new btest version. (Robin Sommer) * Sorting all output for diffing in the external tests. (Robin Sommer) * Cleaned up dead code from the old SSL analyzers. Reported by Julien Sentier. (Seth Hall) * Update/add tests for broccoli IPv6 addr/subnet support. Addresses #448. (Jon Siwek) * Remove connection compressor. Addresses #559. (Jon Siwek) * Refactor IP_Hdr class ctors. Addresses #532. (Jon Siwek) 2.0-121 | 2012-02-24 16:34:17 -0800 * A number of smaller memory fixes and code cleanups. (Julien Sentier) * Add to_subnet bif. Fixes #782). (Jon Siwek) * Fix IPAddr::Mask/ReverseMask not allowing argument of 0. (Jon Siwek) * Refactor IPAddr v4 initialization from string. Fixes #775. (Jon Siwek) * Parse the dotted address string directly instead of canonicalizing and passing to inet_pton. (Jon Siwek) 2.0-108 | 2012-02-24 15:21:07 -0800 * Refactoring a number of usages of new IPAddr class. (Jon Siwek) * Fixed a bug in remask_addr bif. (Jon Siwek) 2.0-106 | 2012-02-24 15:02:20 -0800 * Raise minimum required CMake version to 2.6.3. (Jon Siwek) 2.0-104 | 2012-02-24 14:59:12 -0800 * Add test case for FTP over IPv4. (Daniel Thayer) * Fix IPv6 URLs in ftp.log. (Daniel Thayer) * Add a test for FTP over IPv6 (Daniel Thayer) * Fix parsing of FTP EPRT command and EPSV response. (Daniel Thayer) 2.0-95 | 2012-02-22 05:27:34 -0800 * GeoIP installation documentation update. (Seth Hall) * Decrease strictness of parsing IPv4 strings into addrs. Fixes #775. (Jon Siwek) * Fix memory leak in DNS manager. Fixes #777. (Jon Siwek) * Fix IPAddr/IPPrefix serialization bugs. (Jon Siwek) * Fix compile error. (Jon Siwek) 2.0-86 | 2012-02-17 15:41:06 -0800 * Changing ARP detection to always kick in even if no analyzer is activated. (Robin Sommer) * DNS name lookups performed by Bro now also query AAAA records. DNS_Mgr handles combining the results of the A and AAAA queries for a given hostname such that at the scripting layer, the name resolution can yield a set with both IPv4 and IPv6 addresses. (Jon Siwek) * Add counts_to_addr and addr_to_counts conversion BIFs. (Jon Siwek) * Change HashKey threshold for using H3 to 36 bytes. (Jon Siwek) * Remove mention of --enable-brov6 in docs. (Daniel Thayer) * Remove --enable-brov6 from configure usage text (Daniel Thayer) * Add a test and baseline for addr_to_ptr_name BiF. (Daniel Thayer) * Adding a test and baseline for ptr_name_to_addr BiF. (Seth Hall) * Fix the ptr_name_to_addr BiF to work with IPv6 (Daniel Thayer) * Fix a memory leak that perftools now complains about. (Jon Siwek) * Remove --enable-brov6 flag, IPv6 now supported by default. (Jon Siwek) Some script-layer changes of note: - dns_AAAA_reply event signature changed: the string representation of an IPv6 addr is easily derived from the addr value, it doesn't need to be another parameter. This event also now generated directly by the DNS analyzer instead of being "faked" into a dns_A_reply event. - Removed addr_to_count BIF. It used to return the host-order count representation of IPv4 addresses only. To make it more generic, we might later add a BIF to return a vector of counts in order to support IPv6. - Changed the result of enclosing addr variables in vertical pipes (e.g. |my_addr|) to return the bit-width of the address type which is 128 for IPv6 and 32 for IPv4. It used to function the same way as addr_to_count mentioned above. - Remove bro_has_ipv6 BIF 2.0-57 | 2012-02-10 00:02:35 -0800 * Fix typos in the documentation. (Daniel Thayer) * Fix compiler warning about Brofiler ctor init list order. (Jon Siwek) * Fix missing optional field access in webapp signature_match handler. (Jon Siwek) 2.0-41 | 2012-02-03 04:10:53 -0500 * Updates to the Software framework to simplify the API. (Johanna Amann) 2.0-40 | 2012-02-03 01:55:27 -0800 * Fix typos in documentation. (Daniel Thayer) * Fix sorting of lines in Brofiler coverage.log. (Daniel Thayer) 2.0-38 | 2012-01-31 11:50:53 -0800 * Canonify sorting of lines in Brofiler coverage.log. (Daniel Thayer) 2.0-36 | 2012-01-27 10:38:14 -0800 * New "Brofiler" mode that tracks and records script statements executed during runtime. (Jon Siwek) Use the BROFILER_FILE environment variable to point to a file in which statement usage statistics from Bro script-layer can be output. Script statements that should be ignored can be marked with a "# @no-test" comment. For example: print "don't cover"; # @no-test if ( F ) { # @no-test ... } * Integrated coverage measurement into test-suite. (Jon Siwek) 2.0-20 | 2012-01-25 16:34:51 -0800 * BiF cleanup (Matthias Vallentin) - Rename NFS3::mode2string to a more generic file_mode(). - Unify do_profiling()/make_connection_persistent()/expect_connection() to return any (i.e., nothing) instead of bools. - Perform type checking on count-to-port conversion. Related to #684. - Remove redundant connection_record() BiF. The same functionality is provided by lookup_connection(). - Remove redundant active_connection() BiF. The same functionality is provided by connection_exists(). - exit() now takes the exit code as argument. - to_port() now received a string instead of a count. 2.0-9 | 2012-01-25 13:47:13 -0800 * Allow local table variables to be initialized with {} list expressions. (Jon Siwek) 2.0-7 | 2012-01-25 13:38:09 -0800 * Teach CompHash to allow indexing by records with vector/table/set fields. Addresses #464. (Jon Siwek) 2.0-5 | 2012-01-25 13:25:19 -0800 * Fixed a bug resulting in over-logging of detected webapps. (Seth Hall) * Make communication log baseline test more reliable. (Jon Siwek) * Fixed some broken links in documentation. (Daniel Thayer) 2.0 | 2012-01-11 13:52:22 -0800 * Adding script reference documentation. (The Team). 2.0-beta-194 | 2012-01-10 10:44:32 -0800 * Added an option for filtering out URLs before they are turned into HTTP::Incorrect_File_Type notices. (Seth Hall) * Fix ref counting bug in BIFs that call internal_type. Addresses #740. (Jon Siwek) * Adding back the stats.bro file. (Seth Hall) 2.0-beta-188 | 2012-01-10 09:49:29 -0800 * Change SFTP/SCP log rotators to use 4-digit year in filenames Fixes #745. (Jon Siwek) * Adding back the stats.bro file. Addresses #656. (Seth Hall) 2.0-beta-185 | 2012-01-09 18:00:50 -0800 * Tweaks for OpenBSD support. (Jon Siwek) 2.0-beta-181 | 2012-01-08 20:49:04 -0800 * Add SFTP log postprocessor that transfers logs to remote hosts. Addresses #737. (Jon Siwek) * Add FAQ entry about disabling NIC offloading features. (Jon Siwek) * Add a file NEWS with release notes. (Robin Sommer) 2.0-beta-177 | 2012-01-05 15:01:07 -0800 * Replace the --snaplen/-l command line option with a scripting-layer option called "snaplen" (which can also be redefined on the command line, e.g. `bro -i eth0 snaplen=65535`). * Reduce snaplen default from 65535 to old default of 8192. Fixes #720. (Jon Siwek) 2.0-beta-174 | 2012-01-04 12:47:10 -0800 * SSL improvements. (Seth Hall) - Added the ssl_session_ticket_handshake event back. - Fixed a few bugs. - Removed the SSLv2.cc file since it's not used. 2.0-beta-169 | 2012-01-04 12:44:39 -0800 * Tuning the pretty-printed alarm mails, which now include the covered time range into the subject. (Robin Sommer) * Adding top-level "test" target to Makefile. (Robin Sommer) * Adding SWIG as dependency to INSTALL. (Robin Sommer) 2.0-beta-155 | 2012-01-03 15:42:32 -0800 * Remove dead code related to record type inheritance. (Jon Siwek) 2.0-beta-152 | 2012-01-03 14:51:34 -0800 * Notices now record the transport-layer protocol. (Johanna Amann) 2.0-beta-150 | 2012-01-03 14:42:45 -0800 * CMake 2.6 top-level 'install' target compat. Fixes #729. (Jon Siwek) * Minor fixes to test process. Addresses #298. * Increase timeout interval of communication-related btests. (Jon Siwek) 2.0-beta-145 | 2011-12-19 11:37:15 -0800 * Empty fields are now logged as "(empty)" by default. (Robin Sommer) * In log headers, only escape information when necessary. (Robin Sommer) 2.0-beta-139 | 2011-12-19 07:06:29 -0800 * The hostname notice email extension works now, plus a general mechanism for adding delayed information to notices. (Seth Hall) * Fix &default fields in records not being initialized in coerced assignments. Addresses #722. (Jon Siwek) * Make log headers include the type of data stored inside a set or vector ("vector[string]"). (Johanna Amann) 2.0-beta-126 | 2011-12-18 15:18:05 -0800 * DNS updates. (Seth Hall) - Fixed some bugs with capturing data in the base DNS script. - Answers and TTLs are now vectors. - A warning that was being generated (dns_reply_seen_after_done) from transaction ID reuse is fixed. * SSL updates. (Seth Hall) - Added is_orig fields to the SSL events and adapted script. - Added a field named last_alert to the SSL log. - The x509_certificate function has an is_orig field now instead of is_server and its position in the argument list has moved. - A bit of reorganization and cleanup in the core analyzer. (Seth Hall) 2.0-beta-121 | 2011-12-18 15:10:15 -0800 * Enable warnings for malformed Broxygen xref roles. (Jon Siwek) * Fix Broxygen confusing scoped IDs at start of line as function parameter. (Jon Siwek) * Allow Broxygen markup "##<" for more general use. (Jon Siwek) 2.0-beta-116 | 2011-12-16 02:38:27 -0800 * Cleanup some misc Broxygen css/js stuff. (Jon Siwek) * Add search box to Broxygen docs. Fixes #726. (Jon Siwek) * Fixed major bug with cluster synchronization, which was not working. (Seth Hall) * Fix missing action in notice policy for looking up GeoIP data. (Jon Siwek) * Better persistent state configuration warning messages (fixes #433). (Jon Siwek) * Renaming HTTP::SQL_Injection_Attack_Against to HTTP::SQL_Injection_Victim. (Seth Hall). * Fixed DPD signatures for IRC. Fixes #311. (Seth Hall) * Removing Off_Port_Protocol_Found notice. (Seth Hall) * Teach Broxygen to more generally reference attribute values by name. (Jon Siwek) * SSH::Interesting_Hostname_Login cleanup. Fixes #664. (Seth Hall) * Fixed bug that was causing the malware hash registry script to break. (Seth Hall) * Remove remnant of libmagic optionality. (Jon Siwek) 2.0-beta-98 | 2011-12-07 08:12:08 -0800 * Adapting test-suite's diff-all so that it expands globs in both current and baseline directory. Closes #677. (Robin Sommer) 2.0-beta-97 | 2011-12-06 11:49:29 -0800 * Omit loading local-.bro scripts from base cluster framework. Addresses #663 (Jon Siwek) 2.0-beta-94 | 2011-12-03 15:57:19 -0800 * Adapting attribute serialization when talking to Broccoli. (Robin Sommer) 2.0-beta-92 | 2011-12-03 15:56:03 -0800 * Changes to Broxygen master script package index. (Jon Siwek) - Now only lists packages as those directories in the script hierarchy that contain an __load__.bro file. - Script packages (dirs with a __load__.bro file), can now include a README (in reST format) that will automatically be appended under the link to a specific package in the master package index. 2.0-beta-88 | 2011-12-02 17:00:58 -0800 * Teach LogWriterAscii to use BRO_LOG_SUFFIX environemt variable. Addresses #704. (Jon Siwek) * Fix double-free of DNS_Mgr_Request object. Addresses #661. * Add a remote_log_peer event which comes with an event_peer record parameter. Addresses #493. (Jon Siwek) * Remove example redef of SMTP::entity_excerpt_len from local.bro. Fixes error emitted when loading local.bro in bare mode. (Jon Siwek) * Add missing doc targets to top Makefile; remove old doc/Makefile. Fixes #705. (Jon Siwek) * Turn some globals into constants. Addresses #633. (Seth Hall) * Rearrange packet filter and DPD documentation. (Jon Siwek) 2.0-beta-72 | 2011-11-30 20:16:09 -0800 * Fine-tuning the Sphinx layout to better match www. (Jon Siwek and Robin Sommer) 2.0-beta-69 | 2011-11-29 16:55:31 -0800 * Fixing ASCII logger to escape the unset-field place holder if written out literally. (Robin Sommer) 2.0-beta-68 | 2011-11-29 15:23:12 -0800 * Lots of documentation polishing. (Jon Siwek) * Teach Broxygen the ".. bro:see::" directive. (Jon Siwek) * Teach Broxygen :bro:see: role for referencing any identifier in the Bro domain. (Jon Siwek) * Teach Broxygen to generate an index of Bro notices. (Jon Siwek) * Fix order of include directories. (Jon Siwek) * Catch if logged vectors do not contain only atomic types. (Johanna Amann) 2.0-beta-47 | 2011-11-16 08:24:33 -0800 * Catch if logged sets do not contain only atomic types. (Johanna Amann) * Promote libz and libmagic to required dependencies. (Jon Siwek) * Fix parallel make from top-level to work on more platforms. (Jon Siwek) * Add decode_base64_custom(). Addresses #670 (Jon Siwek) * A bunch of Sphinx-doc reorgs and polishing. (Jon Siwek) 2.0-beta-28 | 2011-11-14 20:09:28 -0800 * Binary packaging script tweaks. We now require CMake 2.8.6. (Jon Siwek) * More default "weird" tuning for the "SYN_with_data" notice. (Seth Hall) * Tiny bugfix for http file extraction along with test. (Seth Hall) 2.0-beta-21 | 2011-11-06 19:27:22 -0800 * Quickstart doc fixes. (Jon Siwek) 2.0-beta-19 | 2011-11-03 17:41:00 -0700 * Fixing packet filter test. (Robin Sommer) 2.0-beta-12 | 2011-11-03 15:21:08 -0700 * No longer write to the PacketFilter::LOG stream if not reading traffic. (Seth Hall) 2.0-beta-10 | 2011-11-03 15:17:08 -0700 * Notice framework documentation update. (Seth Hall) * Fixing compiler warnings (addresses #388) (Jon Siwek) 2.0-beta | 2011-10-27 17:46:28 -0700 * Preliminary fix for SSH login detection: we need a counted measure of payload bytes (not ack tracking and not with the IP header which is what we have now). (Seth Hall) * Fixing send_id() problem. We no longer update &redef functions. Updating code on the fly isn't fully supported. (Robin Sommer) * Tuning the format of the pretty-printed alarm summaries. (Robin Sommer) 1.6-dev-1508 | 2011-10-26 17:24:50 -0700 * Updating submodule(s). (Robin Sommer) 1.6-dev-1507 | 2011-10-26 15:10:18 -0700 * Baseline updates. (Robin Sommer) 1.6-dev-1506 | 2011-10-26 14:48:43 -0700 * Updating submodule(s). (Robin Sommer) 1.6-dev-1505 | 2011-10-26 14:43:58 -0700 * A new base script that pretty-prints alarms in the regular summary. (Robin Sommer) * Adding a dummy log writer WRITER_NONE that just discards everything. (Robin Sommer) 1.6-dev-1498 | 2011-10-26 14:30:15 -0700 * Adding instructions to local.bro how to do ACTION_ALARM by default. (Seth Hall) 1.6-dev-1495 | 2011-10-26 10:15:58 -0500 * Updated unit test baselines. (Seth Hall) 1.6-dev-1491 | 2011-10-25 20:22:56 -0700 * Updating submodule(s). (Robin Sommer) 1.6-dev-1482 | 2011-10-25 19:08:32 -0700 * Fixing bug in log managers predicate evaluation. (Robin Sommer) 1.6-dev-1481 | 2011-10-25 18:17:03 -0700 * Fix a problem with DNS servers being logged that aren't actually servers. (Seth Hall) * Changed generated root cert DN format for RFC2253 compliance. (Jon Siwek) * Removed :bro doc directives from notice documentation. (Seth Hall) * New notice framework docs. (Seth Hall) * Adding sub messages to emails. (Seth Hall) * Adding extra fields to smtp and http to track transaction depth. (Seth Hall) * Fix for SSH login detection heuristic. (Seth Hall) * Removed some fields from http analysis that weren't commonly needed or were wrong. (Seth Hall) * Updated/fixed MSIE version parsing in the software framework. (Seth Hall) * Update Mozilla trust roots to index certs by subject distinguished name. (Jon Siwek) * weird.bro rewrite. (Seth Hall) * More notice email tuning. (Seth Hall) * Slightly restructured http file hashing to fix a bug. (Seth Hall) * Changed the notice name for interesting ssh logins to correctly reflect semantics of the notice. (Seth Hall) * Field name change to notice framwork. $result -> $action - $result is renamed to $action to reflect changes to the notice framework since there is already another result-like field ($suppress_for) and there may be more in the future. - Slipped in a change to add connection information to notice emails too. (Seth Hall) * Small script refinements and documentation updates. (Seth Hall) * Pass over upgrade guide. (Robin Sommer) 1.6-dev-1430 | 2011-10-21 10:39:09 -0700 * Fixing crash with unknown debug streams. Closes #643. (Robin Sommer) * Code to better handle interpreter errors, which can now be turned into non-fatal runtime errors rather than immediate aborts. (Robin Sommer). * Remove old make-src-packages script. (Jon Siwek) * Fixing a bunch of format strings. Closes #567. (Robin Sommer) * Cleaning up some distribution files. (Robin Sommer) * Various test, doc, and installation fixes/tweaks. (Seth Hall, Jon Siwek and Robin Sommer). * Varios smaller policy fixes and tweaks (Seth Hall). * Moving docs from web server into distribution. (Robin Sommer) * Fixing more (small) memory leaks. (Robin Sommer) * Profiling support for DNS_Mgr and triggers. With misc/profiling.bro, both now report a line in prof.log with some counters on usage. (Robin Sommer) * Fixing DNS memory leaks. Closes #534. (Robin Sommer) * Fix code for disabling analyzers. Closes #577. (Robin Sommer) * Changed communication option from listen_encrypted to listen_ssl. (Seth Hall) * Modification to the Communication framework API. (Seth Hall) - Simplified the communication API and made it easier to change to encrypted connections by not having separate variables to define encrypted and unencrypted ports. - Now, to enable listening without configuring nodes just load the frameworks/communication/listen script. - If encrypted listening is desired set the following: redef Communication::listen_encrypted=T; * Connection compressor now disabled by default. Addresses #559. (Robin Sommer) 1.6-dev-1372 | 2011-10-06 18:09:17 -0700 * Filtering some potentially high-volume DNS weirds. (Robin Sommer) * DNS now raises DPD events. Closes #577. (Robin Sommer) * Fixing a bunch of compiler warnings. (Robin Sommer) * Remote logs are auto-flushed if the last write was longer than a second ago. Addresses #498. (Robin Sommer) * Fix missing from previous MIME commit. (Robin Sommer) 1.6-dev-1366 | 2011-10-06 17:05:21 -0700 * Make CompHash computation/recovery for functions deterministic. Closes #636. (Jon Siwek) * Removing unnecessary @load in local.bro. (Robin Sommer) * Optimizing some MIME code. (Robin Sommer) * Speed improvements in logging code. (Robin Sommer) * Consolidating some node-specific functionality from scripts in broctl repo. (Jon Siwek) * Another fix the for 1xx script code. (Robin Sommer) 1.6-dev-1352 | 2011-10-05 16:20:51 -0700 * Fix for optional HTTP::Info status_code. (Jon Siwek) * Teaking some external testing scripts. (Jon Siwek) * HTTP bug fix reported by Martin Holste. (Seth Hall) * More script tuning. (Seth Hall) - Moved some of the weird events back to the base/ directory. - SSL fixes, updates, and performance optimization. * More adjustment to reduce Weird volumes. (Seth Hall) * Fixed an error when calculating x509 certificate hashes (reported by Martin Holste). (Seth Hall) * Clean up to cluster framework to make event handling clearer. (Seth Hall) * Fixed a bug in the notice framework. (Seth Hall) * Bug fix for FTP analysis script. (Seth Hall) 1.6-dev-1333 | 2011-09-29 22:29:51 -0700 * Fixing a number of memory leaks. (Robin Sommer) * Loaded_scripts.log is indented with spaces now and makes more sense to look at. (Seth Hall) * Teach HTTP parser to derive content length of multipart/byteranges bodies. Addresses #488. (Jon Siwek) * Change logging of HTTP 1xx responses to occur in their own columns. Addresses #411. (Jon Siwek) * Fix handling of HTTP 1xx response codes. Addresses #411). * Taking advantage of yet another trick to get installed browser plugins. (Seth Hall) - With the software-browser-plugins script you can watch for Omniture advertising servers to grab the list of installed plugins. - I reorganized the plugin detection a bit too to abstract it better. - Removed the WEB_ prefix from all of the Software::Type HTTP enums. They were essentially redundant due to the full name already being HTTP::SERVER (for example). 1.6-dev-1316 | 2011-09-28 16:50:05 -0700 * Unit test cleanup. Updated README and collected coverage-related tests in a common dir. (Jon Siwek) * Fixes for known-services. (Seth Hall) * Ported and 2.0ized the capture-loss script. (Seth Hall) * Communication fix and extension.(Robin Sommer) - Removing unnecessary log flushing. Closes #498. - Adding new BiF disconnect() that shuts a connection to a peer down. - terminate_connection() now first flushes any still buffered log messages. * Fix for high SSL memory usage by adding &transient attribute to top-level SSL pac array type. Closes #574. (Robin Sommer) * Fix a small bug in the metrics framework. (Seth Hall) * Temporarily removing scripts that aren't ready to be included. Will return before next release. (Seth Hall) * New SSL policy scripts. (Seth Hall) - protocols/ssl/expiring-certs uses time based information from certificates to determine if they will expire soon, have already expired, or haven't yet become valid. - protocols/ssl/extract-certs-pem is a script for taking certs off the line and converting them to PEM certificates with the openssl command line tool then dumping them to a file. * Notice::type_suppression_intervals: table[Notice::Type] of interval can be used to modify the suppression intervals for entire types of notices. (Seth Hall) * EOF SSL protocol violations are only generated a single time now. (Seth Hall) * Script level fixes. (Seth Hall) - Fixed a type name conflict in the Known namespace. - Fixed a DPD framework bug that was causing Reporter messages. - Fixed the notice_policy log. - Predicate functions are now logged. - Predicate functions are now optional. If not given, it's assumed that the result should always apply. (Seth Hall) - Fix a problem with accidental and mistaken HTTP log lines. 1.6-dev-1293 | 2011-09-22 19:44:37 -0700 * Smaller script tweaks. (Seth Hall) * Duplicate notice suppression. (Seth Hall) - Duplicate notices are discovered with the new Notice::Info field $identifier. It's a string that is left up to the notice implementor to define which would indicate a fundamentally duplicate notice. The field is optional and if it's not included it's not possible for notice suppression to take place. - Duplicate notices are suppressed by default for the interval defined by the Notice::default_suppression_interval variable (1 hour by default). - A new notice action was defined ACTION_NO_SUPPRESS to prevent suppression for a specific notice instance. A convenience set named not_suppressed_types was also created to not suppress entire notice types. - A new field was added to the PolicyItem type to modify the length of time a notice should be suppressed if the predicate matches. The field is named $suppress_for. This name makes the code more readable like this: $suppress_for = 1day - New events were created to give visibility into the notice framework's suppression activity. - event Notice::begin_suppression(n: Notice::Info) - event Notice::suppressed(n: Notice::Info) - event Notice::end_suppression(n: Notice::Info) - The suppression.bro script doesn't have a baseline because it is causing a segfault in Bro. This one test is the reason that this is being integrated into a branch instead of master. (Seth Hall) * Fix crash on exit. Addresses #607. (Jon Siwek) * Fix PktSrc setting next_timestamp even when no packet available. (Jon Siwek) * Fix lack of NUL-termination in to_upper/to_lower BIF's return val. (Jon Siwek) * Fixing unit tests and some minor bugs. (Jon Siwek) * Fix broctl cluster log rotation. Addresses #619. (Jon Siwek) * Added session ID to the SSL logging. (Seth Hall) * Adding "install-aux" target + updating bro-aux submodule. (Jon Siwek) * Cleaning up INSTALL and README. (Jon Siwek) * Remove $Id$ tags. (Jon Siwek) * Remove policy.old directory. Addresses #511. (Jon Siwek) * Small rework with ssl base script to reduce memory usage. (Seth Hall) * Updated the mozilla root certs. (Seth Hall) 1.6-dev-1261 | 2011-09-15 17:13:55 -0700 * Memory leak fixes. Addresses #574 (Jon Siwek) * Add configure options for ruby/bindings integration. (Jon Siwek) * Fix filter path_func to allow record argument as a subset of stream's columns. Addresses #600. (Jon Siwek) * Log rotation is now controlled directly through Filter records. (Jon Siwek) * Fix indexing for record types with optional fields. Addresses #378 (Jon Siwek) 1.6-dev-1248 | 2011-09-15 16:01:32 -0700 * Removed custom malloc() implementation for FreeBSD. Closes #557. (Jon Siwek) * Testing/external scripts no longer compute MD5 checksums for SMTP entities. (Robin Sommer) * External tests no longer include the full content of mismatching files in the diagnostics output. (Robin Sommer) 1.6-dev-1241 | 2011-09-14 22:51:52 -0400 * Fixing a major memory utilization issues with SSL analysis. (Seth Hall) * Enhancements to HTTP analysis: (Seth Hall) - More options for the header-names.bro script. - New script for logging header names and values. Closes #519. (Seth Hall) - HTTP body size measurement added to http.log. - The value of the content-length headers has now been removed in the default output but it could be added back locally at an installation by a user. - Added fields to indicate if some parsing interruption happened during the body transfer. Closes #581 (Seth Hall) * Misc smaller usability and correctness updates: (Seth Hall) - Removed an notice definition from the base SSL scripts. - Moved a logging stream ID into the export section for known-services and bumped priority for creating the stream. - Adding configuration knobs for the SQL injection attack detection script and renaming the HTTP::SQL_Injection_Attack notice to HTTP::SQL_Injection_Attack_Against - Bumped priority when creating Known::CERTS_LOG. - Fixing a warning from the cluster framework. (Seth Hall) * Bugfix for log writer, which didn't escape binary stuff in some situations. Closes #585. (Robin Sommer) * A larget set of changes to the testing/external infrastructure. The traces for external test-suites are no longer kept inside the repositories themselves but downloaded separately via curl. This is because git is pretty bad at dealing with large files. See the README for more information. (Robin Sommer) 1.6-dev-1221 | 2011-09-08 08:41:17 -0700 * Updates for documentation framework and script docs. (Jon Siwek) * The script level PF_RING support isn't working so removing it. (Seth Hall) * Delete SSL certificates from memory after ssl_established event. (Seth Hall) * Small fixes for SSL analysis. (Seth Hall) 1.6-dev-1212 | 2011-09-07 16:15:28 -0700 * Internally, the UID generation can now return values from different pool for better reproducability in testing mode. (Gilbert Clark). * Added new BiF unique_id_from(pool: string, prefix: string) that allows the user to specify a randomness pool. (Gilbert Clark) 1.6-dev-1198 | 2011-09-07 11:03:36 -0700 * Extended header for ASCII log that make it easier for scripts to parse Bro log files. (Gilbert Clark) * Potential fix for rotation crashes. Addresses #588. (Robin Sommer) * Added PF_RING load balancing support to the scripting layer, enabled by loading the misc/pf-ring-load-balancing script. (Seth Hall) * Added a BiF setenv() for setting environment variables. (Seth Hall) 1.6-dev-1184 | 2011-09-04 09:34:50 -0700 * FindPCAP now links against thread library when necessary (e.g. PF_RING's libpcap). (Jon Siwek) * Install binaries with an RPATH. (Jon Siwek) * Fix for a case where nested records weren't coerced even though possible. (Jon Siwek) * Changed ASCII writer to delay creation of log after rotation until next write. * Changed default snaplen to 65535 and added a -l/--snaplen command line option to set it explicitly. Addresses #447. (Jon Siwek) * Various updates to logging framework. (Seth Hall) * Changed presentation of enum labels to include namespace. (Jon Siwek) * HTTP analyzer is now enabled with any of the HTTP events. (Seth Hall) * Fixed missing format string that caused some segfaults. (Gregor Maier) * ASCII writer nows prints time interval with 6 decimal places. (Gregor Maier) * Added a Reporter::fatal BIF. (Jon Siwek) * Fixes for GeoIP support. Addresses #538. (Jon Siwek) * Fixed excessive memory usage of SSL analyzer on connections with gaps. (Gregor Maier) * Added a log postprocessing function that can SCP rotated logs to remote hosts. (Jon Siwek) * Added a BiF for getting the current Bro version string. (Jon Siwek) * Misc. doc/script/test cleanup. (Jon Siwek) * Fixed bare-mode @load dependency problems. (Jon Siwek) * Fixed check_for_unused_event_handlers option. (Jon Siwek) * Fixing some more bare-mode @load dependency issues (Jon Siwek) * Reorganizing btest/policy directory to match new scripts/ organization. Addresses #545 (Jon Siwek) * bro scripts generated from bifs now install to $prefix/share/bro/base. Addresses #545 (Jon Siwek) * Changeed/fixed some cluster script error reporting. (Jon Siwek) * Various script normalization. (Jon Siwek) * Add a test that checks each individual script can be loaded in bare-mode. Adressess #545. (Jon Siwek) * Tune when c$conn is set. Addresses #554. (Gregor Maier) * Add ConnSize_Analyzer's fields to conn.log. (Gregor Maier) * Fixing bug in "interesting hostnames" detection. (Seth Hall) * Adding metrics framework intermediate updates. (Seth Hall) 1.6-dev-1120 | 2011-08-19 19:00:15 -0700 * Fix for the CompHash fix. (Robin Sommer) 1.6-dev-1118 | 2011-08-18 14:11:55 -0700 * Fixing key size calculation in composite hash code. (Robin Sommer) 1.6-dev-1116 | 2011-08-18 10:05:07 -0700 * Remove the 'net' type from Bro (addresses #535). * Fix H3 assumption of an 8-bit byte/char. (Jon Siwek) * Allow reading from interface without additional script arguments. Explicitly passing in '-' as an additional command line argument still allows reading a script from stdin. (Jon Siwek) * SSH bruteforcing detection now done with metrics framework. (Seth Hall) * Updates for SQL injection attack detection to match the metrics framework updates. (Seth Hall) * Metrics framework now works on cluster setups. (Seth Hall) * Reclassifying more DNS manager errors as non-fatal errors. (Robin Sommer) * Fix ConnSize_Analyzer when used in conjunction with connection compressor. (Gregor Maier) * Fix reporter using part of the actual message as a format string. (Jon Siwek) 1.6-dev-1095 | 2011-08-13 11:59:07 -0700 * A larger number of script documentation updates. Closes #543. (Jon Siwek) * Workaround for FreeBSD CMake port missing debug flags. (Jon Siwek) * piped_exec() can now deal with null bytes. (Seth Hall) * Fix vector initialization for lists of records with optional types. Closes #485. (Jon Siwek) * Fix redef'ing records with &default empty set fields. Closes #460. (Jon Siwek) * Fix ConnSize_Analyzer when used in conjunction with the connection compressor. (Gregor Maier) * Fix reporter using part of the actual message as a format string. (Jon Siwek) * Fixing reporter's location tracking. Closes #492. (Robin Sommer) * Turning DNS errors into warnings. Closes #255. (Robin Sommer) * Logging's path_func now receives the log record as argument. Closes #555. (Robin Sommer) * Functions can now be logged; their full body gets recorded. Closes #506. (Robin Sommer) * Bugfix for hostname notice email extension. (Seth Hall) * Updates for notice framework. (Seth Hall) - New ACTION_ADD_GEODATA to add geodata to notices in an extension field named remote_location. - Loading extend-email/hostnames by default now that it only does anything when the ACTION_EMAIL action is applied (finally). * Updates to local.bro (Seth Hall) * Added the profiling script. (Seth Hall) * Updates for SSH scripts. (Seth Hall) * ConnSize analyzer is turned on by default now. (Seth Hall) * Updates for the build system and site local scripts for cluster. (Seth Hall) * HTTP now uses the extract_filename_from_content_disposition function. (Seth Hall) * Major SMTP script refactor. Closes #509. (Jon Siwek and Seth Hall) * New variable Site::local_nets_table in utils/site for mapping address to defined local subnet. * Metrics framework updates, more to come. (Seth Hall) 1.6-dev-1061 | 2011-08-08 18:25:27 -0700 * A set of new/changed tests regarding the new policy script organisation. (Robin Sommer) 1.6-dev-1058 | 2011-08-08 16:15:18 -0700 * Reorganisation of the scripts that Bro loads by default. (Seth Hall) - policy/ renamed to scripts/ - By default BROPATH now contains: - scripts/ - scripts/policy - scripts/site - The scripts in scripts/base/protocols/ only do logging and state building. - All of scripts/base/ is loaded by by default. This can however be disabled by switching Bro into "bare mode" using the new command-line option --bare-mode (or -b). The cripts in scripts/base/ don't use relative path loading to ease use of bare mode (to copy and paste that script). - The scripts in scripts/base/frameworks/ add functionality without causing any additional overhead. - All "detection" activity happens through scripts in scripts/policy/. - bro.init was renamed to base/init-bare.bro, and base/all.bro was renamed to init-default.bro. - local.bro now loads more functionality from policy/ and adds more documentation. (Seth Hall) * Adding default_path_func() to the logging framework that makes the default naming scheme script-level controlled. (Robin Sommer) * Reworking logging's postprocessor logic so that postprocessor commands are no longer run by the log writers themselves, but instead by a script level function. (Robin Sommer) * The communication subsystem is now by default off and must be enabled explicitly with a new BiF, enable_communication(). Closes #540. (Robin Sommer) * The hostname notice email extension now only add hostnames for emailed noticed. (Seth Hall) * Cleaning up doc generation. (Seth Hall) 1.6-dev-1044 | 2011-08-05 19:07:32 -0700 * Fixing memory (and CPU) leak in log writer. * Fixing crash in memory profiling. (Robin Sommer) * Fix compiler warning. (Robin Sommer) * Fixing missing sync in cluster setup. (Robin Sommer) 1.6-dev-1038 | 2011-08-05 18:25:44 -0700 * Smaller updates to script docs and their generation. (Jon Siwek) * When using a `print` statement to write to a file that has raw output enabled, NUL characters in string are no longer interpreted into "\0", no newline is appended afterwards, and each argument to `print` is written to the file without any additional separation. (Jon Siwek) * Test portatibility tweaks. (Jon Siwek) * Fixing PktSrc::Statistics() which retured bogus information offline mode. Closes #500. (Jon Siwek) * --with-perftools configure option now assumes --enable-perftools. Closes #527. (Jon Siwek) 1.6-dev-1018 | 2011-07-31 21:30:31 -0700 * Updating CHANGES. (Robin Sommer) 1.6-dev-1016 | 2011-07-30 18:34:28 -0700 * Install example config files dynamically. They'll only get installed when the distribution version differs from existing version on disk. (Jon Siwek) * Fixed memory leak in SSL analyzer. (Seth Hall) * Beginning rework of metrics interface. (Seth Hall) * New/updated unit tests for scripts. (Jon Siwek) * New/updated documentstion for scripts. (Jon Siwek) * A number of fixes for scripts in utils/. (Jon Siwek) 1.6-dev.244 Thu Jul 28 17:08:21 PDT 2011 - mask_addr() now returns subnet (addresses #512). (Jon Siwek) - Normalize Notice::Type identifiers per convention (closes #484). (Jon Siwek) - Fixing default-loaded-scripts test for BSD systems. (Jon Siwek) - New piped_exec() BiF for pipeing data into an external command. (Jon Siwek) 1.6-dev.242 Mon Jul 25 21:42:39 PDT 2011 - Adding a documentation coverage test. (Jon Siwek) - The CMake targets for generating reST docs from policy scripts are now automatically generated via the genDocSourcesList.sh script. (Jon Siwek) - Fixed a number of script error. (Jon Siwek) - Fixes to relative @load'ing. (Jon Siwek) - Fixes to tests. (Robin Sommer) 1.6-dev.240 Sun Jul 24 15:14:26 PDT 2011 - Updated tests and test baselines. (Jon Siwek) - ASCII log writer now prints time values w/ constant 6 digit precision. (Jon Siwek) - Many policy script updates acrsso the board (Seth Hall). - Moving devel-tools to bro-aux. (Robin Sommer) - BugFix for disable_analyzer(), which could cause crashes with some analyzers. (Robin Sommer) - Bugfix for potential segfault in DebugLogger. (Robin Sommer) 1.6-dev.226 Thu Jul 21 15:23:39 PDT 2011 - Extensions to the @load and @unload process. (Jon Siwek) * Make @load statements recognize relative paths. For example a script can do "@load ./foo" to load a script named foo.bro that lives in the same directory or "@load ../bar" to load a script named bar.bro in the parent directory, even if those directories are not contained in BROPATH. * Reimplementation of the @prefixes statement. (Closes #486) Any added prefixes are now used *after* all input files have been parsed to look for a prefixed, flattened version of the input file somewhere in BROPATH and, if found, load it. For example, if "lcl" is in @prefixes, and site.bro is loaded, then a file named "lcl.site.bro" that's in BROPATH would end up being automatically loaded as well. Packages work similarly, e.g. loading "protocols/http" means a file named "lcl.protocols.http.bro" in BROPATH gets loaded automatically. * Fix @unload'd files from generating bro_script_loaded event. * Updates to tests. 1.6-dev.225 Wed Jul 20 17:10:41 PDT 2011 - IRC improvements (Jon Siwek). Including: * Shorten what's displayed in the IRC's log mime_type column for DCC transfers. * Add IRC unit tests. * Fix IRC analyzer supplying wrong type to irc_dcc_message event. * Removed irc_client and irc_server events. * Added is_orig arguments to all other irc events. * Fix analyzer not recognizing Turbo DCC extension message format. * Fix analyzer not generating irc_dcc_message event when irc_privmsg_message event doesn't have a handler registered. - Fixing tests that need a diff canonifier. (Jon Siwek) 1.6-dev.223 Tue Jul 19 19:10:36 PDT 2011 - Adding a script to update CHANGES and VERSION. (Robin Sommer) 1.6-dev.218 Tue Jul 19 18:16:44 PDT 2011 - Comprehensive policy script overhaul/rewrite. (Seth Hall) Changes are too extensive to list individually. - Removing undocumented -H command line flag. (Robin Sommer) - Fixing many tests. (Everybody) - Fixing 0-chunk bug in remote logging. (Robin Sommer) - $PATH is now appropriately set by the bro-path-dev.(sh|csh) scripts. (Seth Hall) - Making valgrind a bit more happy. (Robin Sommer) - New BiF record_field_vals() that returns the fields of a record in a table with meta-information. (Robin Sommer) - Adding a script in aux/devel-tools that extracts a connection from a trace based on uid. (Robin Sommer) - Fixing bug causing crash when running without arguments. (Robin Sommer) - A new event bro_script_loaded() raised for each policy script loaded. Also removing the -l command-line option as that can now be done at the script-level. (Robin Sommer) - Fixing memory leaks. (Gilbert Clark, Seth Hall, Robin Sommer) - Many SSL analysis improvements and fixes. (Seth Hall) - Fixing bug with event priorities potentially being ignored for the handler. (Robin Sommer) - Overhauling the internal reporting of messages to the user. The new Reporter class is now in charge of reporting all errors, warnings, informational messages, weirds, and syslogs; and it passes everything through the script layer. (Robin Sommer) * Removed the alarm statement and the alarm_hook event. (Robin Sommer) - Adding new policy file test-all.bro that loads all other policies. This is for testing only. (Robin Sommer) - A new framework for doing regression testing with larger traces and more complex Bro configurations in testing/external. (Robin Sommer) - Many updates to script doc generation. (Jon Siwek) 1.6-dev.146 Sat Jun 25 18:12:27 PDT 2011 - DNS mapping are now becoming invalid when an entry's TTL expires. (Thomas Other) - Reworking how Bro tracks which scripts are already loaded. Rather than paths, Bro now tracks inode numbers. (Jon Siwek) - New BiF netstats() to query packet capture statistics. The netstats script now uses the new BiF to periocally report packets drops. The net_stats_update() event and the heartbeat_interval global went away. (Seth Hall) - Fixing bug with logging &optional records. Closes #476. (Robin Sommer) - Fixing istate.events-ssl test failing because of expired cert. (Jon Siwek) - A large number of improvements and fixes for Bro's doc mode. (Jon Siwek) - Significant updates for RPC and NFS analyzers (Gregor Maier) * Unify semantics for UDP and TCP connections. * RPC can now log to a log file if desired. * Portmapper can now log general activity to a log file and also log actual port mappings. * NFS analyzer now supports significantly more procedure calls as as file name tracking and file content extraction. - NetBIOS fixes. (Jon Siwek) - A number of unit tests are more robust and portable. (Jon Siwek) - A new BiF unique_id() that returns a string that's unique across Bro instaces with high probablity. (Robin Sommer) - Complete rewrite of the BinPAC SSL analyzer. (Seth Hall) * DER certificates are extracted as strings to be used with corresponding BiFs. * x509_verify function to verify single certs and/or full certificate chains. * Removed hand written SSL analyzer. * The ssl.bro script is just a place-holder for now. New version will come with the other new scripts. - New syslog analyzer. (Seth Hall) - @load now supports loading a directory. With a directory "foo" somewhere in BROPATH, "@load foo" now checks if there's a file "foo/__load__.bro". If so, it reads that file in. (Robin Sommer) - ASCII logger now escapes non-printable characters. Closes #450. (Robin Sommer) - Packaging tweaks and rewrite of 'dist' target. (Jon Siwek) - Changes to allow DEB packaging via CPack, addresses #458. (Jon Siwek) - An extension to the ICMP analyzer to handle redirects. Julien Sentier - Removing old istate test-suite. (Robin Sommer) - A hack to report missing GeoIP support only once. This closes #357, but #455 captures the need for a more general solution. (Robin Sommer) - Bugfix: vectors in records were not initalized. Closes #421. (Robin Sommer) - If IPv6 default is not compiled in, the default BPF filters now excludes IPv6 packets. (Robin Sommer) - New bif bro_has_ipv6() to check whether IPv6 support is compiled in. (Robin Sommer) - Updating btests and a Makefile. "make" now runs all the tests. (Robin Sommer) - Moving the test-scripts from the old test-suite over to btest. (Robin Sommer) - Fix for major bug in POP3 analyzer, which didn't recognize '.' terminators in multi-line replies if the terminator was bare (no newline). This caused it to ignore the rest of the session that it's analyzing. (Vern Paxson) - Fix compiler warning with gcc-4.4.4 (Gregor Maier) - Adding example documentation for a script's use of logging features. (Jon Siwek) - Adding &log attribute to static attr_names array. (Jon Siwek) - Bro can now track packet and byte counts per connection. (Gregor Maier) * If 'use_conn_size_analyzer' is true, the event engine tracks number of packets and raw IP bytes per connection. If report_conn_size_analyzer is true, these values are included as four new columns into conn.log * I changed conn.bro so that the value of report_conn_size_analyzer follows that of use_conn_size_analyzer. For the new conn.log, we probably want to get rid of report_conn_size_analyzer anyway. - Fixing numerous compiler warnings and portability issues. (All) - Switching vectors from being 1-based to 0-based. Note that this is a change that break backwards-compatibility. (Robin Sommer) - Increasing serialization format version for the recent 64-bit changes. (Robin Sommer) - Support for (mixed) MPLS and VLAN traffic, and a new default BPF filter. (Seth Hall and Robin Sommer) * Merging in the patch from #264, which provides support for mixed VLAN and MPLS traffic. * Changing Bro's default filter from being built dynamically to being a static "ip or not ip". To get the old behaviour back (i.e., the dynamically built filter), redef "all_packets" to false. * print-filter.bro now always prints the filter that Bro is actually using, even if overriden from the command line. (Robin Sommer) - Changing the HTTP's analyzers internals to use 64-bit integers. (Gregor Maier). - Fixing bug with deleting still unset record fields of table type. (Robin Sommer) 1.6-dev.99 Fri Apr 22 22:10:03 PDT 2011 - Extending the connection record with a unique identifier. (Robin Sommer) type connection: record { [...] id: string; }; These identifiers very likely unique even across independent Bro runs. - Delete operator for record fields. (Robin Sommer) "delete x$y" now resets record field "x" back to its original state if it is either &optional or has a &default. "delete" may not be used with non-optional/default fields. - Fixing bug with nested record coercions. (Robin Sommer) - Fixing a do_split() bug. (Seth Hall) 1.6-dev.94 Thu Apr 21 19:51:38 PDT 2011 - Fixing generation of config.h. (Jon Siwek) - Updates and tests for NetBIOS name BiF. (Seth Hall) - Fixing do_split bug(), and adding a test. (Seth Hall) - When Bro is given a PRNG seed, it now uses its own internal random number generator that produces consistent results across sytems. Note that this internal generator isn't very good, so it should only be used for testing purpses. (Robin Sommer) - The BTest configuration now sets the environemnt variables TZ=UTC and LANG=C to ensure consistent results. (Robin Sommer) - Logging fixes. (Robin Sommer) 1.6-dev.88 Wed Apr 20 20:43:48 PDT 2011 - Implementation of Bro's new logging framework. We will document this separately. (Robin Sommer) - Already defined record types can now be further extended via the '+=' operator. The added fields must be either &optional or have a &default value. (Robin Sommer) Example: type Foo: record { a: count; b: count &optional; }; redef record Foo += { c: count &default=42; d: count &optional; }; global f: Foo = [$a=21]; print f; Output: [a=21, b=, c=42, d=] - Enabling assignment of empty vectors ("vector()"). (Robin Sommer) - Fixing attributes to allow &default attributes to be associated with records fields of type tables/sets/vector. (Robin Sommer) - '[]' is now a valid record constructor. (Robin Sommer) - A instance of a record type A is now coercable into one of type B if the fields of type A are a subset of those of type B. (Robin Sommer) - A number of bug fixes and enhancements for record/set/table/vector coercion. (Robin Sommer) - Fixing a problem with records that have optional fields when used as table/set indices. Addresses #367. (Robin Sommer) - Fixing an off-by-one error in join_string_vec(). (Seth Hall) - Updating to_count() to cope with 64bit ints. (Seth Hall) - A new BiF count_to_v4_addr() to turn a count into an IPv4 address. (Seth Hall) 1.6-dev.80 Mon Apr 18 14:50:54 PDT 2011 - New framework for generating documentation from Bro scripts. (Jon Siwek) This includes: * Changes to Bro's scanner/parser to facilitate automatic generation of Bro policy script documentation in reStructuredText format. * New command line flags -Z/--doc-scripts to enable the new doc generation mode. * Changes to bifcl to pass comments starting with "##" through into the generated .bro script. * A "doc" build target for the top-level Makefile to first generate reStructuredText for a defined set of Bro policy scripts, and then run that through Sphinx to create HTML documentation. 1.6-dev.78 Mon Apr 18 12:52:55 PDT 2011 - Adding files to CMake build targets so they show up in generated IDE projects. This addresses #413. (Jon Siwek) - Fix unnecessary config.h preprocessor (re)definitions. This addresses #414. (Jon Siwek) - Updating istate tests. (Robin Sommer) - Adding files to CMake build targets so they show up in generated IDE projects. - Adding new environment variable BRO_SEED_FILE to set the seed file for the random number generator. (Robin Sommer) 1.6-dev.71 Fri Apr 1 16:06:33 PDT 2011 - Removing code for the following no longer supported functionality. * Trace rewriting. * DFA state expiration in regexp engine. * Active mapping. * Unused hash functions. (Robin Sommer) - Fixing crashes when SSL is not configured correctly. (Robin Sommer) 1.6-dev.66 Tue Mar 29 21:52:01 PDT 2011 - Initial btest setup (Don Appleman and Robin Sommer) - Porting the istate tests to btest (not finished) (Robin Sommer) 1.6-dev.63 Mon Mar 21 16:31:15 PDT 2011 - Changes to the way user-modifiable config files are installed (Jon Siwek) * Duplicates of the distribution's configuration files are now always installed with a .example suffix * Added --binary-package configure option to toggle configure logic specific to the creation of binary packages. * When not in binary packaging mode, `make install` never overwrites existing configure files in case they've been modified. The previous behavior (CMake's default) would only avoid overwriting modified files if one consistently uses the same build directory and doesn't reconfigure. - Fixed an issue with Mac package's pre-install script not preserving ACLs. (Jon Siwek) - Minor cleanup/refactor of the make-mac/rpm-packages scripts. (Jon Siwek) - Add explicit CMake check for compiler. (Jon Siwek) - Add alternative way to set BROPATH for running bro from build/ dir. (Jon Siwek) - Fixing compiler warnings (Gregor Maier) - Remvoing leftover local variables that caused compile error on Mac OS X. (Gregor Maier) 1.6-dev.53 Fri Feb 25 17:03:05 PST 2011 - Fixing file detector leak in remote communication module. (Scott Campbell) - Updating independent-state tests to work with new setup. (Robin Sommer) 1.6-dev.49 Fri Feb 25 15:37:28 PST 2011 - Enum IDs can have explicitly defined values. (Gregor Maier) - Extensions for the built-in function compiler, bifcl. (Gregor Maier) * Support for policy-layer namespaces. * Support for type declarations in bif files (with access them from C++) * Extended const declarations in bif files. See http://bro.icir.org/devel/bif-doc for more information. 1.6-dev.48 Fri Feb 25 10:53:04 PST 2011 - Preliminary TCP Reassembler fix: deliver data after 2GB by disabling the unused seq_to_skip feature. (Gregor Maier) 1.6-dev.47 Fri Feb 25 10:40:22 PST 2011 - Fixing endianess error in XDR when data is not 4-byte aligned. (Gregor Maier) - Fix for Val constructor with new int64 typedefs. (Gregor Maier) - Updated fix for OS X 10.5 compile error wrt llabs(). (Gregor Maier) - Fix more compiler warning wrt printf format strings. (Gregor Maier) 1.6-dev.45 Tue Feb 8 21:28:01 PST 2011 - Fixing a number of compiler warnings. (Seth Hall and Robin Sommer) 1.6-dev.44 Tue Feb 8 20:11:44 PST 2011 - A number of updates to the SSL analyzer, including support for new ciphers; SSL extensions; and bug fixes. The analyzer does not longer throw weird for exceeding a predefined cipherspec_size anymore. (Seth Hall and Rmkml). - The various split*() BiFs now handle strings containing null bytes correctly. (Seth Hall) - Adding new aux/btest submodule. This is a framework we will use in the future for doing unit tests. (Robin Sommer) 1.6-dev.41 Mon Feb 7 13:43:56 PST 2011 - Smarter way to increase the parent/child pipe's socket buffer. (Craig Leres). - Fixing bug with defining bro_int_t and bro_uint_t to be 64 bits wide on some platforms. (Robin Sommer) 1.6-dev.39 Mon Jan 31 16:42:23 PST 2011 - Login's confused messages now go through weird.bro. (Robin Sommer) 1.6-dev.36 Mon Jan 31 08:45:35 PST 2011 - Adding more configure options for finding dependencies, (Jon Siwek) --with-flex=PATH path to flex executable --with-bison=PATH path to bison executable --with-perl=PATH path to perl executable --with-python=PATH path to Python interpreter --with-python-lib=PATH path to libpython --with-python-inc=PATH path to Python headers --with-swig=PATH path to SWIG executable - Fixing typo in PCAPTests.cmake (Jon Siwek) 1.6-dev.33 Mon Jan 24 15:29:04 PST 2011 - Fixing bug in SMB analyzer. (Robin Sommer) - Configure wrapper now deletes previous CMake cache (Jon Siwek) - Fix for the --with-binpac configure option. (Jon Siwek) 1.6-dev.30 Thu Jan 20 16:32:43 PST 2011 - Changed configure wrapper to create config.status. (Jon Siwek) 1.6-dev.29 Thu Jan 20 16:29:56 PST 2011 - Fixing little problem with initialization of Bro-to-Bro event communication. (Christian Kreibich) 1.6-dev.27 Thu Jan 20 13:52:25 PST 2011 - Fine-tuning of the HTTP analyzer in terms of raising protocol violations and interrupted transfers. (Gregor Maier) 1.6-dev.21 Wed Jan 19 17:36:02 PST 2011 - Added 4 new BiFs and a new record type for testing the entropy of strings. (Seth Hall) find_entropy(data: string): entropy_test_result This is a one shot function that accepts a string and returns the result of the entropy calculations. entropy_test_init(index: any): bool This and the next two functions are for calculating entropy piece-wise. It only needs an index which can be any type of variable. It needs to be something that uniquely identifies the data stream that is currently having it's entropy calculated. entropy_test_add(index: any, data: string): bool This function is used to add data into the entropy calculation. It takes the index used in the function above and the data that you are adding and returns true if everything seemed to work, false otherwise. entropy_test_finish(index: any): entropy_test_result Calling this function indicates that all of the desired data has been inserted into the entropy_test_add function and the entropy should be calculated. This function *must* be called in order to clean up an internal state tracking variable. If this is never called on an index, it will result in a memory leak. The entropy_test_result values have several measures of the entropy, but a good one to work with is the "entropy" attribute. It's a double and as the value approaches 8.0 it can be considered more and more random. For example, a value of 7.832 would be quite random but a value of 4.671 is not very random. 1.6-dev.20 Wed Jan 19 17:30:11 PST 2011 - BRO_DNS_FAKE is now listed in the --help output. (Seth Hall) 1.6-dev.18 Wed Jan 19 16:37:13 PST 2011 - Removing unnecessary expire timer from http_sessions. (Gregor Maier) 1.6-dev.16 Sat Jan 15 14:14:21 PST 2011 - Updates to the build system. (Jonathan Siwek) * ``make dist`` is now available to be used with the top-level Makefile for creating source packages according to #344. * ``make-rpm-packages`` and ``make-mac-packages`` scripts can now generate binary packages according to #295. * Additional configure options to change packaging behavior. * OS X builds will now prefer to link static libraries of optional dependencies that don't come with the vanilla operating system. * Fix for OS X 10.5 compile error dealing with the llabs() function from stdlib. * Installing as a different user than the one that configured/built now works (although, a harmless error message about not being able to write the install manifest may occur). 1.6-dev.3 Wed Dec 8 04:09:38 PST 2010 - Merge with Subversion repository as of r7137. Incorporated change: * Fix for packet processing resumption when a remote Bro dies during state synchronization (Robin Sommer). 1.6-dev.2 Wed Dec 8 03:57:03 PST 2010 - Compatibility fix for OpenSSL 1.0.0 (Christian Kreibich, Gregor Maier). 1.6-dev.1 Sat Nov 27 12:19:47 PST 2010 - Merge with Subversion repository as of r7098. Incorporated changes: * Rotation post-processors are now passed an additional argument indicating whether Bro is terminating (Robin Sommer). * Bro now consistently generates a file_opened event for all fopen() calls. (Robin Sommer). * You can now redefine the email_notice_to function (Robin Sommer). 1.6-dev.0 Fri Nov 26 13:48:11 PST 2010 - The Bro source code is now developed in the new git repositories. See the developer pages at http://www.bro-ids.org for more information on the new development process. - Bro's build and installation setup has been moved from GNU autotools to CMake. As a result of that, layout and specifics of the distribution has changed significantly. - Lots of pieces have been removed from the distribution that are either now unnecessary or are no longer maintained. - As part of the cleanup, a numbef of Bro configure options and their corresponding functionality have been removed, including: * --disable-select-loop * --with-dag * --disable-nbdns * --enable-activemapping * --enable-activemapping * --enable-shippedpcap - The previous configure option --enable-int64 is now enabled by default, and can no longer be disabled. - ClamAV support has been removed, which has been non-functional for a while already. -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1.5.3 Thu Mar 3 08:55:11 PST 2011 - Removing aux/broctl/policy/cluster-addrs.hot.bro from the distribution. The script is no longer needed and could in fact break an installation because it redefines an old variable that has went away. (Robin Sommer) - Smarter way to increase the communication module's pipe's socket buffer size, resulting in a value closer to the allowed maximum. (Craig Leres) - BroControl now also maintains links from the log archive to the current set of logs when running in standalone mode. (Robin Sommer) - Bug fix for a file descriptor leak in the remote communication module. (Scott Campbell) - Bug fix for BroControl to now activate trace-summary's sampling in cluster mode, but not anymore in standalone mode. (Robin Sommer) - Broccoli updates: * Accept empty strings ("") as values in the configuration file. (Craig Leres) * Support for specifying a separate host key for SSL-enabled operation, with documentation update. (Craig Leres) 1.5.2 Wed Jan 12 17:34:55 PST 2011 - Portability fixes for --enable-int64 (Vern Paxson). - Bug fix for Active Mapping support (Kevin Lo). - Broccoli compiler warning fixes (Kevin Lo). - Bug fixes for --enable-int64 and for avoiding bogus statistics / bad memory references when generating profiling information upon exit (Vern Paxson). - Bug fixes for terminating connections (Tyler Schoenke and Vern Paxson). - Removed now-quite-stale SSHv1 overflow detection, as it's more prone to false positives than useful detection (Vern Paxson). - The SWIG file now explicitly lists those pieces from broccoli.h which it wants to wrap, rather than just including all of broccoli.h (Robin Sommer). - http-header.bro now includes a global "include_header: set[string]" If it contains any strings, then only those headers will be processed. If left empty, then you continue to get the current behavior of processing all headers. (Robin Sommer). - Several changes to drop.bro (Robin Sommer): * If True, the new flag Drop::dont_drop_locals indicates that local hosts should never be dropped. On by default. * If True, the new flag Drop::debugging activates extensive debugging output for the catch-and-release logic. Off by default. * The timeout for tracking dropping information is now 1 day rather than 7 days, to better echo the one-restart-a-day semantics used in the past. * Bug fix for hosts once dropped by later cleared; some state for them persisted. - Portability fix for Broccoli Python bindings on 64-bit platforms (Robin Sommer). - The HTTP analyzer no longer attempts to track Server/User-Agent versions, as these are hugely voluminous (Seth Hall). - HTTP and SMTP no longer have extra-short inactivity timeouts, as these were too often leading to premature expiration of a connection (Robin Sommer). - Tracking of HTTP refer[r]er's by setting log_referrer. (Vern Paxson). - The "rst" tool (aux/rst/) now takes an optional "-I " argument that instructs it to inject as payload rather than sending a RST packet (Vern Paxson). must be NUL-terminated, and the NUL is not included. - Bug fix for crashes in the DNS analyzer when processing replies for which no request was seen (Robin Sommer). - Addressed a number of lint nits (Vern Paxson). - Rotation post-processors are now passed an additional argument indicating whether Bro is terminating (Robin Sommer). - Bro now consistently generates a file_opened event for all fopen() calls. (Robin Sommer). - The "cf" utility now ignores a leading "t=" prefix, for compatibility with Bro's "tagged" logging format (Robin Sommer). - You can now redefine the email_notice_to function (Robin Sommer). - Fix for packet processing resumption when a remote Bro dies during state synchronization (Robin Sommer). - OpenSSL/X509 portability fix, at long last (Gregor Maier & Christian Kreibich). - Fix for compatibility with newer versions of autoconf (Gregor Maier). - A larger BroControl update (Robin Sommer, if not marked otherwise): o Increasing default timeouts for scan detector significantly. o Increasing the manager's max_remote_events_processed to something large, as it would slow down the process too much otherwise and there's no other work to be interleaved with it anyway. o Adding debug output to cluster's part of catch-and-release (extends the debugging already present in policy/debug.bro) o Fixing typo in util.py. Closes #223. o Added note to README pointing to HTML version. o Disabling print_hook for proxies' remote.log. o broctl's capstats now reports a total as well, and stats.log tracks these totals. Closes #160. o Avoiding spurious "waiting for lock" messages in cron mode. Closes #206. o Bug fixes for installation on NFS. o Bug fix for top command on FreeBSD 8. o crash-diag now checks whether gdb is available. o trace-summary reports the sample factor in use in its output, and now also applies it to the top-local-networks output (not doing the latter was a bug). o Removed the default twice-a-day rotation for conn.log. The default rotation for conn.log now is now once every 24h, just like for all other logs with the exception of mail.log (which is still rotated twice a day, and thus the alarms are still mailed out twice a day). o Fixed the problem of logs sometimes being filed into the wrong directory (see the (now gone) FAQ entry in the README). o One can now customize the archive naming scheme. See the corresponding FAQ entry in the README. o Cleaned up, and extended, collection of cluster statistics. ${logdir}/stats now looks like this: drwxr-xr-x 4 bro wheel 59392 Apr 5 17:55 . drwxr-xr-x 96 bro wheel 2560 Apr 6 12:00 .. -rw-r--r-- 1 bro wheel 576 Apr 6 16:40 meta.dat drwxr-xr-x 2 bro wheel 2048 Apr 6 16:40 profiling -rw-r--r-- 1 bro wheel 771834825 Apr 6 16:40 stats.log drwxr-xr-x 2 bro wheel 2048 Apr 6 16:25 www stats.log accumulates cluster statistics collected every time "cron" is called. - profiling/ keeps the nodes' prof.logs. - www/ keeps a subset of stats.log in CSV format for easy plotting. - meta.dat contains meta information about the current cluster state (in particular which nodes we have, and when the last stats update was done). Note that there is no Web setup yet to actually visualize the data in www/. o BroControl now automatically maintains links inside today's log archive directory pointing to the current live version of the corresponding log file (if Bro is running). For example: smtp.log.11:52:18-current -> /usr/local/cluster/spool/manager/smtp.log o Alarms mailed out by BroControl now (1) have the notice msg in the subject; and (2) come with the full mail.log entry in the body. o Fixing broctl's top output. (Seth Hall). o Fixing broctl's df output in certain situations. o BroControl fix for dealing with large vsize values reported by "top" (Craig Leres). 1.5.1 Fri Dec 18 15:17:12 PST 2009 - Due to a Python configuration problem, the original 1.5 distribution did not include the BroControl component, which also introduced a portability problem for CentOS. These issues have now been fixed (Robin Sommer and Vern Paxson). 1.5 Wed Dec 16 21:28:47 PST 2009 - Bro now comes with a new framework, BroControl, for managing an operational Bro setup, including support for installation, configuration, and maintainance tasks such a log archival and mail notification. The framework transparently supports both traditional standalone setups as well as cluster installations in which multiple Bro boxes coordinate to analyze a high-volume network link. See aux/broctl/README for more information about BroControl. Note, BroControl supersedes the older BroLite system, which is no longer supported and has been deprecated for a while now. - Numerous adjustments to DPD = dynamic protocol detection (Robin Sommer): o The Analyzer::ProtocolViolation?() method can now be passed the offending data (which POP3, SMTP, and FTP now do). This information is added to the "reason" string passed to the script level. o SMTP now more accurately reports violations. o FTP stops processing when client & server successfully negotiate an AUTH scheme (leading to subsequent encryption). o Analyzer::ProtocolViolation() is virtual, and TCP_ApplicationAnalyzer() overrides it to not report violations for any partial connections, because very likely these arise just due to the analyzer getting confused. o TCP::IsPartial() returns true if any side did not start with a SYN packet (used to be just be for the originator). o The connection_state_remove handler in conn.bro now has a higher &priority so that other handlers for the same event can use determine_service() and see any changes it performs. o DynDisable:max_volume specifies a volume limit (default 10K). Once a connection exceeds this limit, further protocol limitations will neither raise ProtocolViolation notices nor cause the analyzer to be disabled. o The event engine no longer raises protocol_violation events for TCP connections which had gaps, as these have proven too unreliable. (Note that, ideally, the *analyzers* should avoid reporting protocol_violations when they can't reliably parse a connection anymore after a gap; but many don't.) - A set of new script functions provide support for incrementally computing MD5 checksums (Seth Hall). md5_hash_init(index: any): bool Initializes an incremental hashing instance. "index" is a value of arbitrary type, used to identify this particular instance (you can have multiple concurrent instances by using different index values). Returns T on success, F on failure (such as the index is already in use). md5_hash_update(index: any, data: string): bool For the given hashing instance, updates the hash based on the given data. Returns T on success, F on failure (such as the index has not been initialized). md5_hash_finish(index: any): string Returns the MD5-printable hash for the given index and terminates the instance, or the string "" if the index was not active. - Bro now supports a believed-to-be-robust mechanism for estimating the proportion of traffic that it failed to capture ("measurement drops"), which can arise due to overload in either Bro itself, the kernel's packet filter, or problems with the link tapping mechanism (Vern Paxson). The event engine can generate estimates for either live traffic or what was previously recorded in a trace file, though traces subject to some forms of selective omission (such as skipping over parts of a connection to reduce storage) can lead to erroneous values. The estimates are based on observing gaps in TCP data streams, and come in two forms: the rate at which such gaps appear, and the relative volume of data missing due to the gaps. (We've found however that the volume-based estimator is not robust due to occasional packets with incorrect sequence numbers, so this estimator is off by default.) The easy way to get the estimates is to load capture-loss.bro. By default, it generates a CaptureLossSummary notice upon Bro's exit, which can look like: 1130222759.344066 CaptureLossSummary estimated rate = 0.00089124 / 0.000970997 (events/bytes) If the estimated loss is none, however, it suppresses this notice, unless you redef CaptureLoss::summary_if_none to T. You can also get finer-grained access by defining a "gap_report" event handler and redef'ing gap_report_freq to a non-zero interval (such as "10 sec"). This event allows you to pinpoint regions in time that exhibit significant capture loss. See capture-loss.bro for an example of a handler for this event. Finally, these changes include a number of fixes to Bro's ack_above_hole/content_gap analysis, which is now significantly more robust. - GeoIP support now supports ASN lookups via the built-in function lookup_asn(a: addr): count (Scott Campbell and Seth Hall). - The GeoIP built-in's lookup_location() and lookup_asn() now support IPv6 (Seth Hall). Note, the current GeoIP distribution doesn't include any IPv6 databases, so for now these won't succeed, but the hooks are in place for when databases become available. - lookup_location() now falls back back to the country database if the city database isn't available (Seth Hall). - The new SuccessfulPasswordGuessing Notice is generated when a host has been seen attempting password guessing (currently only for FTP sessions) and then successfully logs in (Royal Chan). You can control the threshold for such reports in terms of how many attempts the host must have made by redef'ing the variable password_guessing_success_threshhold, which defaults to 20. - The new script http-detect-passwd.bro analyzes the Web items returned for fetches that appear to be accessing the passwd file (Akhil Dhar). It generates a PasswordFullFetch Notice if it appears that the item includes a full password file, and PasswordShadowFetch if it looks like a shadowed password file. - The new built-in system_env(cmd: string, env: table[string] of string) works like system(), but puts the table entries into the environment before invoking the command (Robin Sommer). Each in the table creates an environment variable of the form "BRO_ARG_", whose value is the corresponding table entry. - The new script function execute_with_notice(cmd: string, notice_info) executes "cmd" with an environment containing the fields of the notice_info, i.e., the information associated with a Notice (Robin Sommer). Per the new system_env() function above, the environment variables appear as "BRO_ARG_", where is the field tag as it appears in notice.log when you enable use_tagging. - The new built-in enable_raw_output(file) acts the same as the attribute &raw_output (Seth Hall). - The new built-in file_opened(f: file) event is generated any time Bro opens a script-level file (Justin Azoff). You can use this, for example, if you want to ensure that a given file has a prelude in it such as human-readable headers, even when the file is rotated. - The notice_info record has a new field aux: table[string] of string &optional which you can use for information specific to a given type of notice (Robin Sommer). Entries in $aux appear as "aux_" tags in notice.log. - Another new notice_info record field is the boolean do_alarm (default=T), which, if set to F, overides a notice action otherwise specifying to generate an alarm (Robin Sommer). In other words, if do_alarm is F, no alarm will be generated independent of the notice action. This is a work-around for the fact that we can't specify more than one action. In particular, we couldn't NOTICE_DROP but then *not* alarm, which we now can by returning NOTICE_DROP yet setting do_alarm to F. - The notice_info record field $dropped now appears in the tagged output format if true (Robin Sommer). - NOTICEs relating to scan detection now no longer include the connection that triggered the notice, as it really doesn't contain any useful information, given that the particular trigger simply depends on the detection algorithm and its parameters (Robin Sommer). However, we do explicitly set $p (port number) in the notice, and also $n with the number of attempts. - drop.bro now hardwires a Catch-and-Release redrop after seeing one connection from a previously-dropped-but-already-released host (Robin Sommer). - drop.bro now provides some new hooks (Robin Sommer): event address_dropped(a: addr) Generated when an address has been dropped. event address_restored(a: addr) Generated when connectivity to an address has been restored, such as using the Catch-and-Release mechanism. event address_cleared(a: addr) Generated when an address that was dropped in the past is no longer being monitored looking for new connections (as part of the Catch-and-Release mechanism). - The new built-in function hexdump(data_str: string) : string returns a hex dump representation of the given input data (Christian Kreibich). The dump renders 16 bytes per line, with hex on the left and ASCII (where printable) on the right. - Bro's notion of when a TCP connection begins now dastes to the first instance of an initial SYN packet seen, rather than the last (Gregor Maier). - The Time Machine script tm-contents.bro now generates event contents_saved: event(c: connection, orig_file: string, resp_file: string) when the content of a connection has been completely saved to disk (Robin Sommer). - The mime.bro script now exports the MIME header callback table, and also marks it as &redef'able so you can modify its entries (Matthias Vallentin). The mime_log file is also now exported. - A new signature file, policy/sigs/http-bots.sig, contains signatures to detect some of the current HTTP based controlled bot families (Seth Hall). - The signature engine's HTTP pattern matching has been fixed (Seth Hall) to align with the documentation at: http://www.bro-ids.org/wiki/index.php/Reference_Manual:_Signatures#Content_conditions In particular, the content condition "http" is now referred to as "http-request" (though "http" still works for backward compatibility), "http-request-header" and "http-reply-header" now provide access to headers seen in only one direction, and similarly for "http-request-body" and "http-reply-body". (This latter is still accessible as "http-body" for backwards compatibility.) - The new script variable max_remote_events_processed: count (default 10) sets a limit on the number of remote events processed in each round, before tending to other inputs (Robin Sommer). - If you set the new script variable dump_used_event_handlers to T, then on startup Bro dumps out all of the event handlers that the loaded set of scripts can invoke (Matthias Vallenti). - Summaries for DNS PTR scanning now use a separate Notice, DNS_PTR_Scan_Summary, rather than overloading DNS_PTR_Scan (Robin Sommer). - scan.bro now provides a table skip_dest_server_ports: set[addr, port] which lists servers (defined as an address and a port) excluded from scan detection computations (Craig Leres and Jay Krous). - When redefining values on the command line directly (using var=value), quotation marks are now implicit only if "var" is a variable of type string (Christian Kreibich). This allows other string-like values (such as enum's) to be passed as well. - scan.bro now explicitly loads conn.bro so that it can itself be loaded independently (Robin Sommer). - login.bro depends on scan.bro (because of tracking authentication "scans"), so now it explicitly loads it (Vern Paxson). - UDP_datagram_length_mismatch is now by default flagged just once per originating host rather than once per connection, as it can generate tons of messages (Vern Paxson). - Removed now-long-boring flagging of access to Solaris "listen" service as "hot" (Vern Paxson). - Removal of libedit, since libreadline provides similar functionality (Christian Kreibich). - Added scripts missing from distribution: dce.bro, ncp.bro, and smb.bro (Vern Paxson). - ssh.bro now exports ssh_ports (Seth Hall) - A number of improvements to inter-Bro communication (Robin Sommer). (1) Remote communication now no longer includes location information for serialized objects; that removes quite a bit of redundacy from the network traffic. (2) The new option 'remote_check_sync_consistency" disables the cross-check on the receiving side of &synchronized state of whether the current value of a variable has the value expected by the sender. Transmitting the original values in addition to the updates generates quite a bit CPU & network load in some cases (in particular, a table of tables). The default for remote_check_sync_consistency is off, and so far that in particular seems to reduce the proxy's load quite a bit. (3) Complete overhaul of the internal caching of serialized objects. The objective of the caching is avoid retransmitting already sent values over and over again. It turns out, however, that some objects are very stable and hardly change or get replaced (e.g., Bro types); while other change all the time and are hardly reused some time later (e.g., Vals). Now we maintain *two* caches independently for these types of objects; one with a low turn-over one and another with a high one. This should reduce CPU load on both sender and receiver sides. The new scheme is only used if both communicating Bros support it; with older Bros, as well as with Broccoli, we continue using the old scheme. - Some reworking of remote printing (Robin Sommer), as follows. Bro now uses a new interprocess message rather than print_hook events, to better manage buffering and associated load (these can produce failures depending on system configuration; see remote.log). A number of timeouts and buffer sizes have been tuned. Internally, EINTR errors are now treated separately from EAGAIN. Finally, even with remote_check_sync_consistency=F, one type of consistency check was still being done; this is no longer the case. - The DNS analyzer now generates events (dns_query_reply/dns_rejected) for replies with zero questions (Robin Sommer). - Perftools support for incompatible changes in the 1.0 API (Robin Sommer). - Rearranged (generally reducing, though not always) some state timeouts associated with scan detection (Robin Sommer). In addition, when a scanning address crosses ignore_scanners_threshold (meaning that it will be ignored from now on anyway), it gets discarded from all state-tracking tables. Finally, the ignore_scanners_threshold now applies all kinds of scans, not just address scans. - Substantial Broccoli updates, including a new initialization requirement that breaks backward compatibility, support for enqueueing serialized event data for transmission, and OpenSSL threadsafe initialization. See aux/broccoli/ChangeLog for details (Christian Kreibich, Robin Sommer, and Matthias Vallentin). - Broccoli hashtable optimisation. See aux/broccoli/ChangeLog for details (Christian Kreibich & Matthias Vallentin). - Broccoli memory leak fixed, see aux/broccoli/ChangeLog for details (Christian Kreibich). - Broccoli: updates to bropipe tool (Steve Chan and Robin Sommer). - Bug fixes for Broccoli Python bindings (Robin Sommer and Matthias Vallentin). - Fixed nasty bug due to module scoping that completely kept stepping-stone detection from working (Vern Paxson). - A serious bug in the packet sorter has been fixed (Robin Sommer). - Bug fix for extra NULs getting embedded in escaped strings (Seth Hall). - Bug fix for HTTP messages that use "Connection: close" rather than length headers, which yielded erroneous reassembled messages with \r\n's when only \n's were present (Bernhard Ager). - Fix for reporting on ICMP flows that are expired from the flow table (Vern Paxson). Previously there was a race condition if the flow was flushed prior to its summary timer expiring. - The -l option (list the scripts that Bro loads) now correctly prints scripts loaded by the prefix mechanism, and uses indentation to indicate the load hierarchy (Robin Sommer). - A bug has been fixed (really, worked around) in drop.bro that prevented dropped addresses from being properly restored (Robin Sommer). - Fixes for deadlocking problems in the Broccoli protocol. See aux/broccoli/ChangeLog for details (Christian Kreibich & Robin Sommer). - Bug fix for DNS analyzer on 64-bit machines (Gregor Maier). - Bug fix for asynchronous DNS lookups to prevent some successful lookups being reported as timed out (Robin Sommer). - Bug fix for tracking line numbers associated with compound statements (Po-Ching Lin). - Fix for a rare condition in which the main Bro process couldn't kill its child process (Robin Sommer). - Fix for file rotation when the underlying file is deleted before the timer expires (Robin Sommer). - Fix for potential crash when communication connections break down, and also for releasing cached objects (Robin Sommer). - Fix for default table entries computed by function invocation to not cache previous results (Robin Sommer). - Fix for Bro's internal DNS resolution (Scott Campbell and Robin Sommer). - Portability fix for DAG packet capture (Gregor Maier). - Portability fix for --enable-brov6 (Robin Sommer). - Portability fixes for FreeBSD (Vern Paxson). - A work around for new_packet() crashing on IPv6 packets (Vern Paxson). For now, IPv6 packets are skipped. Also, for fragments the event handler is now only called for the fully reassembled packet. - The new configuration option --disable-nbdns supports disabling non-blocking DNS at configure time (Sean McCreary). Note, there are some known problems with it in some environments. - A number of configuration fixes and enhancements (Christian Kreibich and Robin Sommer). - Consistency nit for the configuration process (Seth Hall). - A number of reference-counting and other memory management fixes (Robin Sommer). - Bug fix for inter-Bro communication lockup (Seth Hall and Robin Sommer). - Bug fix for computing TCP payload length in new_packet event (Lothar Braun). - Bug fix for sending boolean True values via Broccoli (Seth Hall). - make distcheck fix to clean up .bif.bro files (Christian Kreibich). - Bug fix for DPD's recognition of SSLv2 connections (Seth Hall). - Bug fix for &default for tables indexed by subnets (Seth Hall). - A bug has been fixed that could crash Bro when you called get_event_peer() after a remote connection had already disppeared (Robin Sommer). - Introduced a work-around for crashes that occur when Bro exits due to handling a signal (Robin Sommer). - Bug fix for checkpoint.bro - don't schedule timers for times that aren't actually in the future (Robin Sommer). - Hostname formatting fix for anon.bro (Fabian Schneider). - Bug fix for redundant .log extension in Time Machine log file (reported by CS Lee). - Removed now-outdated special-casing of Linux reporting of packet filter statistics (Peter Wurzinger and Robin Sommer). - A number of memory leaks fixed (Robin Sommer). - Addressed warnings from newer versions of g++ (Robin Sommer and Vern Paxson). - Fixed an invocation issue in the ca-create script that prevented it from working with recent OpenSSL versions (Craig Leres & Christian Kreibich). - Comment fixed in drop-adapt (Justin Azoff). - Duplicate code removed from Val (Seth Hall). 1.4 Fri Oct 17 11:08:52 PDT 2008 - We are no longer supporting a previous Bro release as the "stable" version. Rather, the model now is that the current public release will aim for increasing stability (occasionally updated with fixes), and those who wish to use a "bleeding-edge" snapshot can do so via access to the public SVN source code repository, as explained at http://bro-ids.org/wiki/index.php/Subversion#Public_Access Note that all previous releases remain available from the download page; what is changing is that we no longer commit to support for the most recent of these. - We have clarified the copyright statement that covers most of the code to remove the "advertising clause" that derived from older BSD licenses, and we have removed copyright wording from most source code files. See COPYING for the current wording and a list of files that retain their own copyright notices. - Bro now supports analyzing NetFlow v5 data, i.e., from Cisco routers (Bernhard Ager). NetFlow can be useful for intrusion detection as it allows analysis of traffic from many different points in the network. Bro can now read NetFlow data from a UDP socket, as well as (mostly for debugging purposes) from a file in a specialized format. You can create these files with the programs given in aux/nftools. Command line switches: -Y|--netflow :[=] | read flow from socket This is the usual way of getting NetFlow data into Bro by opening a UDP socket on : and reading all incoming packets. Setting the to 0.0.0.0 should work on most platforms. Optionally you may set an identifier for the source - useful if there are many different sources you want to analyze in parallel. This might also be necessary if you want to use this feature with a clustered Bro. Examples: bro -Y 0.0.0.0:5555 netflow bro -i eth0 -Y 10.0.0.1:1234=src1 brolite netflow -y|--flowfile [=] Used to read from a file. You can optionally include an identifier for the source. Examples: bro -y myflowfile netflow bro -y myflowfile=src1 otherflowfile=src2 netflow Netflow Events: event netflow_v5_header(h: nf_v5_header) Generated upon reading a new NetFlow PDU, as summarized in the argument. The field h_id gives the flow source identifier and a serial number. You can use this field to associate subsequent netflow_v5_record events with their header. event netflow_v5_record (r: nf_v5_record) Every record within a NFv5 PDU generates a corresponding netflow_v5_record() event. The relatively complex timestamp format of NFv5 is already converted to Bro's time type, and the TCP header flags are separated into bools. The distribution includes an example analysis script, netflow.bro. It simply dumps received NetFlow records. If netflow_restitch is T (the default), then Bro performs flow restitching as well, and two script variables become relevant: global netflow_finished_conn_expire = 310 sec &redef; specifies how long to wait for additional flow records after a RST or FIN for const netflow_table_expire = 31 min; Its setting only affects table declarations, and therefore cannot be usefully redef'd. Auxiliary programs: Bro uses a custom format for flow data stored in files, to enable preserving timestamps of the PDU arrivals and the exporter's IP address. The tools nfcollector and ftwire2bro in aux/nftools/ provide ways to manipulate the Bro NF file format. The first dumps NetFlow data from a UDP socket to stdout or to a file in Bro format. The second converts NetFlow data in "wire" format to Bro format, and, while doing so, fakes up the exporter's IP address and timestamp. You can get "wire" format from normal flow-tools files, e.g., by using 'flow-export -f 4'. Please note that the Bro format is just a hack to allow for easier debugging. Therefore the format is not in fact platform independent, and not suitable for data storage. - A new DHCP analyzer generates the following events (Po-Ching Lin): event dhcp_discover(c: connection, msg: dhcp_msg, req_addr: addr) event dhcp_offer(c: connection, msg: dhcp_msg, mask: addr, event dhcp_request(c: connection, msg: dhcp_msg, event dhcp_decline(c: connection, msg: dhcp_msg) event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, event dhcp_nak(c: connection, msg: dhcp_msg) event dhcp_release(c: connection, msg: dhcp_msg) event dhcp_inform(c: connection, msg: dhcp_msg) where dhcp_msg values look like: type dhcp_msg: record { op: count; # 1 = BOOTREQUEST, 2 = BOOTREPLY m_type: count; # the type of DHCP message xid: count; # transaction ID of a DHCP session h_addr: string; # hardware address of the client ciaddr: addr; # original IP address of the client yiaddr: addr; # IP address assigned to the client }; See dhcp.bro for the corresponding analysis script (which could probably use some refinements). Note, this analyzer is implemented using BinPAC, so you will need to specify --use-binpac to activate it. - A BitTorrent analyzer is now available (Nadi Sarrar). See the policy scripts bittorrent.bro and bt-tracker.bro for the events generated for analyzing transfers and tracker dialogs, respectively. - The "Bro Lite" configuration is now deprecated and will not in general be supported (Robin Sommer & Vern Paxson). - "make install" now only installs a core set of files (Robin Sommer). Policy files are now installed in /share/bro/* (or whatever configure determines $datadir to be), which is now in Bro's default search path. It creates a directory /share/bro/site for local policy files, and the default BROPATH is extended to include this. The default path no longer includes policy/local. You can install the additional files used by the (now deprecated) "Bro Lite" configuration using "make install-brolite". - Substantial updates to Broccoli, including support for container types (tables and sets) as well as a new metadata structure for event callbacks, facilitating truly generic event handler implementations (Christian Kreibich, Seth Hall and Robin Sommer). See aux/broccoli/ChangeLog for details. - Extensive changes to allow Bro to process packets captured in the past intermingled with those captured in real-time (Matthias Vallentin and Robin Sommer). This operation reflects combining Bro with use of "Time Machine" functionality for packet capture. - We have unfortunately had to disable support for configuring Bro to use ClamAV, since it turns out that the key interface we need for processing blocks of memory directly rather than whole files is no longer supported by the package, and in fact was buggy even when it was (Robin Sommer). - The new signature option "http-body //" matches on the body data of HTTP entities (Robin Sommer). The matching is done after decompressing the body, if necessary. - The new built-in function identify_data(data: string, return_mime: bool) analyzes the string "data" and returns its type according to libmagic, if installed (Seth Hall). The second argument controls whether it should be returned as a MIME-type or just an identifying string. For example, identify_data("MZpofigu", F) returns the string "MS-DOS executable", and print identify_data("MZpofigu", T) returns "application/x-dosexec". - The new analysis script http-identified-files.bro identifies the type of items returned by Web servers using libMagic (if available) and generates notices for interesting types and mismatches between URLs and types (Seth Hall). You configure it using two variables. watched_mime_types is a pattern (default /application\/x-dosexec/ | /application\/x-executable/ ) for which any MIME type matching the pattern generates a HTTP_WatchedMIMEType notice. mime_types_extensions is a table mapping strings to patterns specifying how URLs for the given MIME type should appear. (Ideally, this would be a table mapping patterns to patterns, but Bro doesn't currently support that.) It defaults to: ["application/x-dosexec"] = /\.([eE][xX][eE]|[dD][lL][lL])/ i.e., do Windows executables end in .exe or .dll. You can also redef the pattern ignored_urls to specify URLs that should not generate complaints. It defaults to matching Windows Update. - The new script http-extract-items.bro extracts the items from HTTP traffic into individual files (Vern Paxson). Files are named: .._._. where is a redef'able prefix (default: "http-item"), is a number uniquely identifying the item, the next four are describe the connection tuple, and is "orig" if the item was transferred from the originator to the responder, "resp" otherwise. - The workings of how Bro interfaces to external programs for dropping/ restoring connectivity of misbehaving hosts has been significantly reworked (Brian Tierney and Robin Sommer). First, dropping decisions used to be made directly by analyzer scripts, such as scan.bro directly calling drop_address(). Now instead the scripts generate Notices and then the notice policy can have an action of NOTICE_DROP to codify that the response to the given Notice is to drop the source. The new notice_action_filter of drop_source drops the source of notices, and drop_source_and_terminate both drops the source and terminates the corresponding connection. So, to drop all sources triggering a specific notice, one can now, e.g., write: redef notice_action_filters += { [Hot::SSH_Overflow] = drop_source }; Related to this change, notice_info has a new field $dropped, set to true if the Notice triggered a (successful) drop. Second, by redef'ing Drop::use_catch_release to T (default F) you can activate "catch-and-release" logic. You use this mode when you need to manage a limited number of possible blocks, or to build in automatic "forgiveness" in situations where blocked sources might become benign (such as due to dynamic IP addresses). If a source has been idle for Drop::drop_time, then it is unblocked. However, if it is again seen as block-worthy, then it is blocked for an interval of Drop::long_drop_time. Third, ICMP scanning is now reported by its own notice, ICMPAddressScan, rather than Scan::AddressScan. - Google's perftools have replaced mpatrol for leak-checking and heap-profiling (Robin Sommer). If Bro is compiled with --enable-perftools and configure finds the perftools, there are two command-line options available: -m turns on leak checking of the main packet loop, with some uninteresting leaks are suppressed. Currently, with one exception (the RPC analyzer; problem not yet found), it reports no leaks when running the test suite. -M turns on heap profiling: Bro will take a snapshot of the heap before starting the main packet loop and another one when finished. These snapshots can then be analyzed with pprof. For more information about the perftools see http://code.google.com/p/google-perftools - Notice tags are now generated in a pseudo-unique fashion that, with high probability, ensures that tags generated by separate Bro processes don't clash when logged to a common location, such as for a Bro cluster (Robin Sommer). Tags are now string's rather than count's, and are associated with all notices, not just that are connection-related. You can however redef the string notice_tag_prefix or the function new_notice_tag to further control how such tags are generated. - Four new built-ins for type conversion (Robin Sommer): function double_to_interval(d: double): interval function addr_to_count(a: addr): count function port_to_count(p: port): count function count_to_port(c: count, t: transport_proto): port - Many policy scripts have been modified to use modules & scoping (Robin Sommer and Matthias Vallentin), which may require updates to existing scripts/refinements. - The new script variable dpd_conn_logs (default F), if true, changes the semantics of the service field in connection logs written to conn.log, as follows (Robin Sommer). It becomes a comma-separated list of analyzers confirmed by DPD to parse the connection's payload. If no analyzer could confirm its protocol, but the connection uses a well-known port, the service is the name of the port with "?" appended (e.g., "http?"), as long as the corresponding analyzer has not declined the connection. In addition, ftp-data sessions are labeled "ftp-data" and portmapper connections are labeled with the specific method-call (just as before). dpd_conn_logs defaults to F because the change in semantics may break scripts that parse conn.logs; but it will likely change to the default in the future. With dpd_conn_logs turned off, conn logs are generated as they used to be, with a few rare exceptions (with previous versions, the service field was sometimes determined while the connection was still alive; now it's always determined at the time when the conn.log entry is written out). - The SSL analyzer has been rewritten using BinPAC, with a number of robustness improvements (Tobias Kiesling). It currently is only used if you execute with --use-binpac. - Python bindings for Broccoli are now available in aux/broccoli/bindings/python/ (Robin Sommer). See README/README.html in that director for details. - The new "auth" option in remote.bro indicates whether a given side is considered "authoritative" for shared state, in which case it sends its initial state to &sync'ed peers (Robin Sommer). When two peers synchronize their state, one side sends its current set of state to the other as soon as the remote connection is established. The one sending the state used to be the one who has been running longer; now it can also be explicitly set via the "auth" flag in the Remote::Destination. - Two new tuning parameters for scan.bro (Robin Sommer): ignore_scanners_threshold (default 0): If a host has scanned more than this many hosts, it is completely excluded from further scan detection. 0 disables. addr_scan_trigger (default 0): A host is only tracked for address scanning once it has contacted this many different hosts. Primarily intended for using a two-stage scan detection with a Bro cluster: first, each node searches locally for scanners by looking for hosts contacting more than addr_scan_trigger destinations. Those hosts which do are then globally tracked throughout the cluster by &synchronizing the scan detector tables. - When Bro serializes functions, it now does so by default using only their name, rather than their full value (Robin Sommer). This prevents propagation of expiration functions associated with tables and sets. Note, currently there is no mechanism provided to switch from the default behavior, but the internal hooks are in place to do so. - The new built-in variable trace_output_file gives the name of the -w output trace file (Robin Sommer). - Bro no longer installs new file rotation timers when shutting down (Robin Sommer). - The new policy scripts remote-print-id{,-reply}.bro support convenient access to printing the identifiers of a remote Bro (Robin Sommer). You use the script remote-print-id.bro to request and receive the printing; the remote Bro must have loaded remote-print-id-reply.bro in order to process the request. Example use: bro -e 'redef PrintID::dst="" PrintID::id=""' remote-print-id - scan.bro has been heavily modified to better support distributed scan analysis (Matthias Vallentin and Robin Sommer). - The check for unused event handlers is now turned off by default (Robin Sommer). To enable, use "redef check_for_unused_event_handlers = T". - The new script drop.bro has been split off from scan.bro to isolate the logic concerning dropping addresses to block scans (Robin Sommer). - The new -l flag lists each script as it is loaded (Robin Sommer). - Textual descriptions of identifiers now include their attributes (Robin Sommer). - The new predefined function prefixed_id() returns a session identifier with its peer-ID prepended if it's associated with a remote Bro (Robin Sommer). This is now used when generating writing log files. - remote.bro now assigns a priority of -10 to its bro_init() event handler to allow others a chance to modify destinations (Robin Sommer). - A large number of BinPAC updates (Ruoming Pang and Robin Sommer). - The new built-in type_name(v): string returns the name of the type of the value v (Vern Paxson). For example, "typename(5.2)" returns "double". This function is mainly for internal debugging (i.e., finding mismatches between values generated by the event engine versus how their type is expected by the script layer). - The new built-in str_shell_escape() does some basic escaping on strings that will be passed to system() (Christian Kreibich). Note, this function isn't ready (robust enough) for routine use, however. - The new built-in disable_print_hook(file) acts the same as the attribute &disable_print_hook (Robin Sommer). - The new script terminate-connection.bro factors out the terminate_connection() functionality that used to be in conn.bro (Robin Sommer). - The new attribute &group= can be associated with event handlers to group them together into a set that can be manipulated as a whole (Robin Sommer). is a string reflecting the name given to the group. The built-in enable_event_group(group: string) turns on all the analyzers in a given group, and disable_event_group(group: string) deactivates them. - The new attribute &raw_output applies to variables of type file, disabling escaping of non-printable characters (Seth Hall). - You can now iterate over the characters in a string value using a "for" loop, e.g., "for ( c in str ) ..." (Robin Sommer). - The new built-in function cat_sep%(sep: string, def: string, ...%): string works similarly to cat(), except that it (a) separates the values by "sep" and (b) substitutes "def" for empty strings (Seth Hall). - The function string_escape() now takes a string of characters to escape rather than a single character (Robin Sommer). Each character in the string is preceded by '\' in the return value (also any embedded '\'s, as before). - The new built-in function global_ids() returns a table of all global identifiers along with associated information (Robin Sommer). The return value has type table[string] of script_id, indexed by the name of the identifier and yielding records with the following fields: type script_id: record { type_name: string; exported: bool; constant: bool; enum_constant: bool; redefinable: bool; value: any &optional; }; - The new script function find_last(str: string, re: pattern) returns the last occurrence of the given pattern in the given string, or an empty string if no match (Robin Sommer). Note that this function returns the match that starts at the largest index in the string, which is not necessarily the longest match. For example, a pattern of /.*/ will return just the final character in the string. - The new script variable record_all_packets, if redef'd to T (default F), instructs Bro to record every packet it processes (Robin Sommer). Prior to introducing this variable, Bro applied a few heuristics to reduce recording volume. Setting this variable also causes packets to be recorded very early in processing, which can be helpful for debugging crashes. - If the new script flag ssl_log_ciphers is set to T (default), ssl.bro logs the ciphers seen (Robin Sommer). - Much more expanded Time Machine support, now located in policy/time-machine/ (Robin Sommer), - The new command line option --status-file (alias -U) specifies the name of a file into which Bro will write an indicator of its current processing status (Robin Sommer). Possible values include "INITIALIZING", "RUNNING", "TERMINATING", "TERMINATED". - The new policy script targeted-scan.bro looks for repeated access from the same source to the same server, to detect things like SSH password-guessing attacks (Jim Mellander). - The "alternative" style for printing strings (i.e., a fmt() argument of "%As") now renders the raw string, other than escape-expanding embedded NULs (Vern Paxson). This change may be temporary, pending development of more fine-grained control over string rendering. - For now we have removed the %S functionality for fmt() (Robin Sommer). %S was meant to print "raw" strings, but later processing of such printing still introduces artifacts. - GeoIP information now includes latitude and longitude (Seth Hall). - ssh.bro now supports the variable skip_processing_after_handshake which directs the event engine to omit any further processing of an SSH connection after its initial handshake (Seth Hall and Robin Sommer). This can help with performance for large file transfers but precludes some kinds of analyses (e.g., tracking connection size). This change also adds a scope of "SSH". - Email notification of notices now allows for separate destinations depending on notice type (in particular, a regular mail destination versus a pager destination), and also escapes the notice to prevent injection attacks (Seth Hall and Robin Sommer). - The new policy script conn-flood.bro is a simple connection-flooding detector, mainly meant as a demonstration (Robin Sommer). - A large number of additions to the TLS/SSL known-ciphers suite (Seth Hall). - Serialization now uses 64-bit IDs to cache items rather than 32-bit, for robustness during long-running execution (Robin Sommer). - The new script variable tcp_max_initial_window specifies, for flows for which ACKs have never been seen, the maximum volume of initial data after which Bro will assume that it is seeing only one side of the connection and will not buffer data for consistency checking awaiting the later arrival of ACKs (Robin Sommer). It defaults to 4 KB. (Note, this used to be an internal value, so the behavior is not new.) Set to 0 to turn off this functionality and have Bro attempt to track all such flows. - The new script variable tcp_max_above_hole_without_any_acks specifies, for flows for which ACKs have never been seen, the maximum volume of data above a sequence hole that Bro will tolerate for a connection before giving up on tracking the flow (Robin Sommer). It defaults to 4 KB. (Note, this differs from tcp_max_initial_window in that this threshold applies to sequence holes rather than the beginning of flows. Like tcp_max_initial_window this used to be an internal value.) Set to 0 to turn off this functionality. - The new script variable tcp_excessive_data_without_further_acks specifies a threshold similar to tcp_max_above_hole_without_any_acks, but for flows for which Bro has seen ACKs (Robin Sommer). It defaults to 10 MB. Set to 0 to turn off the functionality. - Equal signs ("=") in text for notices are now escaped when using the tagged format to keep them unambiguous from the "=" delimiters (Robin Sommer). - The final tallies for notices are now processed as NoticeTally NOTICE's rather than directly alarm'd (Robin Sommer). - WeirdActivity notices now include an associated connection when appropriate (Robin Sommer). - Support for large (> 2^32 bytes) pcap trace files (Po-Ching Lin). - Scoped names ("...::...") are now allowed in signature "eval" constructs (Christian Kreibich). - scan.bro is now decoupled from conn.bro, i.e., you can @load the latter without getting the former (Vern Paxson). As part of this change, the logic to invoke TRW is now in scan.bro. - weird.bro has been updated with a number of missing Weird's (Vern Paxson). - If when using inter-Bro communication the child Bro process terminates, it now also terminates the parent process (Robin Sommer). - BinPAC analyzers now interoperate with DPD (Robin Sommer). - Some http.bro processing options are now exported so they can be accessed in other scripts (Robin Sommer). - SMTP analysis now applies to port 587/tcp as well as 25/tcp (Robin Sommer). - $conn is now set in ServerFound notices (Robin Sommer). - You can now create empty sets and tables using set() and table(), i.e., the usual set/table constructors with no arguments (Vern Paxson). By themselves, these have an unspecified type - you can't use them directly other than to assign them. For example, local bad_guys: set[addr]; ... bad_guys = set(); # start over assuming no bad guys - A number of scripts have been (slightly) simplified to use the new empty set()/table() constructors (Vern Paxson). Note that these still aren't usable for field assignments in record constructors, nor for attributes like &default = ... - Removed unused syntax for declaring sets based on a list of initial values (Vern Paxson). - set() and table() can now be used as arguments to function calls (Vern Paxson). - The vestigial &match attribute has been removed. - POP3 is now recognized using Dynamic Protocol Detection (Seth Hall). - The new event expected_connection_seen(c: connection, a: AnalyzerTag) is generated whenever a connection is seen for which we have previously scheduled an analyzer via expect_connection() (Robin Sommer). - The new built-in capture_state_updates logs all changes applied to &synchronized variables, in a fashion similar to the capture_events() built-in (Robin Sommer). An accompanying policy script, capture-state-updates.bro, turns this on to the file state-updates.bst. - If the new script variable suppress_local_output is set (default: F), Bro suppresses printing to local files if there's a receiver for print_hook events (Robin Sommer). This option is however ignored for files with a &disable_print_hook attribute. - The new notice action filter function file_if_remote specifies that notices from sent from remote source addresses should have an action NOTICE_FILE (Robin Sommer). - The new notice action filter function file_local_bro_notices specifies that notices generated by the local Bro instance (as opposed to a remote peer) should have an action NOTICE_FILE (Robin Sommer). - An arbitrary tag can now be past to post-processors for log rotation (Robin Sommer). - Default inactivity timeouts for interactive services shortened to 1 hour (Robin Sommer). - The scanning variables distinct_{peers,ports,low_ports} are now redef'able (Robin Sommer). - The new -S (--summary-only) option for site-report.pl directs to only generate connection summaries (Brian Tierney) - More useful default config file for edit-brorule.pl (Brian Tierney). - Bro now includes a test suite in testing/istate/ for its "independent state" functionality (Robin Sommer). - Support for parallel builds via make -j (Christian Kreibich). - Bro's default search path now includes includes policy/sigs/ and policy/time-machine/ (Robin Sommer). - Bro's internal processing of interprocess communication has been significantly overhauled to prevent potentially fatal race conditions (Robin Sommer). - Bro now checks calls to fmt() at compile-time to ensure that the correct number of arguments are present (Vern Paxson). This is useful in addition to Bro's run-time checking for arguments matching their corresponding format-specifiers in the case of rarely-executed statements that might not generate such run-time checks in routine testing. - The ports associated with Telnet and Rlogin are now redef'able (Robin Sommer). - MIME processing now removes leading whitespace from MIME headers (Sanmeet Bhatia and Robin Sommer). - TCP "weird" events reported by the connection compressor now match (other than a few rare corner-cases) those produced for normal TCP processing (rmkml and Robin Sommer). - Added Scan::suppress_UDP_scan_checks to control false positives on scan detection in environments with P2P protocols that use UDP (Vern Paxson). - The internal analyzer interface now includes an EndOfData() method that analyzers can use to report that all of a message has been delivered (Robin Sommer). - Fix for a significant memory leak in processing UDP when using -w (Robin Sommer). Note: this change turns off by default trace rewriting for generic UDP traffic. - Two serious regular expression bugs fixed (Vern Paxson). In the first, searching for a regular expression inside a string would fail if the pattern occurred only after an embedded newline. In the second, insufficient buffer was allocated when compiling regular expressions, leading to memory corruption. - Base64 decoding bug fixes (Christian Kreibich and Ruoming Pang). - Automatic rotation of files is now disabled for contents files written by the TCP reassembler, which otherwise leads to mangled files (Robin Sommer). - Bro now ships with an updated version of libpcap (0.9.8), which hopefully fixes problems managing trace files > 4 GB in size. - Significant bug fixes for gzip- and deflate-encoded Web items (Robin Sommer). - Bug fix for secondary-filter.bro (Vern Paxson). - Removed a naming ambiguity regarding TCP states (Vern Paxson). - Bug fix for signature scanner not matching all of its input (Vern Paxson). - Bug fix for using port values in signatures (Robin Sommer). - Minor policy script tweaks: state management for weird's, processing of Notice tags associated with connections, and dependencies for irc-bot.bro (Robin Sommer). - aux/ portability fixes (Vern Paxson). - Workarounds added for a BinPAC deficiency, which is that code in %cleanup clauses can also be executed during recovery from exceptions when parsing new data. This means that any delete's or Unref()'s need to also set the corresponding pointer to nil (Vern Paxson). - Bug fix for crashes with the non-BinPAC SSL analyzer (Robin Sommer). - Tweak to peer-status.bro since Bro now requires events to be declared prior to reference in a "schedule" statement (Robin Sommer). - The signature keyword "enable" now optionally accepts the syntax "foo:bar" to specify "activate analyzer bar as a child of analyzer foo" (Robin Sommer). This is used for example for an XML-over-HTTP analyzer that's in the works. - irc-bot-syslog.bro now uses open_log_file() for its log file (including the logging suffix) rather than a direct open (Vern Paxson). - Bug fix for tracking Blaster across a Bro Cluster (Robin Sommer). - Bug fix for the HTTP BinPAC analyzer chopping the trailing character off of HTTP headers when generating the http_all_headers event (Gregor Maier). - Bug fix for HTTP chunked items for which the chunk size line was terminated by CRLF but the CR and LF came in separate packets (Gregor Maier). - A bug has been fixed that would cause partial lines (for line-oriented protocols) to fail to be processed when a connection terminated (Robin Sommer). - Bro no longer treats a signal arriving before a previous signal has been processed as fatal, nor does it attempt processing of a termination signal if seemingly there are no race conditions to worry about (Robin Sommer). Both of these changes are an attempt to improve Bro's robustness. - Fix for attributes such as &encrypt not working in initial declarations but only in later redef's (Seth Hall and Robin Sommer). - Fixes for memory leaks in SSL processing (Seth Hall and Robin Sommer). - Fix for POP3 analyzer to not treat lines like "." as message terminators (Robin Sommer). - Bug fix for crashes arising from nil pointers in list expressions (Seth Hall and Robin Sommer). - Bug fix: a signature's "enable" would activate the corresponding analyzer even if no event handlers were defined for it (Robin Sommer). - Bug fixes to prevent crashes when mixing set_contents_file() with subsequent explicit close(), and to ensure all data written to file upon connection tear-down (Gert Doering and Robin Sommer). - Configuration support for MacPorts and Fink package management systems (Christian Kreibich & Vern Paxson). - Communication-only Bro's now send out email alarms (Robin Sommer). - Writes to a file that fail due are now run-time errors rather than fatal internal errors, since often these occur due to the disk being full (Robin Sommer). - Byte-order bug fix for lookup_location() (Robin Sommer). - BinPAC portability fix for 64-bit machines (Bernhard Ager and Robin Sommer). - Portability fixes for newer versions of gcc (Jan Gerrit Goebel and Robin Sommer). - Some support for porting to Solaris (Stephan Toggweiler). - Connection compressor bug fix for source and destination having the same IP address, such as when monitoring loopback (Robin Sommer). - Connection compressor bug fix for connections with multiple SYNs (Robin Sommer). - Bug fix for using already-declared local variables for looping over vectors in a "for" loop (Robin Sommer & Vern Paxson). - Bug fix for not processing truncated UDP packets (Tom Kho and Robin Sommer). - Bounds-check added to BinPAC-generated code (Tom Kho and Robin Sommer). - Bug fix for checking whether an IPv6 address is part of a subnet (Seth Hall). - Bug fixes for crashes relating to asynchronous DNS lookups performed at start-up (Robin Sommer). These changes also lowered the timeout before assuming failure from 20 seconds down to 5 seconds. - Portability and const-ness fixes (Kevin Lo and Robin Sommer). - Suppression of some content-gap complaints when running on traces that have been filtered down to only TCP control packets (Robin Sommer). - Removed unnecessary dependency in notice-action-filters.bro that led to errors when loading icmp.bro by itself (Vern Paxson). - Bug fix for potential infinite loop in client communiation (Robin Sommer). - Bug fix in reference counting that could eventually lead to roll-over (Robin Sommer). - Bug fix in communication initialization (Robin Sommer). - Internal documentation fix: timers are specified using absolute time, not relative (Robin Sommer). - Performance improvement for built-in find_all() function when running on large strings (Robin Sommer). - Memory leak fixes (Robin Sommer, Bernhard Ager, Christian Kreibich). - Bug fix for error recovery when encountering an unknown link layer (Bernhard Ager). - Bug fix for reversing client & server in a connection (Po-Ching Lin). - Bug fix for packet_contents when capture length exceeds the IP payload length due to Ethernet frame padding (Christian Kreibich). - Bug fix for tcp_packet event erroneously including Ethernet padding in its contents (Vern Paxson). - Bug fix for lookup_connection built-in (Seth Hall). - Portability nit for libedit tarball (Vern Paxson). - Broccoli portability fix for NetBSD (Christoph Leuzinger). - Type-checking for script-level event invocation was completedly broken - now fixed (Vern Paxson). - Portability fixes for different versions of g++/STL (Nicholas Weaver and Vern Paxson). - Fix for dynamic detection of SSL via DPD (Robin Sommer). - IPv6 portability fix for BinPAC-based DNS analyzer (Vern Paxson). Note, more portability work is needed for it. - Bug fix for bifcl error messages (Vern Paxson). - Minor bug fix for remote communication, plus some improved communication logging (Robin Sommer). - Bug fix for &printhook (Robin Sommer). - Bug fix for error message output (Robin Sommer). - Bug fix for termination cleanup (Robin Sommer). - Bug fix for some Rlogin corner cases (Robin Sommer & Vern Paxson). - Bug fix for bifcl generation of "interval" types (Vern Paxson). - Bug fix for getting connection memory statistics when Bro is exiting (Robin Sommer). - Config fix: --enable-debug now turns off -O2 for gcc (Robin Sommer). - Bug fixes for "heavy" analysis (Vern Paxson). - Broccoli bug fixes for types net and port (Robin Sommer). - Bug fixes for Telnet environment options (Robin Sommer). - Bug fix for accessing remote peer description (Robin Sommer). - A fix for the connection compressor generating new_connection too late (Robin Sommer). - Fixes for DAG support, including configuration and multiple interfaces (Robin Sommer). - Bug fix for serializing time-stamps of table entries (Robin Sommer). - Bug fix for dealing with peer IDs for remote communication (Robin Sommer). - Bug fix to avoid installing timers when timers have already been canceled (Robin Sommer). - Bug fix for interplay between serializing connections and connection compressor (Robin Sommer). - Memory leak fix for enum's (Robin Sommer). - Bug fix for files being closed prior to bro_done() (Vern Paxson). - aux/broccoli/contrib was not included in distribution (Robin Sommer). - Auto-configuration bug fix for BinPAC (Craig Leres). - Bug fix for dynamic protocol detection (Robin Sommer). - A number of configuration fixes for installation and portability (Christian Kreibich, Brian Tierney, Robin Sommer, Dan Kopecek). 1.3 Mon Jul 16 22:11:00 PDT 2007 - The Bro manual has been wikified at: http://www.bro-ids.org/wiki/index.php/User_Manual and this is the format in which it will evolve in the future (Christian Kreibich). - Much more extensive support for SMB, NetBIOS and NCP (Chris Grier). - The new attribute &priority=n defines the order of execution for handlers of the same event (Robin Sommer). Handlers with higher priority are executed first. n is an integer expression that must evaluate to a constant when the script is loaded. Example: > cat foo.bro event bro_init() &priority = -5 { print -5; } event bro_init() &priority = 5 { print 5; } event bro_init() { print 0; } # default priority=0 > ./bro foo.bro 5 0 -5 The connection_state_remove() handler in conn.bro now has priority -10 and therefore executes after all other handlers for this event. This fixes a long-standing problem of sometimes $addl fields not showing up in connection summaries. - The new expressions record(...), table(...), set(...) and vector(...) are constructors for the corresponding aggregate types (Vern Paxson). For example, record($foo = "hi", $bar = -6) is the same as the existing constructor [$foo = "hi", $bar = -6] For tables, sets, and vectors, the "..." values within the ()'s have the same syntax as those that you can list in variable initializations. For example, table([1, T] = "black", [4, F] = "red") returns a table of type "table[count, bool] of string". set(4, 3, -1) is a value of type "set[int]". - You can associate attributes with table() and set() constructors (Robin Sommer). For example: local s = set(1.2.3.4) &read_expire = 5 secs; associates a 5-second read expiration with the set assigned to s. - Bro now explicitly supports port numbers reflecting a transport protocol type of "unknown" (Christian Kreibich). Currently, this means "not TCP, UDP or ICMP". The numerical value of such a port is the IP protocol, so ranges from 0..255. For example: global p: port = 0/unknown; print fmt("%s", p); print fmt("p is TCP? %s", get_port_transport_proto(p) == tcp); print fmt("p is unknown? %s", get_port_transport_proto(p) == unknown_transport); yields 0/unknown p is TCP? F p is unknown? T In comparisons of different protocol types, the following holds: unknown < TCP < UDP < ICMP. - If your system supports "GeoIP" (see http://www.maxmind.com/app/geolitecity for a corresponding city database), then the new script function lookup_location(a: addr): geo_location returns a record of geographic information associated with an address (Seth Hall). The geo_location record has $country_code, $region and $city fields. If no information is available, each of these will be set to empty strings. If Bro hasn't been configured with GeoIP support, or if the address is IPv6 that cannot be directly converted to IPv4, then Bro produces a run-time error and likewise returns empty strings. - Signature-matching on HTTP components now processes the URI with escape sequences expanded (Robin Sommer). Ideally, there would be two signature keywords, one for decoded URIs (corresponding to this case) and one that allows matching against the URI as originally transmitted. - The connection compressor is no longer considered experimental, and is used by default (Robin Sommer). - The new function lookup_hostname(host: string): addr_set asychronously looks up the IPv4 address(es) of the given host via DNS (Robin Sommer). Like lookup_addr(), this function can only be used within a "when" statement. - The new built-in raw_bytes_to_v4_addr(s: string): addr takes a string that points to at least 4 bytes, and returns an address corresponding to interpreting these as being an IPv4 address in network order (Vern Paxson; suggested by Mike Dopheide). - Trace-rewriting support for DNS, SMB (Chris Grier). - The new script function find_all(str: string, re: pattern): string_set returns a string_set giving all occurrences of the pattern "re" in the string "str" (Robin Sommer). (Note that string_set's are unordered.) - The new policy script save-peer-status.bro generates a log to peer_status.$BRO_LOG_SUFFIX of updates received from communication peers (Robin Sommer). - The policy script print-filter.bro now includes two (scoped) variables, terminate_bro and to_file, which control whether to exit after printing the filter (default T) and whether to write to the log file pcap_filter.$BRO_LOG_SUFFIX or (default) to stdout (Robin Sommer). - The new script variable check_for_unused_event_handlers controls whether Bro checks for unused event handlers (Robin Sommer). It defaults to T, which was the past behavior (always report). - Bro now terminates if the only pending activity is future timers (Robin Sommer). It used to wait for those timers to expire, but this can cause fundamental problems if the timers are associated with table management (since these might never completely drain). - Tables and sets inside of records are now initialized to empty values rather than uninitialized (Vern Paxson). - A new variable allow_services_from (in hot.bro) complements the existing allow_service_to variable (Brian Tierney). It specifies that access to the given service from the given originator is allowed. - global_sizes() no longer reports internal variables (Robin Sommer). - The IRC analyzer is now activated if any of the (many) IRC event handlers are defined (Robin Sommer). - The default value for tcp_close_delay is now 5 sec rather than 0 sec (Robin Sommer). This prevents some spurious connection events. - Improved logic for dealing with "reversed" connections such as backscatter (Vern Paxson). - You can now left-justify fields when using fmt() with "%-" like in sprintf (Christian Kreibich). - Updates to DNS query types (Larry Leviton). - Added mechanism to http-header.bro to skip printing some HTTP headers (Larry Leviton). - The IrcHotWord notice now sets the associated connection (Robin Sommer). - If a notice has a tag, it's no longer overridden (Robin Sommer). - ServerFound notices now set the port field (Robin Sommer). - The built-in lookup_ID() now returns the string "" if the ID does not exist, rather than a run-time error (Robin Sommer). - The new tuning option ProtocolDetector::suppress_servers specifies a set of analyzers for which Bro generates ServerFound notices, but not ProtocolFound (Robin Sommer). This both reduces log file size and conserves memory. - A new notice_action_filter, tally_notice_type_and_ignore, works the same as tally_notice_type but returns IGNORE (Robin Sommer) - Setting summary_interval == 0 disables the creation of irc-bots.summary.log (Robin Sommer). - If you @load foo and a directory "foo" is in your path, Bro no longer tries to load it (Robin Sommer). - A number of BinPAC fixes and enhancements (Ruoming Pang, Chris Grier and Vern Paxson). - BinPAC now resides in aux/binpac rather than src/binpac (Ruoming Pang and Christian Kreibich). This reflects a decoupling of it from Bro so that it can be used to generate protocol analyzers for other projects too. - Removed example Inktomi entries from skip_scan_sources initialization, since they no longer exist (Vern Paxson). - The variable make notice_once_per_orig_tally_interval is now redef'able (Brian Tierney). - SIGPROF to the communication child process now logs resource stats to remote.log (Matthias Vallentin). - The new built-in getpid(): count returns Bro's process ID (Robin Sommer). - Patterns for detecting IRC-based bots updated (Robin Sommer). - irc-bot-syslog now logs just bots, not all IRC client/servers (Robin Sommer). - The new variable suppress_notice_actions in notice.bro suppresses notice_actions events for selected notice types (Robin Sommer). - Files opened during operation now rotate just like those opened at startup (Robin Sommer). - ResourceStats now also logs elapsed time and the reported number of packets-on-the-link (Mark Dedlow). - Printing a "file" value now produces its name (Robin Sommer). - Removed deliberate truncation of payload in port 80 FIN packets (Vern Paxson). - remote.log now includes received peer_descriptions (Robin Sommer). - Significant POP3 analyzer speed-ups (Vern Paxson). - Updated README (Vern Paxson). - Fix for "@load a" followed by "@load a.bro" not loading the same file twice (Robin Sommer). - Bug fixes for propagating state operations to uninitialized variables and for spurious state inconsistency messags (Robin Sommer). - Bug fix for sending final sync-points during pseudo-realtime mode (Robin Sommer). - Fix for possible buffer overflow (Christian Kreibich). - Bug fix for spurious end-of-file's during inter-Bro communication (Robin Sommer). - Bug fix for dpd_match_only_beginning=F (Robin Sommer). - Bug fix for updating timestamps (Christian Kreibich). - Bug fix for skipping ADU processing in adu.bro (Christian Kreibich and Zhichun Li). - Fix for ICMPs that carry ICMP headers (or non-TCP/UDP/ICMP headers) within them (Vern Paxson). - Fix for files being rotated after the timer queue has been deleted (Vern Paxson). - Bug fix for signature-matching with IPv6 subnets (Vern Paxson). - Bug fix for connection compressor setting connection origin (Robin Sommer). - Bug fix for interconn.bro when processing peculiar connections (Vern Paxson). - Fix for off-by-one buffer in sscanf call (Christian Kreibich). - Fixed inefficiency/warning flagged by g++ (Vern Paxson). - Bug fix for NUL string termination in SMB processing (Zhichun Li). - Fix for over-ref'ing of file Val's (Vern Paxson). - Fixes for some g++ warnings (Christian Kreibich, Vern Paxson). - gcc 3.4.2 portability fixes (Robin Sommer). - Minor build fixes for Broccoli, including a version bump to match that of Bro. See aux/broccoli/ChangeLog for details. - distcheck fixes (Christian Kreibich). - Configuration portability fixes (Matthias Vallentin, Jean-philippe Luiggi). - OpenBSD portability fixes (Jean-philippe Luiggi, Christian Kreibich). 1.2.1 Mon Dec 11 16:22:58 PST 2006 - Fixed delayed triggering of new_connection events when using the connection compressor. - Fixed tracking of first packet in TCP analyzer. (Reported by Guohan Lu) - The syslog built-in got lost during some previous merge. - Fixed crash if local variable is given as timeout value for table. (Reported by Mike Wood.) - Fixed using "time" values as table indices. - Added ssh to default brolite DPD configuration. - Fixed catching up to real-time in case of lull. - Fixed Broccoli "BRO_DATA_FORMAT_VERSION" to match version in Bro. - Fixed Makefile problem in doc directory. - Fixed Makefile dependency problem in binpac directory. - Added Linux tuning to brolite install script. - Modified Makefile to include broccoli/contrib. - Adding missing initialization to remote serializer. - Minor documentation updates for reference manual and Broccoli. 1.2 Tue Oct 17 12:09:49 PDT 2006 - Bro now supports DPD, dynamic protocol detection (Robin Sommer, Holger Dreger, and Michael Mai). With DPD, Bro can analyze protocols regardless of what port numbers they use: it infers the protocol based on which application analyzers can parse it without error. Adding this functionality involved extensive changes to Bro's internals, but also now enables multiple Bro analyzers to work on the same connection, either concurrently or one nested inside the other (we have not taken much advantage of this latter capability yet, but see the FTP events discussed below). There are a number of new policy scripts, events, and variables associated with DPD processing, as follows. Scripts: You activate DPD by @load'ing dpd.bro. It in turn instructs Bro to load the signature file policy/sigs/dpd.sig. Note that Bro uses signatures to expedite deciding which analyzers to try on a given connection; it does *not* simply use the signatures to make the determination of which protocol is in use, as this is insufficiently robust. (At this point, Bro provides signatures for FTP, IRC, HTTP, SMTP, and SSH. In the future we plan to add other protocols.) Along with dpd.bro, you need to @load detect-protocols.bro or detect-protocols-http.bro. The former enables general detection of application-layer protocols, while the latter does further inspection of HTTP sessions to characterize applications running on top of HTTP such as Gnutella or SOAP. (Loading dpd.bro is separate from loading one of these scripts because in principle Bro could use a different means than signatures to activate the analyzers, although currently it does not.) If you @load dyn-disable.bro, then once an analyzer determines that it does not match a given connection, it is deactivated (and a Notice is generated). Otherwise, it still proceeds to try its best to analyze the connection (to possibly be more robust against evasion). The scripts dce.bro and smb.bro enable DPD for the Windows DCE and SMB protocols, respectively. (Note that analysis of these protocols is undergoing a major expansion, not yet complete.) Events: event protocol_confirmation(c: connection, atype: count, aid: count) Generated when the given connection has been confirmed as conforming with the application type (protocol) specified by atype. aid is a globally unique analyzer ID that identifies a particular analyzer instance. The values for atype are symbolic names associated with each of Bro's analyzers, such as ANALYZER_IRC. See the initialization at the beginning of Analyzer.cc for the full set of names. The function analyzer_name(atype: count): string translates these symbolic names into text. For example, analyzer_name(ANALYZER_IRC) yields "IRC". event protocol_violation(c: connection, atype: count, aid: count, reason: string) Generated when the given connection has been found to violate the protocol of the given application type, with "reason" giving details. Variables: dpd_buffer_size: count (default 1024) Specifies how much pending data Bro keeps for connections that have not been classified yet. Once this fills, the data is deleted, though classification can still continue (see below). dpd_match_only_beginning: bool (default T) If set, specifies that Bro should stop signature matching if it has processed dpd_buffer_size bytes. dpd_ignore_ports: bool (default F) If set, then Bro does not take into consideration the port numbers associated with connections when attempting to classify them (which can otherwise help the process in some cases). dpd_reassemble_first_packets: bool (default T) If set, then Bro does TCP stream reassembly before applying signature-matching to detect protocols. likely_server_ports: set[port] Specifies a list of ports that Bro will consider as likely used by servers. For example, if Bro sees a connection that has already been established (so it does not know which side sent the initial SYN), and one side uses a port in this set, then it will assume that that side is the server (connection responder). The set is empty unless you populate it or @load server-ports.bro, which specifies a large number of values. dpd_config: table[AnalyzerTag] of dpd_protocol_config Specifies the DPD configuration associated with each tag. The type dpd_protocol_config is simply: type dpd_protocol_config: record { ports: set[port] &optional; }; i.e., an optional $ports field specifying a set of ports associatd with the tag. For example, ftp.bro now includes the equivalent of: redef dpd_config += { [ANALYZER_FTP] = [$ports = 21/tcp] }; Functions: The function expect_connection(orig: addr, resp: addr, resp_p: port, analyzer: count, tout: interval) is called to alert Bro that a new connection is expected, initiated by orig to a server running on resp's port resp_p (note: orig's port is not specified) which will correspond to the specified analyzer (e.g., "FILE", which is used to analyze files transferred by FTP - see next item). "tout" is a timeout to associate with the waiting. The function function disable_analyzer(cid: conn_id, aid: count) instructs Bro to disable the analyzer that generated the current event, assuming the analyzer is associated with the given connection ID. This is used by the dyn-disable.bro script discussed above. - A much more complete BinPAC compiler, along with new HTTP, DNS, and RPC/Portmap analyzers in binpac (Ruoming Pang). The flag "--use-binpac" activates the BinPAC-based analyzers (currently for HTTP and DNS). See www.cs.princeton.edu/~rpang/binpac-paper.pdf for a description of BinPAC, and let Ruoming know if you are interested in using BinPAC to build new analyzers. - A new type of analyzer, FILE, analyzes the contents of a connection as though it were a data file (Robin Sommer). Currently, it can generate two events: event file_transferred(c: connection, prefix: string, descr: string, mime_type: string) Indicates that the connection transferred a file. "prefix" is the beginning of the file's data; "descr" and "mime_type" are indicators of the file's type, as reported by the "libmagic" library. descr/mime_type are only set if Bro is configured on a system that includes the "libmagic" library. event file_virus(c: connection, virname: string) Indicates the connection transferred an executable corresponding to a known virus of the given name. This functionality is only available if Bro is configured on a system that includes the "libclamav" library. Note, this analyzer is enabled via a call to expect_connection by the FTP analyzer. - New events relating to IRC analysis (Robin Sommer): event irc_client(c: connection, prefix: string, data: string) Generated upon seing a client message sent over the given IRC connection. "prefix" is the command's prefix as defined by the IRC protocol. It is used by servers to indicate the true origin of the message; it may be empty. "data" contains the message. event irc_server(c: connection, prefix: string, data: string) Same for server messages. event irc_user_message(c: connection, user: string, host: string, server: string, real_name: string) Generated upon seeing an IRC "USER" command. event irc_password_message(c: connection, password: string) Generated upon seeing an IRC "PASS" command. event irc_channel_topic(c: connection, channel: string, topic: string) Generated upon seeing an IRC server reply that includes the channel topic. event irc_global_users(c: connection, prefix: string, msg: string) Generated upon seeing an IRC server reply that includes a count of the number of IRC users. - The new experimental script irc-bot.bro tracks IRC-based bots (Robin Sommer). The accompanying script irc-bot-syslog.bro syslog's the state of the bot analysis every IrcBot::summary_interval seconds (default 1 minute). - The new script proxy.bro looks for open Web proxies by matching incoming requests to a server with outgoing requests it makes (Robin Sommer). It generates HTTPProxyFound Notices when it finds one. - Changes to notices.bro (Robin Sommer): - notice_policy_item's now have a default $result of NOTICE_FILE and a default $priority of 1. - The new notice_action_filter, notice_alarm_per_orig, alarms on the first NoticeType from a specific source. Subsequent instances are tallied. - notice_action_filters now reside in the new script notice-action-filter.bro (automatically loaded by notice.bro). - The notice actions NOTICE_ALARM_PER_CONN, NOTICE_ALARM_PER_ORIG, and NOTICE_ALARM_ONCE have been removed, as they were never actually implemented. - If the notice_policy returns IGNORE or FILE, the action_filters filters are no longer consulted. - A new attribute for tables and sets, &mergeable, changes the semantics of assignments, as follows (Robin Sommer). Given two &mergeable tables/sets A and B, an assignment "A = B" becomes actually a join "A = A \cup B" (i.e., union). The envisoned use is to help avoid race conditions when doing remote state synchronization. - The semantics of &synchronized expire_funcs has changed (Robin Sommer). Now, when a table entry is expired and the operation is propagated to a a peer, the peer will call its expire_function. - TRW analysis now skips UDP traffic because it currently treats all UDP connections as failures (Robin Sommer). - trw.bro has been split into trw-impl.bro (the algorithm) and trw.bro (which simply activates the analysis), to facilitate writing scripts that have hooks into TRW analysis but don't presume it's active (Robin Sommer). - The option report_remote_notices in remote.bro has been replaced by a new script you include, remote-report-notices.bro (Robin Sommer). - The new function connect_peer() explicitly connects to a remote host (Robin Sommer). - The new script remote-send-id.bro sends the current value of an ID to a remote Bro and then terminates processing (Robin Sommer). It's intended for use from the command-line, as in bro -e "redef dst="" id="" remote-send-id The other scripts must set up the connection. is an index into Remote::destinations corresponding to the destination. - New built-ins {suspend,resume}_state_updates() can be called to temporarily avoid propagating updates to &sync'ed values (Robin Sommer). This can avoid duplicated activity. - The new function terminate_communication() instructs Bro to end its communication with remote peers (Robin Sommer). - The new event remote_state_access_performed is raised when remote state access has been performed (Robin Sommer). This is primarily for debugging. - The log() built-in has been renamed to ln() to avoid conflict (Vern Paxson). - bifcl now generates event generation wrapper functions from event.bif (Ruoming Pang). For example, to generate event http_reply, currently one writes: val_list* vl = new val_list; vl->append(BuildConnVal()); vl->append(new StringVal(fmt("%.1f", reply_version))); vl->append(new Val(reply_code, TYPE_COUNT)); if ( reply_reason_phrase ) vl->append(reply_reason_phrase); else vl->append(new StringVal("")); ConnectionEvent(http_reply, vl); In the future, one will be able to just call bro_event_http_reply(), and the code generated by bifcl looks like: void bro_event_http_reply(Connection* c, StringVal* version, bro_uint_t code, StringVal* reason) { val_list* vl = new val_list; vl->append(c->BuildConnVal()); vl->append(version); vl->append(new Val(code, TYPE_COUNT)); vl->append(reason); mgr.QueueEvent(http_reply, vl, SOURCE_LOCAL, c); } Accompanying this change is a semantic shift to types "string" and "port" in .bif files. They used to be translated to C++ types BroString* and uint32, respectively. Now they are translated to StringVal* and PortVal*. The functions in bro.bif are changed accordingly, and please be aware of this change when you write built-in functions in future. Also for this change, the parameter 'new' for rsh_request has been renamed 'new_session', as 'new' is a reserved word for C++. - Some ICMP "connections" now have services identified ("icmp-echo", "icmp-unreach") rather than just listing the service as "other" (Ruoming Pang). - The new option remote_trace_sync_interval specifies an interval after which each Bro will stop processing its trace and wait for all others to signal that they have reached the same time (Robin Sommer). The intent is support for operating Bro in a distributed cluster fashion (and in particular for debugging such clusters when running off-line on traces). This option only works in pseudo-realtime mode, and requires the new global remote_trace_sync_peers to give the total number of remote peers (not including self). Signaling is done via a new communication message type. - Extensions for DNS transformation/anonymization, including introduction of trace transformation for protocols other than TCP (Jason Lee). Not yet fully developed/debugged. - Extensions for HTTP transformation/anonymization (Martin Casado). Not yet fully developed/debugged. - The $conn field is now included in HTTPProxyFound notices (Robin Sommer). - Changed service inference algorithm to favor lower-numbered likely-servers over higher-numbered ones (Vern Paxson). - In pseudo-realtime mode, Bro now uses real-time for deciding which peer should send state (Robin Sommer). - Time synchronization for Bro's running on traces in pseudo-realtime mode added (Robin Sommer). - Avoidance of false content gaps improved when sorting packets with out-of-order timestamps (Ruoming Pang). - Packets from the packet sorter are now more robustly drained upon termination of input (Ruoming Pang). - Documentation for deep-copy updated (Christian Kreibich). - Nasty fragment reassembly bug fixed (Vern Paxson). - Serious bugs in EDNS0 processing fixed (Vern Paxson). - Fixed significant misfeature of interconn.bro that stopped all processing of a connection once it makes a detection (Vern Paxson). - Fixes for &read_expire operation across synchronizes tables (Robin Sommer). - Fixes for multiple peers exchanging initial &sync state simultaneously (Robin Sommer). - Improvements to graceful termination of Bro when communicating with remote peers (Robin Sommer). - Fix for ICMP analyzer not always generating icmp_sent events (Robin Sommer). This appears to still need some work, as now it generates redundant events. - Fix for initial exchange of &sync state which could lead to referencing unknown IDs (Robin Sommer). - Fix to scan detection for differing semantics of connection compressor vs. non-compressor (Robin Sommer). - Bug fix for distinguishing regular expression matches of length 0 from those of length 1 (Ruoming Pang). - Fix for SSH version parsing in the presence of content gaps (Robin Sommer). - Bug fix for IRC that could lead to crashes (Robin Sommer). - Bug fix to refrain from adding new timers when a connection has already been removed from the connection table (Robin Sommer). - Bug fix for packet_contents not including the transport-layer header (Robin Sommer). - Some memory leaks fixed (Robin Sommer). - A bunch of portability and distribution problems fixed (Christian Kreibich, Robin Sommer, Vern Paxson). 1.1 Mon May 15 10:50:33 PDT 2006 - Bro now supports a "when" statement for taking action upon something becoming true asynchronously (Robin Sommer). This provides a powerful new mechanism with numerous applications. Syntax: when '(' ')' [timeout '{ '}'] where the first can be a single statement or a block enclosed in {}'s, but the set associated with "timeout" must be enclosed in {}'s (to reduce ambiguities in Bro's grammar). Bro executes the first statement when becomes true. If you give a timeout and the condition has not been satisfied before it expires, Bro executes the second statement instead. A simple example: global t: table[addr] of count; event connection_established(c: connection) { local orig = c$id$orig_h; if ( orig !in t ) { t[orig] = 1; when ( t[orig] == 5 ) print fmt("%s has established 5 connections", orig); timeout 1 hr { print fmt("%s has NOT established 5 connections", orig); delete t[orig]; } } else ++t[orig]; } Notes: - The condition may be evaluated more than once, and at arbitrary times. - When the when-body is executed, the condition is guaranteed to be still satisfied. - Expression reevaluation is primarily triggered by modifications to globals. However, reevaluations do not take place immediately but potentially at a later point. This means that if we change a global to a value which would execute the trigger but then change it back, the change may go unnoticed. - Inside the condition you may introduce new locals. For example, when ( (local x = foo()) && x == 42 ) ... Such an assignment always yields true as its expression value (but the assignment might be delayed, for example if foo() is a delayed function call - see below). Delaying function calls ======================= Functions called inside the condition of a when-clause may delay their results until they're ready. This works for both script-level and built-in functions. For script-level functions, there is a new construct, "return ", to delay a function's result. When used, the function returns at the time the when-stmt's condition becomes true, and it yields the value that the when-stmt's body then returns. Toy example: global X: table[string] of count; function a() : count { # This delays until condition becomes true. return when ( "a" in X ) { return X["a"]; } timeout 5 min { return 0; } } event bro_init() { # Installs a trigger which fires if a() returns 42. when ( a() == 42 ) { print "Yippie!"; } X["a"] = 42; } There's also a new built-in function which can delay lookup_addr(host: addr) performs asynchronous DNS address->hostname lookups. Example: local h; addr; [...] when (local name = lookup_addr(h)) { print h, name; } See the function gen_hot_notice_with_hostnames() in conn.bro for a more worked-out example of using the "when" clause to translate the local address in SensitiveConnection notices to a hostname (contributed by Brian Tierney). This functionality is activated by redef'ing xlate_hot_local_addr to T. Here is the full evaluation model of a when's condition: - The condition may be evaluated more than once, at arbitrary times. - It is always fully evaluated, no matter whether some former evaluation has been suspended by a delaying function call. - All function calls which do not delay are always *fully* executed each time the condition is evaluated. - Function calls which delay are only executed *once*; their result is cached and re-used in the case the condition is evaluated again. - The condition is guaranteed to be true when the body is executed (potentially using cached function results) - By default Bro now uses a configuration similar to what used to be activated using reduce-memory.bro, along with some additional state timeouts that are new (Robin Sommer and Vern Paxson). This allows for better state management out-of-the-box, at the cost of some precision of analysis and resilience to evasion. In particular, the intent is to move towards being able to run Bro continuously without inexorably growing the amount of memory used until exhaustion. You can access a configuration similar to the previous default state management settings by loading heavy-analysis.bro. It turns on a load-prefix of "heavy", so when you load XXX.bro, a file heavy.XXX.bro will also be automatically loaded if present. Note that, as was the case for reduce-memory, you need to load heavy-analysis prior to other files for it to have effect. - The new module clear-passwords.bro monitors login/FTP/IRC/POP traffic for cleartext passwords (Jason Lee). - The new script service-probe.bro looks for remote hosts that repeatedly connect to the same service on local hosts (for a configurable set of services and connection sizes) in order to detect brute-forcing attacks such as password-guessing (Jim Mellander). - A new ARP analyzer generates three events: event arp_request(mac_src: string, mac_dst: string, SPA: addr, SHA: string, TPA: addr, THA: string); event arp_reply(mac_src: string, mac_dst: string, SPA: addr, SHA: string, TPA: addr, THA: string); event bad_arp(SPA: addr, SHA: string, TPA: addr, THA: string, explanation: string); with a corresponding policy script arp.bro (Chema Gonzalez and Vern Paxson). It writes logs to arp.$BRO_LOG_SUFFIX. It has not been tested much yet. - Bro Lite changes (Jason Lee): - default user for is now user 'bro' - now uses the correct sysctl on FreeBSD 6 - now uses the correct Perl path if site-report.pl not installed into '/usr/local/bro' - no longer prompts to encrypt email unless you pick to email reports - The default Bro Lite install now only checkpoints Bro once a week (Brian Tierney). - Implicit Bro file extensions (such as .bro for policy scripts and .sig for signatures) are now searched for first rather than only if the non-extension-version of the file doesn't exist (Vern Paxson). For example, running "bro -r trace mt" now first searches $BROPATH for "mt.bro" before searching for "mt", whereas it used to do these in the other order. - There's now a simpler mechanism for redef'ing variables on the command-line (Christian Kreibich). Any command line arguments of the form = are now expanded into policy code of the form "redef var=val;", where is wrapped in quotation marks if the value appears to be a string and doesn't have quotation marks already. This works with strings with whitespace such as foo="Hello World"; however, note that it means you can't use the mechanism to redef an enum value. - The Bro distribution now includes (and builds by default) Christian Kreibich's Broccoli library (Bro C Client Library), which enables programs to communicate with running Bro's (Christian Kreibich and Jason Lee). Configure with --disable-broccoli to turn this off. - Built-in functions log(x: double): double and exp(x: double): double which do natural logarithms and their inverses (Jaeyeon Jung). - The new built-in function gethostname() returns the local host's name (Jason Lee & Robin Sommer). - The new built-in function reading_traces() returns true if Bro is reading trace files (Robin Sommer). - The new built-ins suspend_processing() and continue_processing() provide script-level control for instructing the event engine to stop or resume processing packets (Robin Sommer). This is useful for coordinating simultaneous processing by multiple Bro's. - Email notices are now by default sent via /bin/mail, with "[Bro Alarm]" in the subject. - redef'ing a function now replaces the existing body rather than supplementing it (Robin Sommer), which was a bug. - You can now configure Bro to process encapsulated IP packets either by setting, as before, a fixed encap_hdr_size (for VLANs), or setting parse_udp_tunnels to T (Ruoming Pang). For the latter, you specify a UDP tunnel port using udp_tunnel_port (the previous variable "tunnel_port" has gone away); or you can leave it set to its default of 0/udp, in which case Bro will look for IP encapsulated in UDP packets on any port. - Added a simple form of profiling based on sampling the work done per-packet (Vern Paxson). The event engine generates a event load_sample(samples: load_sample_info, CPU: interval, dmem: int) event every load_sample_freq packets (roughly; it's randomized), where load_sample_freq defaults to 20. "samples" is simply a set[string]; it contains the names of the functions, event handlers, and their source files that were accessed during the processing of the sampled packet, along with an estimate of the CPU cost of processing the packet and (currently broken) memory allocated/freed. - Bro now includes experimental support for Endace DAG cards (Gregor Maier and Robin Sommer). To activate, configure with --with-DAG=/path/to/dagtool/installation and use "dag0" as the network interface. You may need to configure the card with the dagtools first. In general, if dagsnap works, Bro should work as well. - Log rotation has changed in a number of ways (Mark Dedlow & Robin Sommer): * The new variable log_rotate_base_time: string, if defined, specifies that logs should be rotated at log_rotate_base_time + i * rotate_interval intervals. Format is as a string in 24-hour time, "%H:%M", e.g, "12:00". This format may change in the future to instead be a Bro time type. * RotateLogs::date_format can be redefined to change format of timestamps in rotated files. * RotateLogs::build_name() can be redefined to implement an arbitrary naming scheme for rotated files. Note, this code has not been extensively tested. - Bro now by default builds a version of malloc bundled with its distribution (Vern Paxson & Brian Tierney). - The syntax for the clone operator now looks like a function call, "copy(x)" (Vern Paxson). - The new flag DNS::logging (default F), if T, disables generation of dns.log (which is often uninteresting and very large), though it still performs analysis leading to NOTICEs (Robin Sommer). - A new global, hostile_domain_list, has been added to dns.bro which lists domains to be flagged if A or MX records are queried (Scott Campbell). - Added globals dns_skip_all_{auth,addl} to skip all DNS AUTH/ADDL processing (Vern Paxson). Skipping these is on (true) by default, because such processing is quite expensive. - backdoor.bro now turns off by default some detectors that from experience have too many false positives, or (such as for HTTP) too many uninteresting true positives (Brian Tierney). In addition: - the module now generates a BackdoorFound notice for each backdoor - the new variable dump_backdoor_packets (default F) if set causes the packet that triggered the backdoor detection to be written to backdoor-packets/: