pagesift Copyright (C) 2026 YOLOW92 This product includes ideas, rules and data derived from the following projects. No source code from them is included; the algorithms were reimplemented in Rust. Binary distributions (release archives, the Docker image, the npm and Python packages) carry this file so that the notices below travel with the binaries. - trafilatura (https://github.com/adbar/trafilatura), Apache License 2.0, Copyright (C) Adrien Barbaresi and contributors. Content-container class/id patterns, boilerplate patterns, the tag whitelist, link-density tests and the extraction arbitration heuristics used by the article mode (upstream crate yolowhtml, re-exported by crates/pagesift-content). - Mozilla Readability (https://github.com/mozilla/readability), Apache License 2.0, Copyright (c) 2010 Arc90 Inc and the Mozilla contributors. The paragraph scoring, candidate selection and sibling rules and class/id word lists (upstream crate yolowhtml, re-exported by crates/pagesift-content). - htmldate (https://github.com/adbar/htmldate), Apache License 2.0, Copyright (C) Adrien Barbaresi and contributors. The search order, meta attribute lists, date patterns, candidate selection and plausibility rules of the publication-date search (upstream crate yolowhtml, re-exported by crates/pagesift-content). Its annotated evaluation set is used, unmodified and not redistributed, by benchmarks/dates/dates.py. - courlan (https://github.com/adbar/courlan), Apache License 2.0, Copyright (C) Adrien Barbaresi and contributors. The tracking-parameter rules used when cleaning crawl URLs (crates/pagesift-frontier/src/frontier/). - Heritrix (https://github.com/internetarchive/heritrix3), Apache License 2.0, Copyright (C) Internet Archive. The session-ID stripping and pathological-path rules of the crawl frontier (crates/pagesift-frontier/src/frontier/). - Scrapy (https://github.com/scrapy/scrapy), BSD 3-Clause License, Copyright (c) Scrapy developers. The retry policy (status codes, retry count, rescheduling instead of waiting) of the crawler (crates/pagesift-engine/src/crawl/). - semchunk (https://github.com/isaacus-dev/semchunk), MIT License, Copyright (c) Isaacus. The recursive chunking algorithm: splitter priorities and greedy merging of sibling splits (crates/pagesift-extract/src/recursive.rs). - pdfplumber (https://github.com/jsvine/pdfplumber), MIT License, Copyright (c) 2015, Jeremy Singer-Vine. The lattice table algorithm: snapping and joining ruling lines, intersections, smallest cells and grouping cells into tables (upstream crate yolowpdf, re-exported by crates/pagesift-pdf). - CFF standard strings (Adobe Technical Note #5176, Appendix A), as listed in pdfminer.six (https://github.com/pdfminer/pdfminer.six), MIT License (upstream crate yolowpdf, re-exported by crates/pagesift-pdf). This product also includes data from the following project, compiled into the binaries. - Adobe cmap-resources (https://github.com/adobe-type-tools/cmap-resources), BSD 3-Clause License, Copyright 1990-2019 Adobe. All rights reserved. The CID-to-Unicode tables of the Adobe-GB1, Adobe-CNS1, Adobe-Japan1 and Adobe-Korea1 character collections (cid2code.txt), converted to the binary tables in upstream crate yolowpdf (re-exported by crates/pagesift-pdf) and used to decode CJK text in PDFs whose CID fonts carry no ToUnicode map. The license text follows. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. Neither the name of Adobe nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. The PDF benchmark uses olmOCR-bench (https://huggingface.co/datasets/allenai/olmOCR-bench, ODC-BY 1.0) and its scoring code (https://github.com/allenai/olmocr, Apache License 2.0); both are downloaded, not redistributed. The chunking and query-filter benchmarks use the evaluation data of Chroma's chunking_evaluation (https://github.com/brandonstarxel/chunking_evaluation, MIT License); it is downloaded, not redistributed. The URL rewrite-rule learning follows the DUST idea from Bar-Yossef, Keidar and Schonfeld, "Do not crawl in the DUST: different URLs with similar text" (WWW 2007), and Dasgupta et al., "De-duping URLs via rewrite rules" (KDD 2008). - markup5ever (https://github.com/servo/html5ever), MIT OR Apache-2.0. The HTML named character reference table (upstream crate yolowhtml-dom, re-exported by crates/pagesift-html). - rustls (https://github.com/rustls/rustls), Apache-2.0 OR ISC OR MIT. crates/pagesift-rustls is the rustls 0.23.46 source with one added hook; its license files and the list of changes are in that directory.