# softhauzpy — Full Technical & API Reference

Version: 1.0.0
Python Compatibility: >= 3.9
License: MIT
Author: Urate, Karen

---

## System Overview

[![Run softhauzpy Test Suite](https://github.com/softhauz/softhauzpy/actions/workflows/tests.yml/badge.svg)](https://github.com/softhauz/softhauzpy/actions/workflows/tests.yml)

`softhauzpy` is a Python package designed for software engineers and web application architects needing scalable web data tools and intelligent search capabilities. It provides web utilities including web scraping tools, content extraction pipelines, and in-house search engine components without relying heavily on external search services.

---

## Dependencies & Installation

### Core Dependencies
- `requests >= 2.34.2`
- `beautifulsoup4 >= 4.14.3`
- `nltk >= 3.9.4` (optional; degrades gracefully if missing)

### Installation
```bash
pip install softhauzpy
```

## Complete API Reference

### 1. Input Processing & Text Extraction

#### `softhauzpy.detect_input_type(value: str) -> str`
Detects whether an input string is a network URL, an HTML file path, or a raw string payload.
- **Parameters:**
  - `value` (`str`): Input string to evaluate.
- **Returns:** `"url"`, `"html_file"`, or `"raw_string"`.

#### `softhauzpy.extract_pure_text(page_url: str, *, title: str | None = None, author: str | None = None, description: str | None = None, creation_date: str | None = None, modified_date: str | None = None, assigned_location: str | None = None) -> dict`
Fetches or parses HTML and extracts pure body text excluding script, style, and metadata tags.
- **Parameters:**
  - `page_url` (`str`): URL, local HTML file path, or raw HTML content string.
  - Keyword-only metadata flags: `title`, `author`, `description`, `creation_date`, `modified_date`, `assigned_location`.
- **Returns:** `dict` with keys `url`, `title`, `author`, `description`, `creation_date`, `modified_date`, `content`, `meta_data`.

#### `softhauzpy.tokenize(text: str, *, remove_stopwords: bool = True, stem: bool = True, min_token_len: int = 2) -> list[str]`
Normalizes raw text into clean tokens by lowercasing, stripping punctuation, filtering stopwords, and optional Porter stemming.

---

### 2. Scraping & Web Crawling

#### `softhauzpy.fetch_page(url: str, *, timeout: int = 10, retries: int = 3, delay: float = 1.0, headers: dict | None = None, session: requests.Session | None = None) -> requests.Response | None`
Fetches a single URL with built-in retry mechanisms and exponential backoff delay.

#### `softhauzpy.parse_html(html: str | bytes, *, parser: str = "html.parser") -> BeautifulSoup`
Parses raw HTML string or bytes into a BeautifulSoup syntax tree.

#### `softhauzpy.extract_metadata(soup: BeautifulSoup, url: str = "") -> dict`
Extracts title, meta description, keywords, OpenGraph properties, canonical URL, language, and author.

#### `softhauzpy.extract_links(soup: BeautifulSoup, base_url: str, *, same_domain_only: bool = True, exclude_extensions: list[str] | None = None) -> list[str]`
Extracts and normalizes absolute hyperlinks from a page, automatically ignoring `mailto:`, `tel:`, `#`, and specified binary file extensions.

#### `softhauzpy.crawl_site(start_url: str, *, max_pages: int = 200, same_domain_only: bool = True, delay: float = 0.5, session: requests.Session | None = None) -> list[dict]`
Breadth-first search (BFS) crawler starting from `start_url`.
- **Returns:** `list[dict]` where each dictionary contains `url`, `html`, `soup`, and `status_code`.

#### `softhauzpy.build_sitemap_urls(base_url: str, *, session: requests.Session | None = None) -> list[str]`
Parses a site's `/sitemap.xml` or `/sitemap_index.xml` to build a seed queue of page URLs.

---

### 3. Search Engine, Indexing & Scoring Functions

#### `softhauzpy.get_search_results_list(page_list: list = None, keywords: str = '') -> list`
Filters a list of page tuple entries based on whether `keywords` are contained within the extracted text.
- **Expected Page Structure:** Tuple of `(url, title, author, description, creation_date, modified_date, assigned_location)`.
- **Returns:** List of filtered page tuples matching the keywords.

#### `softhauzpy.get_document_score(url: str, *, query: str = '', remove_stopwords: bool = False, stem: bool = False, unit: float = 1.0) -> float`
Calculates a custom document score by counting token occurrences multiplied by the metric unit preference.

#### `softhauzpy.build_inverted_index(documents: list[dict], *, text_field: str = "text", id_field: str = "url") -> dict`
Creates an inverted index mapping tokens to document occurrences.
- **Returns:** `{ token: [(doc_id, frequency), ...] }`

#### `softhauzpy.compute_tfidf(documents: list[dict], *, text_field: str = "text", id_field: str = "url") -> dict[str, dict[str, float]]`
Calculates full corpus Term Frequency-Inverse Document Frequency (TF-IDF) relevance weights across all documents.
- **Returns:** `{ doc_id: { token: tfidf_score } }`

#### `softhauzpy.search_index(query: str, index: dict, tfidf: dict[str, dict[str, float]], *, top_k: int = 10) -> list[tuple[str, float]]`
Ranks documents against a search query using the inverted index and pre-calculated TF-IDF scores.
- **Returns:** Ordered list of `(doc_id, score)` tuples, highest score first.

---

### 4. Helper Utilities & Persistence

#### `softhauzpy.chunk_text(text: str, *, chunk_size: int = 300, overlap: int = 50) -> list[str]`
Splits long documents into overlapping word-level windows for chunk indexing.

#### `softhauzpy.generate_snippet(text: str, query: str, *, window: int = 40, max_length: int = 300) -> str`
Generates a query-centered excerpt surrounded by ellipses.

#### `softhauzpy.highlight_query_terms(snippet: str, query: str, *, open_tag: str = "<mark>", close_tag: str = "</mark>") -> str`
Wraps matching non-stopword query tokens in HTML tags for UI display.

#### `softhauzpy.fingerprint_page(text: str) -> str`
Computes a stable SHA-256 digest of normalized page content to detect content updates.

#### Persistence Functions
- `softhauzpy.save_index(index: dict, tfidf: dict, metadata: list[dict], path: str = "search_index.json") -> None`
- `softhauzpy.load_index(path: str = "search_index.json") -> tuple[dict, dict, list]`
- `softhauzpy.incremental_update(url: str, index: dict, tfidf: dict, metadata: list[dict], fingerprints: dict[str, str], *, session: requests.Session | None = None) -> bool`