# Multiscrape
---
> [!TIP]
>
> ## 👋 A quick note
>
> I run **[Smart Home Newsletter](https://smarthomenewsletter.com/?utm_source=github&utm_medium=readme&utm_campaign=multiscrape)** — a weekly curated digest for smart home enthusiasts.
>
> Each issue highlights the most relevant smart home **articles, news, and projects** from the past week.
>
> Since you're here, you're clearly into home automation — so you might genuinely enjoy the newsletter.
>
> 👉 [Subscribe at smarthomenewsletter.com](https://smarthomenewsletter.com/?utm_source=github&utm_medium=readme&utm_campaign=multiscrape)
---
[![GitHub Release][releases-shield]][releases]
[![License][license-shield]](LICENSE)
[![pre-commit][pre-commit-shield]][pre-commit]
[![Black][black-shield]][black]
[![hacs][hacsbadge]][hacs]
[![Project Maintenance][maintenance-shield]][user_profile]
[![BuyMeCoffee][buymecoffeebadge]][buymecoffee]
[![Discord][discord-shield]][discord]
[![Community Forum][forum-shield]][forum]
## Need help with Multiscrape?
If you don't manage to scrape the value you are looking for, please [enable debug logging](#debug-logging) and `log_response`. This will provide you with a lot of information for continued investigation. `log_response` will write all responses to files. If the value you want to scrape is not in the files with the output from BeautifulSoup (\*-soup.txt), Multiscrape will not be able to scrape it. Most likely it is retrieved in the background by javascript. Your best chance in this case, is to investigate the network traffic in de developer tools of your browser, and try to find a json response containing the value you are looking for.
If all of this doesn't help, use the home assistant forum. I cannot give everyone personal assistance and please don't create github issues unless you are sure your issue isn't caused by configuration.
Check the [wiki](https://github.com/danieldotnl/ha-multiscrape/wiki) for a scraping guide and other details on the functionality of this component.
## Important note: be a good citizen and be aware of your responsibility
You and you alone, are accountable for your scraping activities. Be a good (web) citizen. Set reasonable `scan_interval` timings, seek explicit permission before scraping, and adhere to local and international laws. Respect website policies, handle data ethically, mind resource usage, and regularly monitor your actions. Uphold these principles to ensure ethical and sustainable scraping practices.
## Introduction
This Home Assistant custom component can scrape multiple fields (using CSS selectors) from a single HTTP request (the existing scrape sensor can scrape a single field only). The scraped data becomes available in separate sensors.
It is based on both the existing [Rest sensor](https://www.home-assistant.io/integrations/rest/) and the [Scrape sensor](https://www.home-assistant.io/integrations/scrape). Most properties of the Rest and Scrape sensor apply.
## Installation
[![hacs][hacsbadge]][hacs]
[](https://my.home-assistant.io/redirect/hacs_repository/?owner=danieldotnl&repository=ha-multiscrape&category=integration)\
Install via HACS (default store) or install manually by copying the files in a new 'custom_components/multiscrape' directory.
## Example configuration (YAML)
_This code example is to be placed into /config/configuration.yaml_
```yaml
multiscrape:
- name: HA scraper
resource: https://www.home-assistant.io
scan_interval: 3600
sensor:
- unique_id: ha_latest_version
name: Latest version
select: ".release-date"
value_template: "{{ value | trim }}"
- unique_id: ha_release_date
icon: >-
{% if is_state('binary_sensor.ha_version_check', 'on') %}
mdi:alarm-light
{% else %}
mdi:bat
{% endif %}
name: Release date
select: ".release-date"
attribute: "title"
value_template: "{{ (value.split('released')[1]) }}"
binary_sensor:
- unique_id: ha_version_check
name: Latest version == 2021.7.0
select: ".release-date"
value_template: '{{ value | trim == "2021.7.0" }}'
attributes:
- name: Release notes link
select: ".release-date"
attribute: href
```
### Advanced Example Configuration (YAML)
For background on splitting the HA configuration, see the [HA Documentation](https://www.home-assistant.io/docs/configuration/splitting_configuration/).
_Inside the configuration.yaml file_
```yaml
multiscrape: !include multiscrape.yaml
```
Make a new file named /config/multiscrape.yaml
_Inside the multiscrape.yaml file. Syntax is the same but starting at the resource level_
```yaml
- resource: https://www.home-assistant.io
scan_interval: 3600
sensor:
- unique_id: ha_latest_version
name: Latest version
select: ".release-date"
value_template: "{{ value | trim }}"
- unique_id: ha_release_date
icon: >-
{% if is_state('binary_sensor.ha_version_check', 'on') %}
mdi:alarm-light
{% else %}
mdi:bat
{% endif %}
name: Release date
select: ".release-date"
attribute: "title"
value_template: "{{ (value.split('released')[1]) }}"
binary_sensor:
- unique_id: ha_version_check
name: Latest version == 2021.7.0
select: ".release-date"
value_template: '{{ value | trim == "2021.7.0" }}'
attributes:
- name: Release notes link
select: ".release-date"
attribute: href
```
## Options
Based on latest (pre) release.
| name | description | required | default | type |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------- | -------- | ------- | ----------------- |
| name | The name for the integration. | False | | string |
| resource | The url for retrieving the site or a template that will output an url. Not required when `resource_template` is provided. | True | | string |
| resource_template | A template that will output an url after being rendered. Only required when `resource` is not provided. | True | | template |
| authentication | Configure HTTP authentication. `basic` or `digest`. Use this with username and password fields. | False | | string |
| username | The username for accessing the url. | False | | string |
| password | The password for accessing the url. | False | | string |
| headers | The headers for the requests. | False | | template - list |
| params | The query params for the requests. | False | | template - list |
| method | The method for the request. Either `POST`, `GET`, or `PUT`. | False | GET | string |
| payload | Optional payload to send with a `POST` or `PUT` request. | False | | template - string |
| verify_ssl | Verify the SSL certificate of the endpoint. | False | True | boolean |
| log_response | Log the HTTP responses and HTML parsed by BeautifulSoup in files. (Will be written to/config/multiscrape/name_of_config) | False | False | boolean |
| timeout | Defines max time to wait data from the endpoint. | False | 10 | int |
| scan_interval | Determines how often the url will be requested. | False | 60 | int |
| max_retries | Number of automatic retries after a failed run. Only applies when `scan_interval` is 0. See [Retries](#retries). | False | 3 | int |
| parser | Determines the parser to be used with beautifulsoup. `lxml-xml` for xml recommended and `lxml` for everything else. | False | lxml | string |
| list_separator | Separator to be used in combination with `select_list` features. | False | , | string |
| form_submit | See [Form-submit](#form-submit) | False | | |
| sensor | See [Sensor](#sensorbinary-sensor) | False | | list |
| binary_sensor | See [Binary sensor](#sensorbinary-sensor) | False | | list |
| button | See [Refresh button](#refresh-button) | False | | list |
### Sensor/Binary Sensor
Configure the sensors that will scrape the data.
| name | description | required | default | type |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- | --------------- |
| unique_id | Will be used as entity_id and enables editing the entity in the UI | False | | string |
| name | Friendly name for the sensor | False | | string |
| | See [Selector](#Selector) fields | True | | |
| attributes | See [Sensor attributes](#sensor-attributes) | False | | list |
| unit_of_measurement | Defines the units of measurement of the sensor | False | | string |
| device_class | Sets the device_class for [sensors](https://www.home-assistant.io/integrations/sensor/) or [binary sensors](https://www.home-assistant.io/integrations/binary_sensor/) | False | | string |
| state_class | Defines the state class of the sensor, if any. (measurement, total or total_increasing) (not for binary_sensor) | False | None | string |
| icon | Defines the icon or a template for the icon of the sensor. The value of the selector (or value_template when given) is provided as input for the template. For binary sensors, the value is parsed in a boolean. | False | | string/template |
| picture | Contains a path to a local image and will set it as entity picture | False | | string |
| force_update | Sends update events even if the value hasn’t changed. Useful if you want to have meaningful value graphs in history. | False | False | boolean |
### Refresh button
Configure a refresh button to manually trigger scraping.
| name | description | required | default | type |
| --------- | ------------------------------------------------------------------ | -------- | ------- | ------ |
| unique_id | Will be used as entity_id and enables editing the entity in the UI | False | | string |
| name | Friendly name for the button | False | | string |
### Sensor attributes
Configure the attributes on the sensor that can be set with additional scraping values.
| name | description | required | default | type |
| ---- | ----------------------------------------- | -------- | ------- | ------ |
| name | Name of the attribute (will be slugified) | True | | string |
| | See [Selector](#Selector) fields | True | | |
### Form-submit
Configure the form-submit functionality which enables you to submit a (login) form before scraping a site. More details on how this works [can be found on the wiki](https://github.com/danieldotnl/ha-multiscrape/wiki/Form-submit-functionality).
| name | description | required | default | type |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- | --------------------- |
| resource | The url for the site with the form | False | | string |
| select | CSS selector used for selecting the form in the html. When omitted, the input fields are directly posted. | False | | string |
| input | A dictionary with name/values which will be merged with the input fields on the form. Values are rendered as templates when the form is submitted. See [Form input templates](#form-input-templates). | False | | template - dictionary |
| input_filter | A list of input fields that should not be submitted with the form | False | | string - list |
| submit_once | Submit the form only once on startup instead of each scan interval | False | False | boolean |
| resubmit_on_error | Resubmit the form after a scraping error is encountered | False | True | boolean |
| variables | See [Form Variables](#Form-Variables) | False | | list |
#### Form input templates
> **Breaking change:** `form_submit: input:` values are now rendered as templates. If an existing value contains a literal `{{`, `{%` or `{#` (for example in a password), Home Assistant will refuse to load the whole multiscrape entry with a template syntax error. Escape the marker: `"{{ '{{' }}literal"`, `"{{ '{%' }}literal"` or `"{{ '{#' }}literal"`. Values are also stripped of leading and trailing whitespace, so `" secret "` is submitted as `secret`.
The values in `input` are rendered as [Home Assistant templates](https://www.home-assistant.io/docs/configuration/templating/) at the moment the form is submitted. The input fields scraped from the form are available as the `form` variable, so a value can be computed from another field on the same form.
This is what you need for login forms that hash the password in JavaScript using a nonce from a hidden field. For example, AT&T BGW320 gateways post `md5(access_code + nonce)`:
```yaml
multiscrape:
- resource: http://192.168.1.254/cgi-bin/nattable.ha
form_submit:
resource: http://192.168.1.254/cgi-bin/login.ha
select: "form"
input:
password: "**********"
hashpassword: "{{ md5('' ~ form.nonce) }}"
resubmit_on_error: true
sensor:
- unique_id: att_nat_sessions_in_use
name: NAT sessions in use
select: 'table[summary*="summary of session information"] tr:nth-child(2) td'
```
`md5`, `sha1`, `sha256` and `sha512` are built-in Home Assistant template functions (and filters), so no extra configuration is needed.
Notes:
- Use `form['field-name']` instead of `form.field-name` for field names that are not valid identifiers.
- Fields listed in `input_filter` are still available in `form`, so you can use a field to compute a value without submitting the field itself.
- `form` only contains `` elements with a `name` attribute. `
