` with an ID of `content` becomes available in the DOM: ```bash shot-scraper https://.../ \ --wait-for 'document.querySelector("div#content")' ``` Here's an example that waits for a specific element to become available (in this case a cookie consent overlay) and then removes it before the screenshot is taken: ```bash shot-scraper -h 800 'https://www.spiegel.de/international/' \ --wait-for "() => { const div = document.querySelector('[id^="sp_message_container"]'); if (div) { div.remove(); return true; } }" ``` If the expression takes longer than 30 seconds to resolve `shot-scraper` will exit with an error. ## Executing custom JavaScript You can use custom JavaScript to modify the page after it has loaded (after the 'onload' event has fired) but before the screenshot is taken using the `--javascript` option: ```bash shot-scraper https://simonwillison.net/ \ -o simonwillison-pink.png \ --javascript "document.body.style.backgroundColor = 'pink';" ``` For larger scripts that would be awkward to include on the command line, use `--js-file` to read the JavaScript from a file instead: ```bash shot-scraper https://simonwillison.net/ \ -o simonwillison-modified.png \ --js-file remove-banners.js ``` Use `--js-file -` to read the script from standard input, or `--js-file gh:username/script` to load it from `github.com/username/shot-scraper-scripts/script.js` on GitHub. The `--js-file` option is also available for the `pdf`, `html`, `accessibility` and `har` commands. ## Using JPEGs instead of PNGs Screenshots default to PNG. Use a `-o` filename ending in `.jpg` or `.jpeg` to save as JPEG, or `.webp` to save as WebP. You can also use `--quality X` to save as a JPEG with the specified quality, in order to reduce the filesize. 80 is a good value to use here: ```bash shot-scraper https://simonwillison.net/ \ -h 800 -o simonwillison.jpg --quality 80 ``` ``` % ls -lah simonwillison.jpg -rw-r--r--@ 1 simon staff 168K Mar 9 13:53 simonwillison.jpg ``` ## Using WebP Save a screenshot as WebP by specifying a filename ending in `.webp`: ```bash shot-scraper https://simonwillison.net/ -h 800 -o simonwillison.webp ``` WebP screenshots are lossless by default, equivalent to `--quality 100`. Use a lower quality to apply lossy compression: ```bash shot-scraper https://simonwillison.net/ \ -h 800 -o simonwillison.webp --quality 80 ``` WebP also works with selectors: ```bash shot-scraper https://simonwillison.net/ -s '#bighead' -o bighead.webp ``` WebP images are limited to [16,383 pixels in either dimension](https://developers.google.com/speed/webp/faq#what_is_the_maximum_size_a_webp_image_can_be), so use `--height` or a selector when capturing a page that exceeds that limit. ## Specifying the image format Use `--format png`, `--format jpeg` or `--format webp` to select the format explicitly. This takes precedence over the output filename extension. Without `-o`, the automatic filename uses the selected format. This writes to `datasette-io.webp`: ```bash shot-scraper https://datasette.io/ --format webp ``` Use `--format` with `-o -` to write WebP or JPEG to standard output: ```bash shot-scraper https://datasette.io/ --format webp -o - > datasette.webp ``` The `--quality` option accepts values from 0 to 100 for JPEG and WebP. Combining `--format png` with `--quality` is an error because PNG does not use a quality setting. ## Device scale factor The `--scale-factor` option sets a specific device scale factor, which effectively simulates different device pixel ratios. This setting is useful for testing high-definition displays or emulating screens with various pixel densities. For example, setting `--scale-factor 3` results in an image with a CSS pixel ratio of 3, which is ideal for emulating a high-resolution display, such as Apple iPhone 12 screens. To take a screenshot with a scale factor of 3 (tripled resolution), run the following command: ```bash shot-scraper https://simonwillison.net/ -o simon.png \ --width 390 --height 844 --scale-factor 3 ``` This will multiply both the width and height of the screenshot by 3, resulting in an image that is 1170px wide and 2532px high, matching the iPhone 12's screen. The `--scale-factor` option takes a positive float as input. For example, setting `--scale-factor 2.625` simulates the Google Pixel 6's CSS pixel ratio. ## Retina images The `--retina` option is a shortcut to set a device scale factor of 2. This means that an image will have its resolution effectively doubled, emulating the display of images on [retina](https://en.wikipedia.org/wiki/Retina_display) or higher pixel density screens. ```bash shot-scraper https://simonwillison.net/ -o simon.png \ --width 400 --height 600 --retina ``` This example will produce an image that is 800px wide and 1200px high. Note: The `--retina` option should not be used in conjunction with the `--scale-factor` flag as they are mutually exclusive. If both are provided, the command will raise an error to prevent conflicts. ## Transparent background The `--omit-background` option instructs the browser to ignore the default background, allowing for the capture of a page with a transparent background. The default Chromium browser supports this for PNG and WebP images. JPEG does not support transparency. ```bash shot-scraper https://simonwillison.net/ -o simon.png \ --width 400 --height 600 --omit-background ``` WebP can preserve transparency when using `--quality`: ```bash shot-scraper https://simonwillison.net/ -o simon.webp \ --width 400 --height 600 --omit-background --quality 80 ``` ## Interacting with the page Sometimes it's useful to be able to manually interact with a page before the screenshot is captured. Add the `--interactive` option to open a browser window that you can interact with. Then hit `` in the terminal when you are ready to take the shot and close the window. ```bash shot-scraper https://simonwillison.net/ -o after-interaction.png \ --height 800 --interactive ``` This will output: ``` Hit to take the shot and close the browser window: # And after you hit ... Screenshot of 'https://simonwillison.net/' written to 'after-interaction.png' ``` ## Logging all requests It can sometimes be useful to see a list of all of the requests that the browser made while it was rendering a page. Use `--log-requests` to output newline-delimited JSON representing each request, including requests for images and other assets. Pass `-` to output the list to standard output, or use a filename to write to a file on disk. The output looks like this: ```bash shot-scraper http://datasette.io/ --log-requests - ``` ``` {"method": "GET", "url": "http://datasette.io/", "status": 302, "size": null, "timing": {"startTime": 1663211674984.7068, "domainLookupStart": 0.698, "domainLookupEnd": 1.897, "connectStart": 1.897, "secureConnectionStart": -1, "connectEnd": 18.726, "requestStart": 18.86, "responseStart": 99.75, "responseEnd": 101.75000000162981}} {"method": "GET", "url": "https://datasette.io/", "status": 200, "size": 34592, "timing": {"startTime": 1663211675085.51, "domainLookupStart": 0.187, "domainLookupEnd": 0.197, "connectStart": 0.197, "secureConnectionStart": 15.719, "connectEnd": 63.854, "requestStart": 64.098, "responseStart": 390.231, "responseEnd": 399.268}} {"method": "GET", "url": "https://datasette.io/static/site.css", "status": 200, "size": 3952, "timing": {"startTime": 1663211675486.027, "domainLookupStart": -1, "domainLookupEnd": -1, "connectStart": -1, "secureConnectionStart": -1, "connectEnd": -1, "requestStart": 0.408, "responseStart": 99.407, "responseEnd": 100.433}} ... ``` Note that the `size` field here will be the size of the response in bytes, but in some circumstances this will not be available and it will be returned as `"size": null`. ## Browser arguments Additional arguments to pass to the browser instance. The list of Chromium flags can be found [here](https://peter.sh/experiments/chromium-command-line-switches/). For example, to remove font render hinting: ```bash shot-scraper https://simonwillison.net/ -o no-hinting.png \ --height 800 --browser-arg "--font-render-hinting=none" ``` To add multiple arguments, add `--browser-arg` for each argument: ```bash shot-scraper https://simonwillison.net/ -o no-hinting-no-gpu.png \ --height 800 --browser-arg "--font-render-hinting=none" --browser-arg "--disable-gpu" ``` ## Taking screenshots of local HTML files You can pass the path to an HTML file on disk to take a screenshot of that rendered file: ```bash shot-scraper index.html -o index.png ``` CSS and images referenced from that file using relative paths will also be included. ## Tips for executing JavaScript If you are using the `--javascript` option to execute code, that code will be executed after the page load event has fired but before the screenshot is taken. You can use that code to do things like hide or remove specific page elements, click on links to open menus, or even add annotations to the page such as this [pink arrow example](https://simonwillison.net/2022/Mar/10/shot-scraper/#a-complex-example). This code hides any element with a `[data-ad-rendered]` attribute and the element with `id="ensNotifyBanner"`: ```javascript document.querySelectorAll( '[data-ad-rendered],#ensNotifyBanner' ).forEach(el => el.style.display = 'none') ``` You can execute that like so: ```bash shot-scraper https://www.latimes.com/ -o latimes.png --javascript " document.querySelectorAll( '[data-ad-rendered],#ensNotifyBanner' ).forEach(el => el.style.display = 'none') " ``` In some cases you may need to add a pause that executes during your custom JavaScript before the screenshot is taken - for example if you click on a button that triggers a short fading animation. You can do that using the following pattern: ```javascript new Promise(takeShot => { // Your code goes here // ... setTimeout(() => { // Resolving the promise takes the shot takeShot(); }, 1000); }); ``` If your custom code defines a `Promise`, `shot-scraper` will wait for that promise to complete before taking the screenshot. Here the screenshot does not occur until the `takeShot()` function is called. If you see errors relating to CSP headers such as "Failed to fetch dynamically imported module" you can work around them using {ref}`the --bypass-csp option`. ## Viewing console.log() output Almost all of the `shot-scraper` commands accept a `--log-console` option, which will cause them to output any calls to `console.log()` to standard error while the command is running. This includes both `console.log()` calls in the existing page JavaScript, as well as any calls to that method that you include in your own custom JavaScript. For example, running `--log-console` while taking a screenshot of the Facebook homepage will show this warning message, which Facebook logs to the developer tools console to help protect people from social engineering attacks: ``` % shot-scraper shot facebook.com --log-console .d8888b. 888 888 d88P Y88b 888 888 Y88b. 888 888 This is a browser feature intended for "Y888b. 888888 .d88b. 88888b. 888 developers. If someone told you to copy-paste "Y88b. 888 d88""88b 888 "88b 888 something here to enable a Facebook feature "888 888 888 888 888 888 Y8P or "hack" someone's account, it is a Y88b d88P Y88b. Y88..88P 888 d88P scam and will give them access to your "Y8888P" "Y888 "Y88P" 88888P" 888 Facebook account. 888 888 888 See https://www.facebook.com/selfxss for more information. Screenshot of 'http://facebook.com' written to 'facebook-com.png' ``` ## `shot-scraper shot --help` Full `--help` for this command: ``` Usage: shot-scraper shot [OPTIONS] URL Take a single screenshot of a page or portion of a page. Usage: shot-scraper www.example.com This will write the screenshot to www-example-com.png Use "-o" to write to a specific file: shot-scraper https://www.example.com/ -o example.png You can also pass a path to a local file on disk: shot-scraper index.html -o index.png Using "-o -" will output to standard out: shot-scraper https://www.example.com/ -o - > example.png Use -s to take a screenshot of one area of the page, identified using one or more CSS selectors: shot-scraper https://simonwillison.net -s '#bighead' Options: -a, --auth FILENAME Path to JSON authentication context file -w, --width INTEGER Width of browser window, defaults to 1280 -h, --height INTEGER Height of browser window and shot - defaults to the full height of the page -o, --output FILE -s, --selector TEXT Take shot of first element matching this CSS selector --selector-all TEXT Take shot of all elements matching this CSS selector --js-selector TEXT Take shot of first element matching this JS (el) expression --js-selector-all TEXT Take shot of all elements matching this JS (el) expression -p, --padding INTEGER When using selectors, add this much padding in pixels -j, --javascript TEXT Execute this JS prior to taking the shot --js-file TEXT Read JavaScript to execute from this file, use - for stdin or gh:username/script to load from github.com/username/shot-scraper- scripts/script.js --scale-factor FLOAT Device scale factor. Cannot be used together with '--retina'. --retina Use device scale factor of 2. Cannot be used together with '--scale-factor'. --omit-background Omit the default browser background from the shot, making it possible take advantage of transparency. Does not work with JPEGs. --format [png|jpeg|webp] Image format. Overrides the output extension; use with -o - for stdout. --quality INTEGER RANGE JPEG or WebP quality, e.g. 80. Defaults to JPEG unless WebP is selected. WebP defaults to 100 (lossless). [0<=x<=100] --wait INTEGER Wait this many milliseconds before taking the screenshot --wait-for TEXT Wait until this JS expression returns true --timeout INTEGER Wait this many milliseconds before failing -i, --interactive Interact with the page in a browser before taking the shot --devtools Interact mode with developer tools --log-requests FILENAME Log details of all requests to this file --log-console Write console.log() to stderr -b, --browser [chromium|firefox|webkit|chrome|chrome-beta] Which browser to use --browser-arg TEXT Additional arguments to pass to the browser --user-agent TEXT User-Agent header to use --reduced-motion Emulate 'prefers-reduced-motion' media feature --fail Fail with an error code if a page returns an HTTP error --skip Skip pages that return HTTP errors --bypass-csp Bypass Content-Security-Policy --silent Do not output any messages --auth-password TEXT Password for HTTP Basic authentication --auth-username TEXT Username for HTTP Basic authentication --help Show this message and exit. ```