{ "cells": [ { "cell_type": "markdown", "metadata": { "button": false, "new_sheet": false, "run_control": { "read_only": false }, "slideshow": { "slide_type": "slide" } }, "source": [ "# Testing Graphical User Interfaces\n", "\n", "In this chapter, we explore how to generate tests for Graphical User Interfaces (GUIs), abstracting from our [previous examples on Web testing](WebFuzzer.ipynb). Building on general means to extract user interface elements and to activate them, our techniques generalize to arbitrary graphical user interfaces, from rich Web applications to mobile apps, and systematically explore user interfaces through forms and navigation elements." ] }, { "cell_type": "markdown", "metadata": { "button": false, "new_sheet": false, "run_control": { "read_only": false }, "slideshow": { "slide_type": "subslide" } }, "source": [ "**Prerequisites**\n", "\n", "* We build on the Web server introduced in the [chapter on Web testing](WebFuzzer.ipynb)." ] }, { "cell_type": "markdown", "metadata": { "button": false, "new_sheet": true, "run_control": { "read_only": false }, "slideshow": { "slide_type": "slide" } }, "source": [ "## Automated GUI Interaction\n", "\n", "In the [chapter on Web testing](WebFuzzer.ipynb), we have shown how to test Web-based interfaces by directly interacting with a Web server using the HTTP protocol, and processing the retrieved HTML pages to identify user interface elements. While these techniques work well for user interfaces that are based on HTML only, they fail as soon as there are interactive elements that use JavaScript to execute code within the browser, and generate and change the user interface without having to interact with the browser." ] }, { "cell_type": "markdown", "metadata": { "button": false, "new_sheet": true, "run_control": { "read_only": false }, "slideshow": { "slide_type": "subslide" } }, "source": [ "In this chapter, we therefore take a different approach to user interface testing. Rather than using HTTP and HTML as the mechanisms for interaction, we leverage a dedicated _UI testing framework_, which allows us to\n", "\n", "* query the program under test for available user interface elements, and\n", "* query the UI elements for how they can be interacted with.\n", "\n", "Although we will again illustrate our approach using a Web server, the approach easily generalizes to _arbitrary user interfaces_. In fact, the UI testing framework we use, *Selenium*, also comes in variants that run for Android apps." ] }, { "cell_type": "markdown", "metadata": { "button": false, "new_sheet": true, "run_control": { "read_only": false }, "slideshow": { "slide_type": "subslide" } }, "source": [ "### Our Web Server, Again\n", "\n", "As in the [chapter on Web testing](WebFuzzer.ipynb), we run a Web server that allows us to order products." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "button": false, "new_sheet": false, "run_control": { "read_only": false }, "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "import fuzzingbook_utils" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from WebFuzzer import init_db, start_httpd, webbrowser, print_httpd_messages, print_url, ORDERS_DB" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "import html" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "db = init_db()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "This is the address of our web server:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "httpd_process, httpd_url = start_httpd()\n", "print_url(httpd_url)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Using `webbrowser()`, we can retrieve the HTML of the home page, and use `HTML()` to render it." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from IPython.display import display, Image" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from fuzzingbook_utils import HTML, rich_output" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "HTML(webbrowser(httpd_url))" ] }, { "cell_type": "markdown", "metadata": { "button": false, "new_sheet": false, "run_control": { "read_only": false }, "slideshow": { "slide_type": "subslide" } }, "source": [ "### Remote Control with Selenium\n", "\n", "Let us take a look at the GUI above. In contrast to the [chapter on Web testing](WebFuzzer.ipynb), we do not assume we can access the HTML source of the current page. All we assume is that there is a set of *user interface elements* we can interact with." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "[Selenium](https://www.seleniumhq.org) is a framework for testing Web applications by _automating interaction in the browser_. Selenium provides an API that allows one to launch a Web browser, query the state of the user interface, and interact with individual user interface elements. The Selenium API is available in a number of languages; we use the [Selenium API for Python](https://selenium-python.readthedocs.io/index.html)." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "A Selenium *web driver* is the interface between a program and a browser controlled by the program." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from selenium import webdriver" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "The following code starts a Firefox browser in the background, which we then control through the web driver." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "BROWSER = 'firefox'" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "If you prefer a Chrome browser, uncomment the following line:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# Uncomment for using Chrome instead\n", "# BROWSER = 'chrome'" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "The browser is _headless_, meaning that it does not show on the screen." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "HEADLESS = True" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "**Note**: If the notebook server runs locally (i.e. on the same machine on which you are seeing this), you can also set `HEADLESS` to `False` and see what happens right on the screen as you execute the notebook cells. This is very much recommended for interactive sessions." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# Uncomment for interactive sessions\n", "# HEADLESS = False" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "We pass the `HEADLESS` setting to our browser." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "if BROWSER == 'firefox':\n", " options = webdriver.FirefoxOptions()\n", "if BROWSER == 'chrome':\n", " options = webdriver.ChromeOptions()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "if HEADLESS and BROWSER == 'chrome':\n", " options.add_argument('headless')\n", "else:\n", " options.headless = HEADLESS\n", "options.arguments" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Since we want to take screen shots, we set a minimum resolution." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "profile = webdriver.firefox.firefox_profile.FirefoxProfile()\n", "ZOOM = 1.4\n", "profile.set_preference(\"layout.css.devPixelsPerPx\", repr(ZOOM))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "All is set! We can now start the browser, and obtain a _web driver_ object such that we can interact with it." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "if BROWSER == 'firefox':\n", " gui_driver = webdriver.Firefox(firefox_profile=profile, options=options)\n", "if BROWSER == 'chrome':\n", " gui_driver = webdriver.Chrome(options=options)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "We set the window size such that it fits our order form exactly; this is useful for not wasting too much space when taking screen shots." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "if BROWSER == 'firefox':\n", " gui_driver.set_window_size(700, 300)\n", "if BROWSER == 'chrome':\n", " gui_driver.set_window_size(700, 210 if HEADLESS else 340)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "We can now interact with the browser programmatically. First, we have it navigate to the URL of our Web server:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_driver.get(httpd_url)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "We see that the home page is actually accessed, together with a (failing) request to get a page icon:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "print_httpd_messages()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "To see what the \"headless\" browser displays, we can obtain a screenshot. We see that it actually displays the home page." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "Image(gui_driver.get_screenshot_as_png())" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Filling out Forms\n", "\n", "To interact with the Web page through Selenium and the browser, we can _query_ Selenium for individual elements. For instance, we can access the UI element whose `name` attribute (as defined in HTML) is `\"name\"`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "name = gui_driver.find_element_by_name(\"name\")" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Once we have an element, we can interact with it. Since `name` is a text field, we can send it a string using the `send_keys()` method; the string will be translated into appropriate key strokes." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "name.send_keys(\"Jane Doe\")" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "In the screenshot, we can see that the `name` field is now filled:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "Image(gui_driver.get_screenshot_as_png())" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "In a similar fashion, we can fill out the email, city, and ZIP fields:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "email = gui_driver.find_element_by_name(\"email\")\n", "email.send_keys(\"j.doe@example.com\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "city = gui_driver.find_element_by_name('city')\n", "city.send_keys(\"Seattle\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "zip = gui_driver.find_element_by_name('zip')\n", "zip.send_keys(\"98104\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "Image(gui_driver.get_screenshot_as_png())" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "The check box for terms and conditions is not filled out, but clicked instead using the `click()` method." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "terms = gui_driver.find_element_by_name('terms')\n", "terms.click()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "Image(gui_driver.get_screenshot_as_png())" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "The form is now fully filled out. By clicking on the `submit` button, we can place the order:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "submit = gui_driver.find_element_by_name('submit')\n", "submit.click()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "We see that the order is being processed, and that the Web browser has switched to the confirmation page." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "print_httpd_messages()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "Image(gui_driver.get_screenshot_as_png())" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Navigating\n", "\n", "Just as we fill out forms, we can also navigate through a Web site by clicking on links. Let us go back to the home page:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_driver.back()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "Image(gui_driver.get_screenshot_as_png())" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "We can query the web driver for all elements of a particular type. Querying for HTML anchor elements (``) for instance, gives us all links on a page." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "button": false, "new_sheet": false, "run_control": { "read_only": false }, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "links = gui_driver.find_elements_by_tag_name(\"a\")" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "We can query the attributes of UI elements – for instance, the URL the first anchor on the page links to:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "links[0].get_attribute('href')" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "What happens if we click on it? Very simple: We switch to the Web page being referenced." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "links[0].click()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "print_httpd_messages()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "Image(gui_driver.get_screenshot_as_png())" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Okay. Let's get back to our home page again." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_driver.back()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "print_httpd_messages()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "Image(gui_driver.get_screenshot_as_png())" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Writing Test Cases\n", "\n", "The above calls, interacting with a user interface automatically, are typically used in *Selenium tests* – that is, code snippets that interact with a Web site, occasionally checking whether everything works as expected. The following code, for instance, places an order just as above. It then retrieves the `title` element and checks whether the title contains a \"Thank you\" message, indicating success." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "def test_successful_order(driver, url):\n", " name = \"Walter White\"\n", " email = \"white@jpwynne.edu\"\n", " city = \"Albuquerque\"\n", " zip_code = \"87101\"\n", " \n", " driver.get(url)\n", " driver.find_element_by_name(\"name\").send_keys(name)\n", " driver.find_element_by_name(\"email\").send_keys(email)\n", " driver.find_element_by_name('city').send_keys(city)\n", " driver.find_element_by_name('zip').send_keys(zip_code)\n", " driver.find_element_by_name('terms').click()\n", " driver.find_element_by_name('submit').click()\n", " \n", " title = driver.find_element_by_id('title')\n", " assert title is not None\n", " assert title.text.find(\"Thank you\") >= 0\n", "\n", " confirmation = driver.find_element_by_id(\"confirmation\")\n", " assert confirmation is not None\n", "\n", " assert confirmation.text.find(name) >= 0\n", " assert confirmation.text.find(email) >= 0\n", " assert confirmation.text.find(city) >= 0\n", " assert confirmation.text.find(zip_code) >= 0\n", " \n", " return True" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "test_successful_order(gui_driver, httpd_url)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "In a similar vein, we can set up automated test cases for unsuccessful orders, canceling orders, changing orders, and many more. All these test cases would be automatically run after any change to the program code, ensuring the Web application still works." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Of course, writing such tests is quite some effort. Hence, in the remainder of this chapter, we will again explore how to automatically generate them." ] }, { "cell_type": "markdown", "metadata": { "button": false, "new_sheet": false, "run_control": { "read_only": false }, "slideshow": { "slide_type": "subslide" } }, "source": [ "## Retrieving User Interface Actions\n", "\n", "To automatically interact with a user interface, we first need to find out which elements there are, and which user interactions (or short *actions*) they support." ] }, { "cell_type": "markdown", "metadata": { "button": false, "new_sheet": false, "run_control": { "read_only": false }, "slideshow": { "slide_type": "subslide" } }, "source": [ "### User Interface Elements\n", "\n", "We start with finding available user elements. Let us get back to the order form." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_driver.get(httpd_url)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "Image(gui_driver.get_screenshot_as_png())" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Using `find_elements_by_tag_name()` (and other similar `find_elements_...()` functions), we can retrieve all elements of a particular type, such as HTML `input` elements." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "button": false, "new_sheet": false, "run_control": { "read_only": false }, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "ui_elements = gui_driver.find_elements_by_tag_name(\"input\")" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "For each element, we can retrieve its HTML attributes, using `get_attribute()`. We can thus retrieve the `name` and `type` of each input element (if defined)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "for element in ui_elements:\n", " print(\"Name: %-10s | Type: %-10s | Text: %s\" % (element.get_attribute('name'), element.get_attribute('type'), element.text))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "button": false, "new_sheet": false, "run_control": { "read_only": false }, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "ui_elements = gui_driver.find_elements_by_tag_name(\"a\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "for element in ui_elements:\n", " print(\"Name: %-10s | Type: %-10s | Text: %s\" % (element.get_attribute('name'), element.get_attribute('type'), element.text))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### User Interface Actions\n", "\n", "Similarly to what we did in the [chapter on Web fuzzing](WebFuzzer.ipynb), our idea is now to mine a _grammar_ for the user interface – first for an individual user interface *page* (i.e., a single Web page), later for all pages offered by the application. The idea is that a grammar defines _legal sequences of actions_ – clicks and keystrokes – that can be applied on the application." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "We assume the following actions:\n", "\n", "1. `fill(, )` – fill the UI input element named `` with the text ``.\n", "1. `check(, )` – set the UI checkbox `` to the given value `` (True or False)\n", "1. `submit()` – submit the form by clicking on the UI element ``.\n", "1. `click()` – click on the UI element ``, typically for following a link." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "This sequence of actions, for instance would fill out the order form:\n", "\n", "```python\n", "fill('name', \"Walter White\")\n", "fill('email', \"white@jpwynne.edu\")\n", "fill('city', \"Albuquerque\")\n", "fill('zip', \"87101\")\n", "check('terms', True)\n", "submit('submit')\n", "```" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Our set of actions is deliberately defined to be small – for real user interfaces, one would also have to define interactions such as swipes, double clicks, long clicks, right button clicks, modifier keys, and more. Selenium supports all of this; but in the interest of simplicity, we focus on the most important set of interactions." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" }, "toc-hr-collapsed": false }, "source": [ "### Retrieving Actions\n", "\n", "As a first step in mining an action grammar, we need to be able to retrieve possible interactions. We introduce a class `GUIGrammarMiner`, which is set to do precisely that." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "class GUIGrammarMiner(object):\n", " def __init__(self, driver, stay_on_host=True):\n", " self.driver = driver\n", " self.stay_on_host = stay_on_host\n", " self.grammar = {}" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Our first task is to obtain the set of possible interactions. Given a single UI page, the method `mine_input_actions()` of `GUIGrammarMiner` returns a set of *actions* as defined above. It first gets all `input` elements, followed by `button` elements, finally followed by links (`a` elements), and merges them into a set. (We use a `frozenset` here since we want to use the set as an index later.)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "class GUIGrammarMiner(GUIGrammarMiner):\n", " def mine_state_actions(self):\n", " return frozenset(self.mine_input_element_actions()\n", " | self.mine_button_element_actions()\n", " | self.mine_a_element_actions())" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Input Element Actions\n", "\n", "Mining input actions goes through the set of input elements, and returns an action depending on the input type. If the input field is a text, for instance, the associated action is `fill()`; for checkboxes, the action is `check()`." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "The respective values are placeholders depending on the type; if the input field is a number, for instance, the value becomes ``. As these actions later become part of the grammar, they will be expanded into actual values during grammar expansion." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from selenium.common.exceptions import StaleElementReferenceException" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class GUIGrammarMiner(GUIGrammarMiner):\n", " def mine_input_element_actions(self):\n", " actions = set()\n", " \n", " for elem in self.driver.find_elements_by_tag_name(\"input\"):\n", " try:\n", " input_type = elem.get_attribute(\"type\")\n", " input_name = elem.get_attribute(\"name\")\n", " if input_name is None:\n", " input_name = elem.text\n", "\n", " if input_type in [\"checkbox\", \"radio\"]:\n", " actions.add(\"check('%s', )\" % html.escape(input_name))\n", " elif input_type in [\"text\", \"number\", \"email\", \"password\"]:\n", " actions.add(\"fill('%s', '<%s>')\" % (html.escape(input_name), html.escape(input_type)))\n", " elif input_type in [\"button\", \"submit\"]:\n", " actions.add(\"submit('%s')\" % html.escape(input_name))\n", " elif input_type in [\"hidden\"]:\n", " pass\n", " else:\n", " # TODO: Handle more types here\n", " actions.add(\"fill('%s', <%s>)\" % (html.escape(input_name), html.escape(input_type)))\n", " except StaleElementReferenceException:\n", " pass\n", "\n", " return actions" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Applied on our order form, we see that the method gets us all input actions:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_grammar_miner = GUIGrammarMiner(gui_driver)\n", "gui_grammar_miner.mine_input_element_actions()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Button Element Actions\n", "\n", "Mining buttons works in a similar way:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class GUIGrammarMiner(GUIGrammarMiner):\n", " def mine_button_element_actions(self):\n", " actions = set()\n", " \n", " for elem in self.driver.find_elements_by_tag_name(\"button\"):\n", " try:\n", " button_type = elem.get_attribute(\"type\")\n", " button_name = elem.get_attribute(\"name\")\n", " if button_name is None:\n", " button_name = elem.text\n", " if button_type == \"submit\":\n", " actions.add(\"submit('%s')\" % html.escape(button_name))\n", " elif button_type != \"reset\":\n", " actions.add(\"click('%s')\" % html.escape(button_name))\n", " except StaleElementReferenceException:\n", " pass\n", "\n", " return actions" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Our order form has no `button` elements. (The `submit` button is an `input` element, and was handled above)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_grammar_miner = GUIGrammarMiner(gui_driver)\n", "gui_grammar_miner.mine_button_element_actions()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Link Element Actions\n", "\n", "When following links, we need to make sure that we stay on the current host – we want to explore a single Web site only, not all of the Internet. To this end, we check the `href` attribute of the link to check whether it still points to the same host. If it does not, we give it a special action `ignore()`, which, as the name suggests, will later be ignored as it comes to executing these actions. We still return an action, though, as we use the set of actions to characterize a state in the application." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class GUIGrammarMiner(GUIGrammarMiner):\n", " def mine_a_element_actions(self):\n", " actions = set()\n", "\n", " for elem in self.driver.find_elements_by_tag_name(\"a\"):\n", " try:\n", " a_href = elem.get_attribute(\"href\")\n", " if a_href is not None:\n", " if self.follow_link(a_href):\n", " actions.add(\"click('%s')\" % html.escape(elem.text))\n", " else:\n", " actions.add(\"ignore('%s')\" % html.escape(elem.text))\n", " except StaleElementReferenceException:\n", " pass\n", "\n", " return actions" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "To check whether we can follow a link, the method `follow_link()` checks the URL:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from urllib.parse import urljoin, urlsplit" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "class GUIGrammarMiner(GUIGrammarMiner):\n", " def follow_link(self, link):\n", " if not self.stay_on_host:\n", " return True\n", " \n", " current_url = self.driver.current_url\n", " target_url = urljoin(current_url, link)\n", " return urlsplit(current_url).hostname == urlsplit(target_url).hostname " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "In our application, we would not be allowed to follow a link to `foo.bar`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_grammar_miner = GUIGrammarMiner(gui_driver)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_grammar_miner.follow_link(\"ftp://foo.bar/\")" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Following a link to `localhost`, though, works well:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_grammar_miner.follow_link(\"https://127.0.0.1/\")" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "When adapting this for other user interfaces, similar measures would be taken to ensure we stay in the same application." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Running this method on our page gets us the set of links:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_grammar_miner = GUIGrammarMiner(gui_driver)\n", "gui_grammar_miner.mine_a_element_actions()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### All Together\n", "\n", "Let us now apply `mine_state_actions()` on our current page to retrieve all elements. We see that we get the union of all three sets." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_grammar_miner = GUIGrammarMiner(gui_driver)\n", "gui_grammar_miner.mine_state_actions()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "We assume that we can identify a user interface *state* from the set of interactive elements it contains – that is, the current Web page is identified by the set above. This is in contrast to [Web fuzzing](WebFuzzer.ipynb), where we assumed the URL to uniquely characterize a page – but with JavaScript, the URL can stay unchanged although the page contents change, and UIs other than the Web may have no concept of unique URLs. Therefore, we say that the way a UI can be interacted with uniquely defines its state." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Models for User Interfaces" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### User Interfaces as Finite State Machines\n", "\n", "Now that we can retrieve UI elements from a page, let us go and systematically explore a user interface. The idea is to represent the user interface as a *finite state machine* – that is, a sequence of *states* that can be reached by interacting with the individual user interface elements." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Let us illustrate such a finite state machine by looking our Web server. The following diagram shows the states our server can be in:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from graphviz import Digraph" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from IPython.display import display" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from GrammarFuzzer import dot_escape" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "dot = Digraph(comment=\"Finite State Machine\")\n", "dot.node(dot_escape(''))\n", "dot.edge(dot_escape(''), dot_escape(''))\n", "dot.edge(dot_escape(''), dot_escape(''), \"click('Terms and conditions')\")\n", "dot.edge(dot_escape(''), dot_escape(''), \"fill(...)\\lsubmit('submit')\")\n", "dot.edge(dot_escape(''), dot_escape(''), \"click('order form')\")\n", "dot.edge(dot_escape(''), dot_escape(''), \"click('order form')\")\n", "display(dot)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Initially, we are in the `` state. From here, we can click on `Terms and Conditions`, and we'll be in the `Terms and Conditions` state, showing the page with the same title. We can also fill out the form and place the order, having us end in the `Thank You` state (again showing the page with the same title). From both `` and ``, we can return to the order form by clicking on the `order form` link." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### State Machines as Grammars\n", "\n", "To systematically explore a user interface, we must retrieve its finite state machine, and eventually cover all states and transitions. In the presence of forms, such an exploration is difficult, as we need a special mechanism to fill out forms and submit the values to get to the next state. There is a trick, though, which allows us to have a single representation for both states and (form) values. We can *embed the finite state machine into a grammar*, which is then used for both states and form values." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "To embed a finite state machine into a grammar, we proceed as follows:\n", "\n", "1. Every _state_ $\\langle s \\rangle$ in the finite state machine becomes a _symbol_ $\\langle s \\rangle$ in the grammar.\n", "2. Every _transition_ in the finite state machine from $\\langle s \\rangle$ to $\\langle t \\rangle$ and actions $a_1, a_2, \\dots$ becomes an _alternative_ of $\\langle s \\rangle$ in the form $a_1, a_2, dots$ $\\langle t \\rangle$ in the grammar." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "The above finite state machine thus gets encoded into the grammar\n", "\n", "```\n", " ::= \n", " ::= click('Terms and Conditions') | \n", " fill(...) submit('submit') \n", " ::= click('order form') \n", " ::= click('order form') \n", "```" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Expanding this grammar gets us a stream of actions, navigating through the user interface:\n", "\n", "```\n", "fill(...) submit('submit') click('order form') click('Terms and Conditions') click('order form') ...\n", "```\n", "\n", "This stream is actually _infinite_ (as one can interact with the UI forever); to have it end, one can introduce an alternative `` that simply expands to the empty string, without having any expansion (state) follow." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Retrieving State Grammars\n", "\n", "Let us build a method that retrieves a grammar from the _current state_ of a user interface. We first define a constant `GUI_GRAMMAR` that servers as template for all sorts of input types. We will use this to fill out forms." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "\\todo{}: Have a generic interface `BaseGrammarMiner` with `__init__()` and `mine_grammar()`" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from Grammars import new_symbol" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from Grammars import nonterminals, START_SYMBOL\n", "from Grammars import extend_grammar, unreachable_nonterminals, opts, crange, srange\n", "from Grammars import syntax_diagram, is_valid_grammar" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class GUIGrammarMiner(GUIGrammarMiner):\n", " START_STATE = \"\"\n", " UNEXPLORED_STATE = \"\"\n", " FINAL_STATE = \"\"\n", "\n", " GUI_GRAMMAR = ({\n", " START_SYMBOL: [START_STATE],\n", " UNEXPLORED_STATE: [\"\"],\n", " FINAL_STATE: [\"\"],\n", "\n", " \"\": [\"\"],\n", " \"\": [\"\", \"\"],\n", " \"\": [\"\", \"\", \"\"],\n", " \"\": crange('a', 'z') + crange('A', 'Z'),\n", " \n", " \"\": [\"\"],\n", " \"\": [\"\", \"\"],\n", " \"\": crange('0', '9'),\n", " \n", " \"\": srange(\". !\"),\n", "\n", " \"\": [\"@\"],\n", " \"\": [\"\", \"\"],\n", " \n", " \"\": [\"True\", \"False\"],\n", "\n", " # Use a fixed password in case we need to repeat it\n", " \"\": [\"abcABC.123\"],\n", " \n", " \"\": \"\",\n", " })" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "syntax_diagram(GUIGrammarMiner.GUI_GRAMMAR)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "The method `mine_state_grammar()` goes through the actions mined from the page (using `mine_state_actions()`) and creates a grammar for the current state. For each `click()` and `submit()` action, it assumes a new state follows, and introduces an appropriate state symbol into the grammar – a state symbol that now will be marked as ``, but will be expanded later as the appropriate state is seen." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class GUIGrammarMiner(GUIGrammarMiner):\n", " def new_state_symbol(self, grammar):\n", " return new_symbol(grammar, self.START_STATE)\n", "\n", " def mine_state_grammar(self, grammar={}, state_symbol=None):\n", " grammar = extend_grammar(self.GUI_GRAMMAR, grammar)\n", "\n", " if state_symbol is None:\n", " state_symbol = self.new_state_symbol(grammar)\n", " grammar[state_symbol] = []\n", "\n", " alternatives = []\n", " form = \"\"\n", " submit = None\n", "\n", " for action in self.mine_state_actions():\n", " if action.startswith(\"submit\"):\n", " submit = action\n", " \n", " elif action.startswith(\"click\"):\n", " link_target = self.new_state_symbol(grammar)\n", " grammar[link_target] = [self.UNEXPLORED_STATE]\n", " alternatives.append(action + '\\n' + link_target)\n", " \n", " elif action.startswith(\"ignore\"):\n", " pass\n", "\n", " else: # fill(), check() actions\n", " if len(form) > 0:\n", " form += '\\n'\n", " form += action\n", "\n", " if submit is not None:\n", " if len(form) > 0:\n", " form += '\\n'\n", " form += submit\n", "\n", " if len(form) > 0:\n", " form_target = self.new_state_symbol(grammar)\n", " grammar[form_target] = [self.UNEXPLORED_STATE]\n", " alternatives.append(form + '\\n' + form_target)\n", " \n", " alternatives += [self.FINAL_STATE]\n", "\n", " grammar[state_symbol] = alternatives\n", " \n", " # Remove unused parts\n", " for nonterminal in unreachable_nonterminals(grammar):\n", " del grammar[nonterminal]\n", "\n", " assert is_valid_grammar(grammar)\n", " \n", " return grammar" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Let us show `mine_state_grammar()` in action. Here's the grammar for the home page:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_grammar_miner = GUIGrammarMiner(gui_driver)\n", "state_grammar = gui_grammar_miner.mine_state_grammar()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "From the start state (``), we can go and either click on \"terms and conditions\", ending in ``, or fill out the form, ending in ``." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "state_grammar[GUIGrammarMiner.START_STATE]" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Both these states are yet unexplored:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "state_grammar['']" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "state_grammar['']" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "state_grammar['']" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "To better see the state structure, the function `fsm_diagram()` shows the resulting state grammar as a finite state machine. (This assumes that the grammar actually encodes a state machine.)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from collections import deque" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from fuzzingbook_utils import unicode_escape" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "def fsm_diagram(grammar, start_symbol=START_SYMBOL):\n", " def left_align(label):\n", " return dot_escape(label.replace('\\n', r'\\l')).replace(r'\\\\l', '\\\\l')\n", "\n", " dot = Digraph(comment=\"Grammar as Finite State Machine\")\n", "\n", " symbols = deque([start_symbol])\n", " symbols_seen = set()\n", " \n", " while len(symbols) > 0:\n", " symbol = symbols.popleft()\n", " symbols_seen.add(symbol)\n", " dot.node(symbol, dot_escape(unicode_escape(symbol)))\n", " \n", " for expansion in grammar[symbol]:\n", " nts = nonterminals(expansion)\n", " if len(nts) > 0:\n", " target_symbol = nts[-1]\n", " if target_symbol not in symbols_seen:\n", " symbols.append(target_symbol)\n", "\n", " label = expansion.replace(target_symbol, '')\n", " dot.edge(symbol, target_symbol, left_align(unicode_escape(label)))\n", " \n", " \n", " return display(dot)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Here's our current grammar as a state machine. We see that it nicely reflects what we can see from our Web server's home page:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "fsm_diagram(state_grammar)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Given the grammar, we can use any of our grammar fuzzers to create valid input sequences:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from GrammarFuzzer import GrammarFuzzer" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "gui_fuzzer = GrammarFuzzer(state_grammar)\n", "while True:\n", " action = gui_fuzzer.fuzz()\n", " if action.find('submit(') > 0:\n", " break\n", "print(action)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "These actions, however, must also be _executed_ such that we can explore the user interface. This is what we do in the next section." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Executing User Interface Actions" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "To execute actions, we introduce a `Runner` class, conveniently named `GUIRunner`. The aim of its `run()` method is to execute the actions as given in an action string. The way we do this is fairly simple: We introduce four methods named `fill()`, `check()`, `submit()` and `click()`, and run `exec()` on the action string to have the Python interpreter invoke these methods." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Running `exec()` on third-party input is dangerous, as the names of UI elements may contain valid Python code. We restrict access to the four functions defined above, and also set `__builtins__` to the empty dictionary such that built-in Python functions are not available during `exec()`. This will prevent accidents, but as we will see in the [chapter on information flow](InformationFlow.ipynb), it is still possible to inject Python code. To prevent such injection attacks, we use `html.escape()` to quote angle and quote characters in all third-party strings." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from Fuzzer import Runner" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class GUIRunner(Runner):\n", " def __init__(self, driver):\n", " self.driver = driver\n", " \n", " def run(self, inp):\n", " def fill(name, value):\n", " self.do_fill(html.unescape(name), html.unescape(value))\n", " def check(name, state):\n", " self.do_check(html.unescape(name), state)\n", " def submit(name):\n", " self.do_submit(html.unescape(name))\n", " def click(name):\n", " self.do_click(html.unescape(name))\n", " \n", " exec(inp, {'__builtins__': {}},\n", " {'fill': fill, 'check': check, 'submit': submit, 'click': click})\n", " \n", " return inp, self.PASS" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "To identify elements in an action, we first search them by their name, and then by the displayed link text." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from selenium.common.exceptions import NoSuchElementException\n", "from selenium.common.exceptions import ElementClickInterceptedException, ElementNotInteractableException" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "class GUIRunner(GUIRunner):\n", " def find_element(self, name):\n", " try:\n", " return self.driver.find_element_by_name(name)\n", " except NoSuchElementException:\n", " return self.driver.find_element_by_link_text(name)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "The implementations of the actions simply defer to the appropriate Selenium methods, introducing explicit delays such that the page can reload and refresh." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from selenium.webdriver.support.ui import WebDriverWait" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class GUIRunner(GUIRunner):\n", " # Delays (in seconds)\n", " DELAY_AFTER_FILL = 0.1\n", " DELAY_AFTER_CHECK = 0.1\n", " DELAY_AFTER_SUBMIT = 1\n", " DELAY_AFTER_CLICK = 1" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "class GUIRunner(GUIRunner):\n", " def do_fill(self, name, value):\n", " element = self.find_element(name)\n", " element.send_keys(value)\n", " WebDriverWait(self.driver, self.DELAY_AFTER_FILL)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class GUIRunner(GUIRunner):\n", " def do_check(self, name, state):\n", " element = self.find_element(name)\n", " if bool(state) != bool(element.is_selected()):\n", " element.click()\n", " WebDriverWait(self.driver, self.DELAY_AFTER_CHECK)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "class GUIRunner(GUIRunner):\n", " def do_submit(self, name):\n", " element = self.find_element(name)\n", " element.click()\n", " WebDriverWait(self.driver, self.DELAY_AFTER_SUBMIT)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class GUIRunner(GUIRunner):\n", " def do_click(self, name):\n", " element = self.find_element(name)\n", " element.click()\n", " WebDriverWait(self.driver, self.DELAY_AFTER_CLICK)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Let us try out `GUIRunner`. We create a runner on our Web server, and let it execute a `fill()` action:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_driver.get(httpd_url)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_runner = GUIRunner(gui_driver)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_runner.run(\"fill('name', 'Walter White')\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "Image(gui_driver.get_screenshot_as_png())" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "A `submit()` action submits the order. (Note that our Web server does no effort whatsoever to validate the form.)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "gui_runner.run(\"submit('submit')\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "Image(gui_driver.get_screenshot_as_png())" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Of course, we can also execute action sequences generated from the grammar. This allows us to fill the form again and again, using values matching the type given in the form." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_driver.get(httpd_url)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_fuzzer = GrammarFuzzer(state_grammar)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "while True:\n", " action = gui_fuzzer.fuzz()\n", " if action.find('submit(') > 0:\n", " break" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "print(action)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_runner.run(action)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "Image(gui_driver.get_screenshot_as_png())" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Exploring User Interfaces\n", "\n", "So far, our grammar retrieval and execution of actions is limited to the current user interface state (i.e., the current page shown). To systematically explore a user interface, we must explore all states, notably those ending in `` – and whenever we reach a new state, again retrieve its grammar such that we may be able to reach other states. Since some states can only be reached by generating inputs, test generation and user interface exploration _take place at the same time._ \n", "\n", "Consequently, we introduce a `GUIFuzzer` class, which generates inputs for all forms and follows all links, and which updates its grammar (i.e., its user interface model as a finite state machine) every time it encounters a new state. " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Exploring states and updating the grammar at the same time is a fairly complex operation, so we need to introduce quite a number of methods before we can put this to use. The `GUIFuzzer` constructor sets three important attributes:\n", "\n", "1. `state_symbol`: This holds the symbol of the current state (e.g. ``).\n", "2. `state`: This holds the set of actions for the current state, as returned by the `GUIGrammarMiner` method `mine_state_actions()`.\n", "3. `states_seen`: This maps the states seen (as in `state`) to the respective symbols.\n", "\n", "Let us show these three attributes after initialization." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from Grammars import is_nonterminal" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from GrammarFuzzer import GrammarFuzzer" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class GUIFuzzer(GrammarFuzzer):\n", " def __init__(self, driver, \n", " stay_on_host=True,\n", " log_gui_exploration=False, \n", " disp_gui_exploration=False, \n", " **kwargs):\n", " self.driver = driver\n", " self.miner = GUIGrammarMiner(driver)\n", " self.stay_on_host = True\n", " self.log_gui_exploration = log_gui_exploration\n", " self.disp_gui_exploration = disp_gui_exploration\n", " self.initial_url = driver.current_url\n", "\n", " self.states_seen = {} # Maps states to symbols\n", " self.state_symbol = GUIGrammarMiner.START_STATE\n", " self.state = self.miner.mine_state_actions()\n", " self.states_seen[self.state] = self.state_symbol\n", " \n", " grammar = self.miner.mine_state_grammar()\n", " super().__init__(grammar, **kwargs)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "gui_driver.get(httpd_url)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "The initial state symbol is always ``:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_fuzzer = GUIFuzzer(gui_driver)\n", "gui_fuzzer.state_symbol" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "The current state is characterized by the available UI actions:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_fuzzer.state" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "`states_seen` maps this state to its symbol:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "gui_fuzzer.states_seen[gui_fuzzer.state]" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "The `restart()` method gets us back to the initial URL and resets the state. This is what we use with every new exploration." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "class GUIFuzzer(GUIFuzzer):\n", " def restart(self):\n", " self.driver.get(self.initial_url)\n", " self.state = GUIGrammarMiner.START_STATE" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "When producing a sequence of actions from the grammar, we want to know which final state we are to be in. We can retrieve this path from the _derivation tree_ produced – it is the last symbol being expanded." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "while True:\n", " action = gui_fuzzer.fuzz()\n", " if action.find('click(') >= 0:\n", " break" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from GrammarFuzzer import display_tree" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "tree = gui_fuzzer.derivation_tree\n", "display_tree(tree)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "class GUIFuzzer(GUIFuzzer):\n", " def fsm_path(self, tree):\n", " \"\"\"Return sequence of state symbols\"\"\"\n", " (node, children) = tree\n", " if node == GUIGrammarMiner.UNEXPLORED_STATE:\n", " return []\n", " elif children is None or len(children) == 0:\n", " return [node]\n", " else:\n", " return [node] + self.fsm_path(children[-1])" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "This is the path in the finite state machine towards the \"fuzzed\" state:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "gui_fuzzer = GUIFuzzer(gui_driver)\n", "gui_fuzzer.fsm_path(tree)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "This is its last element:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "class GUIFuzzer(GUIFuzzer):\n", " def fsm_last_state_symbol(self, tree):\n", " \"\"\"Return current (expected) state symbol\"\"\"\n", " for state in reversed(self.fsm_path(tree)):\n", " if is_nonterminal(state):\n", " return state\n", " assert False" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_fuzzer = GUIFuzzer(gui_driver)\n", "gui_fuzzer.fsm_last_state_symbol(tree)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "As we run (`run()`) the fuzzer, we create an action (via `fuzz()`) and retrieve and update the state symbol (`state_symbol`) we are supposed to be in after running this action. After actually running the action in the given `GUIRunner`, we retrieve and update the current state, using `update_state()`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class GUIFuzzer(GUIFuzzer):\n", " def run(self, runner):\n", " assert isinstance(runner, GUIRunner)\n", " \n", " self.restart()\n", " action = self.fuzz()\n", " self.state_symbol = self.fsm_last_state_symbol(self.derivation_tree)\n", "\n", " if self.log_gui_exploration:\n", " print(\"Action\", action.strip(), \"->\", self.state_symbol)\n", "\n", " result, outcome = runner.run(action)\n", " \n", " if self.state_symbol != GUIGrammarMiner.FINAL_STATE:\n", " self.update_state()\n", "\n", " return self.state_symbol, outcome" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "When updating the current state, we check whether we are in a new or in a previously seen state, and invoke `update_new_state()` or `update_existing_state()`, respectively." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "class GUIFuzzer(GUIFuzzer):\n", " def update_state(self):\n", " if self.disp_gui_exploration:\n", " display(Image(self.driver.get_screenshot_as_png()))\n", "\n", " self.state = self.miner.mine_state_actions()\n", " if self.state not in self.states_seen:\n", " self.states_seen[self.state] = self.state_symbol\n", " self.update_new_state()\n", " else:\n", " self.update_existing_state()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Finding a new state means that we mine a new grammar for the newly found state, and update our existing grammar with it." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "class GUIFuzzer(GUIFuzzer):\n", " def set_grammar(self, new_grammar):\n", " self.grammar = new_grammar\n", " \n", " if self.disp_gui_exploration and rich_output():\n", " display(fsm_diagram(self.grammar))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class GUIFuzzer(GUIFuzzer):\n", " def update_new_state(self):\n", " if self.log_gui_exploration:\n", " print(\"In new state\", unicode_escape(self.state_symbol), unicode_escape(repr(self.state)))\n", "\n", " state_grammar = self.miner.mine_state_grammar(grammar=self.grammar, \n", " state_symbol=self.state_symbol)\n", " del state_grammar[START_SYMBOL]\n", " del state_grammar[GUIGrammarMiner.START_STATE]\n", " self.set_grammar(extend_grammar(self.grammar, state_grammar))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "If we find an existing state, we need to _merge_ both states. If, for instance, we find that we are in existing `` rather than in the expected ``, we replace all instances of `` in the grammar by ``. The method `replace_symbol()` takes care of the renaming; `update_existing_state()` sets the grammar accordingly." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from Grammars import exp_string, exp_opts" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "def replace_symbol(grammar, old_symbol, new_symbol):\n", " \"\"\"Return a grammar in which all occurrences of `old_symbol` are replaced by `new_symbol`\"\"\"\n", " new_grammar = {}\n", " \n", " for symbol in grammar:\n", " new_expansions = []\n", " for expansion in grammar[symbol]:\n", " new_expansion_string = exp_string(expansion).replace(old_symbol, new_symbol)\n", " if len(exp_opts(expansion)) > 0:\n", " new_expansion = (new_expansion_string, exp_opts(expansion))\n", " else:\n", " new_expansion = new_expansion_string\n", " new_expansions.append(new_expansion)\n", " \n", " new_grammar[symbol] = new_expansions\n", " \n", " # Remove unused parts\n", " for nonterminal in unreachable_nonterminals(new_grammar):\n", " del new_grammar[nonterminal]\n", "\n", " return new_grammar" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class GUIFuzzer(GUIFuzzer): \n", " def update_existing_state(self):\n", " if self.log_gui_exploration:\n", " print(\"In existing state\", self.states_seen[self.state])\n", "\n", " if self.state_symbol != self.states_seen[self.state]:\n", " if self.log_gui_exploration:\n", " print(\"Replacing expected state %s by %s\" %\n", " (self.state_symbol, self.states_seen[self.state]))\n", " \n", " new_grammar = replace_symbol(self.grammar, self.state_symbol, \n", " self.states_seen[self.state])\n", " self.state_symbol = self.states_seen[self.state]\n", " self.set_grammar(new_grammar)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "This concludes our definitions for `GUIFuzzer`. We can now put it to use, enabling its logging mechanisms to see what it is doing." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_driver.get(httpd_url)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_fuzzer = GUIFuzzer(gui_driver, log_gui_exploration=True, disp_gui_exploration=True)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Running it the first time yields a new state:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "gui_fuzzer.run(gui_runner)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "The next actions fill out the order form." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "gui_fuzzer.run(gui_runner)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "gui_fuzzer.run(gui_runner)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "At this point, our GUI model is fairly complete already. In order to systematically cover _all_ states, random exploration is not efficient enough, though." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Covering States\n", "\n", "During exploration as well as during testing, we want to _cover_ all states and transitions between states. How can we achieve this?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "It turns out that _we already have this._ Our `GrammarCoverageFuzzer` from the [chapter on coverage-based grammar testing](GrammarCoverageFuzzer.ipynb) strives to systematically _cover all expansion alternatives_ in a grammar. In the finite state model, these expansion alternatives translate into transitions between states. Hence, applying the coverage strategy from `GrammarCoverageFuzzer` to our state grammars would automatically cover one transition after another." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "How do we get these features into `GUIFuzzer`? Using _multiple inheritance_, we can create a class `GUICoverageFuzzer` which combines the `run()` method from `GUIFuzzer` with the coverage choices from `GrammarCoverageFuzzer`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from GrammarCoverageFuzzer import GrammarCoverageFuzzer" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from fuzzingbook_utils import inheritance_conflicts" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Since the `__init__()` constructor is defined in both superclasses, we need to define our own constructor that serves both:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "inheritance_conflicts(GUIFuzzer, GrammarCoverageFuzzer)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "class GUICoverageFuzzer(GUIFuzzer, GrammarCoverageFuzzer):\n", " def __init__(self, *args, **kwargs):\n", " GUIFuzzer.__init__(self, *args, **kwargs)\n", " self.reset_coverage()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "With `GUICoverageFuzzer`, we can set up a method `explore_all()` that keeps on running the fuzzer until there are no unexplored states anymore:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class GUICoverageFuzzer(GUICoverageFuzzer): \n", " def explore_all(self, runner, max_actions=100):\n", " actions = 0\n", " while GUIGrammarMiner.UNEXPLORED_STATE in self.grammar and actions < max_actions:\n", " actions += 1\n", " if self.log_gui_exploration:\n", " print(\"Run #\" + repr(actions))\n", " try:\n", " self.run(runner)\n", " except ElementClickInterceptedException:\n", " pass\n", " except ElementNotInteractableException:\n", " pass\n", " except NoSuchElementException:\n", " pass" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Let us use this to fully explore our Web server:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_driver.get(httpd_url)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_fuzzer = GUICoverageFuzzer(gui_driver)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_fuzzer.explore_all(gui_runner)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Success! We have covered all states:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "fsm_diagram(gui_fuzzer.grammar)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "We can retrieve the expansions covered so far, which of course cover all states." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "gui_fuzzer.covered_expansions" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Still, we haven't seen all expansions covered. A few digits and letters remain to be used." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "gui_fuzzer.missing_expansion_coverage()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Running the fuzzer again and again will eventually cover these expansions too, leading to letter and digit coverage within the order form." ] }, { "cell_type": "markdown", "metadata": { "button": false, "new_sheet": false, "run_control": { "read_only": false }, "slideshow": { "slide_type": "slide" } }, "source": [ "## Exploring Large Sites\n", "\n", "Our GUI fuzzer is robust enough to handle exploration even on nontrivial sites such as [fuzzingbook.org](https://www.fuzzingbook.org). Let us demonstrate this:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_driver.get(\"https://www.fuzzingbook.org/\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "Image(gui_driver.get_screenshot_as_png())" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "book_runner = GUIRunner(gui_driver)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "button": false, "new_sheet": false, "run_control": { "read_only": false }, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "book_fuzzer = GUICoverageFuzzer(gui_driver, log_gui_exploration=True) # , disp_gui_exploration=True)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "We explore the first 10 states of the site:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "book_fuzzer.explore_all(book_runner, max_actions=10)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "After the first 10 actions already, we can see that the finite state model is quite complex, with hundreds of transitions still left to explore. Most of the yet unexplored states will eventually merge with existing states, yielding one state per chapter. Still, following _all_ links on _all_ pages will take quite some time." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# Inspect this graph in the notebook to see it in full glory\n", "fsm_diagram(book_fuzzer.grammar)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "We now have all the basic capabilities we need: We can automatically explore large Web sites; we can explore \"deep\" functionality by filling out forms; and we can have our coverage-based fuzzer automatcially focus on yet unexplored states. Still, there is a lot more one can do; the [exercises](#Exercises) will give you some ideas." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "We are done, so we clean up. We shut down our Web server, quit the Web driver (and the associated browser), and finally clean up temporary files left by Selenium." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "httpd_process.terminate()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "gui_driver.quit()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "import os" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "for temp_file in [ORDERS_DB, \"geckodriver.log\", \"ghostdriver.log\"]:\n", " if os.path.exists(temp_file):\n", " os.remove(temp_file)" ] }, { "cell_type": "markdown", "metadata": { "button": false, "new_sheet": true, "run_control": { "read_only": false }, "slideshow": { "slide_type": "slide" } }, "source": [ "## Lessons Learned\n", "\n", "* _Selenium_ is a powerful framework for interacting with user interfaces, especially Web-based user interfaces.\n", "* A _finite state model_ can encode user interface states and transitions.\n", "* Encoding user interface models into a _grammar_ integrates generating text (for forms) and generating user interactions (for navigating)\n", "* To systematically explore a user interface, cover all _state transitions_, which is equivalent to covering all _expansion alternatives_ in the equivalent grammar." ] }, { "cell_type": "markdown", "metadata": { "button": false, "new_sheet": false, "run_control": { "read_only": false }, "slideshow": { "slide_type": "slide" } }, "source": [ "## Next Steps\n", "\n", "From here, you can learn how to\n", "\n", "* [fuzz in the large](FuzzingInTheLarge.ipynb). running a myriad of fuzzers on a same system" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Background\n", "\n", "Automatic testing of graphical user interfaces is a rich field – in research as in practice.\n", "\n", "Coverage criteria for GUIs as well as how to achieve them were first discussed in \\cite{Memon2001}. Memon also introduced the concept of *GUI Ripping* \\cite{Memon2003} – the process in which the software's GUI is automatically traversed by interacting with all its user interface elements.\n", "\n", "The CrawlJax tool \\cite{Mesbah2012} uses dynamic state changes in Web user interfaces to identify candidate elements to interact with. As our approach above, it uses the set of interactable user interface elements as a state in a finite-state model.\n", "\n", "The [Alex framework](https://learnlib.github.io/alex/) uses a similar approach to learn automata for web applications. Starting from a set of test inputs, it produces a mixed-mode behavioral model of the application." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Exercises\n", "\n", "As powerful our GUI fuzzer is at this point, there are still several possibilities left for further optimization and extension. Here are some ideas to get you started. Enjoy user interface fuzzing!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Exercise 1: Stay in Local State\n", "\n", "Rather than having each `run()` start at the very beginning, have the miner start from the current state and explore states reachable from there." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Exercise 2: Going Back\n", "\n", "Make use of the web driver `back()` method and go back to an earlier state, from which we could again start exploration. (Note that a \"back\" functionality may not be available on non-Web user interfaces.)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Exercise 3: Avoiding Bad Form Values\n", "\n", "Detect that some form values are _invalid_, such that the miner does not produce them again." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Exercise 4: Saving Form Values\n", "\n", "Save _successful_ form values, such that the tester does not have to infer them again and again." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Exercise 5: Same Names, Same States\n", "\n", "When the miner finds a link with a name it has already seen, it is likely to lead to a state already seen, too; therefore, one could give its exploration a lower priority." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Exercise 6: Combinatorial Coverage\n", "\n", "Extend the grammar miner such that for every boolean value, there is a separate value to be covered." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Exercise 7: Implicit Delays\n", "\n", "Rather than using _explicit_ (given) delays, use _implicit_ delays and wait for specific elements to appear. these elements could stem from previous explorations of the state." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Exercise 8: Oracles\n", "\n", "Extend the grammar miner such that it also produces _oracles_ – for instance, checking for the presence of specific UI elements." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Exercise 9: More UI Elements\n", "\n", "Run the miner on a Web site of your choice. Find out which other types of user interface elements and actions need to be supported." ] } ], "metadata": { "ipub": { "bibliography": "fuzzingbook.bib", "toc": true }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": true, "title_cell": "", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": true }, "toc-autonumbering": false }, "nbformat": 4, "nbformat_minor": 2 }