name,field id,description,description - details,data type,queryable Name,name,"The 'domain' result, with `www.` removed if at the beginning",This field exists to create a consistent normalization of website names by eliminating the presense or absence of `www.` at the start of the string. ,string,no Initial URL,initial_url,URL ingested to begin a scan,Generated by prepending `https://` to the front of the 'initial_domain' entry.,string,"yes, exact, via path" Initial Domain,initial_domain,Domain of the Initial URL,More specifically a Fully Qualified Domain Name. Often referred to as the hostname. Will not contain a protocol or path. E.g. 'x.y.gov' or 't.mil',string,no Initial Base Domain,initial_base_domain,Base Domain of the Initial URL,The second and top level domains of the `initial_domain` entry. Intended to represent the registered second level domain that unifies one or more websites. E.g. 'n.gov' or 'p.mil'. ,string,"yes, exact" Initial Top Level Domain,initial_top_level_domain,Top Level Domain of the Initial URL,The top level domain of the 'initial_domain' entry. E.g. 'gov' or 'mil'. ,string,no Redirect,redirect,Whether Name is different from the Initial Domain,"Compares the 'initial_domain' and 'name' fields to see if they are the same. Since neither field contains a leading `www.`, this allows for the human friendly if technically generous analysis of whether the initial_url redirected by asking if it resolved to a different domain. This is useful because of how common it is for domains to resolve to homepages by way of a 3xx redirect. ",boolean,"yes, exact" URL,url,Final URL that resolves after the Initial URL has loaded and fully resolved. ,This is the exact URL which is then scanned. ,string,no Domain,domain,Domain,Derived by stripping the protocol and any path from the 'URL' entry. More specifically a Fully Qualified Domain Name. Often referred to as the hostname. E.g. 'x.y.gov' or 't.mil',string,no Base Domain,base_domain,Base Domain,The second and top level domains of the `domain` entry. Intended to represent the registered second level domain that unifies one or more websites. E.g. 'n.gov' or 'p.mil'. ,string,"yes, exact" Top Level Domain,top_level_domain,Top Level Domain,The top level domain of the 'domain' entry. E.g. 'gov' or 'mil'. ,string,no Live,live,Whether the URL returns a 2** status code,Derived from the 'status_code' result. ,boolean,"yes, exact" Filter,filter,Whether the Initial URL would be filtered out if suspected non-public/non-websites are filtered out,"Generated during the Index building process by using a series of 'begins with' or 'contains' filters. This serves as an optional way to remove live domains that many people do not think of as part of an agency's public web presence (e.g. staging sites, admin login pages). ",boolean,no Status Code,status_code,HTTP status code,The last HTTP status code of the final URL (the 'url' entry). 2** (success); 4** (client error); 5** (server error). 1** (informational) and 3** (redirection) should not be possible since another status code should inherently come after it. ,numeric,no Media Type,media_type,Media type,Previously known as MIME type. ,string,no Agency,agency,Agency owner of the Initial Base Domain,"For .gov domains, the organization name that registered the domain. For other TLD's, the best estimation of the registering agency.",string,"yes, exact" Bureau,bureau,Bureau owner of the Initial Base Domain,"For .gov domains, the suborganization name that registered the domain. For other TLD's, the best estimation of the registering suborganization.",string,"yes, exact" Branch,branch,Brach of government of the agency owner of the Initial Base Domain,Derived from the associated field in each domain registry index. ,string,"yes, exact" 404 Test,404_test,Whether the Initial Domain return 404s correctly (TRUE) or gives a misleading 2** code (FALSE),"A site's 404/Page Not Found page should be accompanied by a 404 HTTP status code, yet they often do not. This is tested by taking the 'Initial URL' and appending a random string for a path and then noting the result. ",boolean,no Data Source,source_list,Which source datasets included this Initial URL?,"The Site Scanning website index, which serves as the target URL list, is assembled by combining and deduplicating several source files, each of which is assigned a name which carries into this field.. ",list,no Scan Status - Date,scan_date,Scan date/time,"Specifically, when the scan data is written to the database, which is usually within seconds of being generated",datetime,no Scan Status - Primary,primary_scan_status,Status of the Primary Scan,Either 'Completed' or one of several error messages described here - https://github.com/GSA/site-scanning-documentation/blob/main/pages/scan_statuses.md,string,"yes, exact (from a set of specified values)" Scan Status - Accessibility,accessibility_scan_status,Status of the Accessibility Scan ,Either 'Completed' or one of several error messages described here - https://github.com/GSA/site-scanning-documentation/blob/main/pages/scan_statuses.md,string,no Scan Status - DNS,dns_scan_status,Status of the DNS Scan ,Either 'Completed' or one of several error messages described here - https://github.com/GSA/site-scanning-documentation/blob/main/pages/scan_statuses.md,string,no Scan Status - Not Found,not_found_scan_status,Status of the 404 Test Scan ,Either 'Completed' or one of several error messages described here - https://github.com/GSA/site-scanning-documentation/blob/main/pages/scan_statuses.md,string,no Scan Status - Performance,performance_scan_status,Status of the Performance Scan,Either 'Completed' or one of several error messages described here - https://github.com/GSA/site-scanning-documentation/blob/main/pages/scan_statuses.md,string,no Scan Status - Robots.txt,robots_txt_scan_status,Status of the Robots.txt Scan ,Either 'Completed' or one of several error messages described here - https://github.com/GSA/site-scanning-documentation/blob/main/pages/scan_statuses.md,string,no Scan Status - Security,security_scan_status,Status of the Security Scan ,Either 'Completed' or one of several error messages described here - https://github.com/GSA/site-scanning-documentation/blob/main/pages/scan_statuses.md,string,no Scan Status - Sitemap.xml,sitemap_xml_scan_status,Status of the Sitemap.xml Scan ,Either 'Completed' or one of several error messages described here - https://github.com/GSA/site-scanning-documentation/blob/main/pages/scan_statuses.md,string,no Scan Status - WWW,www_scan_status,Status of the WWW Scan ,Either 'Completed' or one of several error messages described here - https://github.com/GSA/site-scanning-documentation/blob/main/pages/scan_statuses.md,string,no Homepage Hash,page_hash,MD5 Hash of the URL,The full HTML content of the page that loads is extracted from the page as a string and then hashed. Other filetypes are handled similarly. ,string,no Accessibility - Violations,accessibility_violations,Count of violations detected for certain accessibility categories that are detectable through automated scanning.,These violations were identified and selected in close coordination with the U.S. Access Board and GSA’s Government-wide IT Accessibility Program. Site Scanning uses axe-core rules to detect these violations. Use of automated scanning for accessibility does not replace the need for manual testing or assessment.,string,no Analytics - Pageviews,pageviews,Combined pageviews over the last 30 days for this domain.,Per the Top Hostnames User Traffic Report on analytics.usa.gov,string,no Analytics - Visits,visits,Combined visits over the last 30 days for this domain.,Per the Top Hostnames User Traffic Report on analytics.usa.gov,string,no DNS - IPv6,ipv6,Presence of a DNS AAAA record,,boolean,no DNS - Hostname,hostname,"Domain of underlying system, which often suggests use of a cloud or CDN provider",Only hostnames that contain one of several strings in them are included in order to highlight commonly used cloud services. ,string,no Infrastructure - CMS Provider,cms,Content management system,"(currently Adobe, DNN, Drupal, Joomla, Sharepoint, Percussion, Silverstripe, Wordpress, Cloud.gov)",string,no Infrastructure - Login Provider,login_provider,Name of detected login service,,string,no Infrastructure - Login Detected,login,List of login indicators detected,,object,no Infrastructure - Site Search Detected,site_search,Is a site search functionality detected?,,boolean,no Infrastructure - Search.gov Detected,search_dot_gov,Is the use of Search.gov detected?,,boolean,no Infrastructure - DAP Detected,dap,Is the Digital Analytics Program (DAP) tag detected?,,boolean,"yes, exact" Infrastructure - DAP Parameters,dap_parameters,List of DAP parameters detected,,object,no Infrastructure - DAP Version,dap_version,Version number of the DAP snippet,,string,no Infrastructure - Google Analytics Tag ID,ga_tag_id,Tag ID of any detected google analytics snippets,,string,no Infrastructure - Third Party Service Domains,third_party_service_domains,List of distinct third party domains detected,,list,no Infrastructure - Third Party Service URLs,third_party_service_urls,List of distinct third party urls detected,,list,no Infrastructure - Third Party Service Count,third_party_service_count,Count of distinct third party domains detected,,numeric,no Infrastructure - Cookie Domains,cookie_domains,List of domains associated with each cookie sent by a server,,list,no Mobile - Viewport Meta Tag Detected,viewport_meta_tag,Whether a certain meta tag needed for mobile friendliness is present and compliant,"Specifically, is there a viewport meta tag in the ; does it contain a content attribute; does the content attribute's value contain the text 'width='?",boolean,no Performance - Cumulative Layout Shift,cumulative_layout_shift,Performance indicator that measures how much layouts shift for a user while the page loads for a user.,Expressed by a layout shift score.,string,no Performance - Largest Contentful Paint,largest_contentful_paint,Performance indicator that counts in milliseconds when what is likely the main content on the page loads,,string,no Required Links - URL,required_links_url,"Whether certain required links are detected, as seen by the hyperlinks present on the page. ","Searches for certain character strings anywhere in the URL of a hyperlink, suggesting a required link (per digital.gov/resources/required-web-content-and-links) or a link to a Spanish-language site or section. ",list,no Required Links - Text,required_links_text,"Whether certain required links are detected, as seen by the hyperlinked text present on the page. ","Searches for certain character strings anywhere in any hyperlinked text, suggesting a required link (per digital.gov/resources/required-web-content-and-links) or a link to a Spanish-language site or section",list,no Security - HTTPS Required,https_enforced,Whether the Final URL enforces HTTPS,,boolean,no Security - HSTS Preloaded,hsts,Whether the Final URL domain is preloaded for HSTS,,boolean,no SEO - title,title,Meta page title,,string,no SEO - description,description,Meta page description,,string,no SEO - keywords,keywords,Meta page keywords,,string,no Open Graph - og:title,og_title,Open Graph page title,,string,no Open Graph - og:description,og_description,Open Graph description,,string,no Open Graph - article:published_time,og_article_published,Open Graph article published time,,datetime,no Open Graph - article:modified_time,og_article_modified,Open Graph article modified time,,datetime,no Open Graph - og:image,og_image,Open Graph image link,,string,no Open Graph - og:type,og_type,Open Graph type tag,,string,no Open Graph - og:url,og_url,Open Graph object URL,,string,no Other Tags - Canonical Link,canonical_link,Canonical URL HTML element,,string,no Other Tags - Language,language,Meta language tag,,string,no Other Tags - Language Links,language_link,HTML link attributes for the langauge of linked pages,,string,no Other Tags - Main Element Detected,main_element_present,Presence of `
` element,,boolean,no Robots.txt - Detected,robots_txt_detected,Is the `robots_txt_final_url` live (has a 2xx code) and is its entire path `/robots.txt`?,Note that a robots.txt file that redirects to a subdirectory would then not count given that the protocol appears to require that the file live in the top directory.,boolean,no Robots.txt - Final URL,robots_txt_url,"`Robots.txt` Final URL after redirects (301, 302, etc.)",,string,no Robots.txt - Final URL - Status Code,robots_txt_status_code,HTTP status code of the `robots.txt` Final URL,,numeric,no Robots.txt - Final URL - Media Type,robots_txt_media_type,Media type of the `robots.txt` Final URL,,string,no Robots.txt - Final URL - Filesize,robots_txt_filesize,Filesize of the `robots.txt` Final URL (bytes),,numeric,no Robots.txt - Crawl Delay,robots_txt_crawl_delay,`Robots.txt` specified crawl delay (seconds),,numeric,no Robots.txt - Sitemap Locations,robots_txt_sitemap_locations,"List of additional sitemap locations (URLs) found in the robots.txt (limited to 2,000 characters)",,array,no Sitemap.xml - Detected,sitemap_xml_detected,Is the `sitemap_xml_url` live?,,boolean,no Sitemap.xml - Final URL,sitemap_xml_url,"`Sitemap.xml` Final URL after redirects (301, 302, etc.)",,string,no Sitemap.xml - Final URL - Status Code,sitemap_xml_status_code,HTTP status code of the `sitemap.xml` Final URL,,numeric,no Sitemap.xml - Final URL - Media Type,sitemap_xml_media_type,Media type of the `sitemap.xml` Final URL,,string,no Sitemap.xml - Final URL - Filesize,sitemap_xml_filesize,Filesize of the `sitemap.xml` Final URL (bytes),,numeric,no Sitemap.xml - Items Count,sitemap_xml_count,Count of URLs in the detected `sitemap.xml`,,numeric,no Sitemap.xml - Last Modified,sitemap_xml_lastmod,Most recent `Lastmod` value in the sitemap.xml file,,datetime,no Sitemap.xml - PDF Count,sitemap_xml_pdf_count,Count of URLs ending in `.pdf` in the detected `sitemap.xml`,,numeric,no Sitemap.xml - Hash,sitemap_xml_page_hash,MD5 hash of the sitemap.xml file,,string,no USWDS - Favicon,uswds_favicon,Presence of USWDS favicon file (part of the standard header banner) in markup,,integer,no USWDS - Favicon in CSS,uswds_favicon_in_css,Presence of USWDS favicon file (part of the standard header banner) in CSS,,integer,no USWDS - Public Sans Font,uswds_publicsans_font,Presence of Public Sans font files in CSS,,integer,no USWDS - In-page CSS,uswds_inpage_css,Presence of '.usa-' class in a