openapi: 3.2.0 info: title: Extract Article API description: Diffbot's Extraction APIs include various endpoints to extract JSON data/fields from different types of web pages. The Analyze endpoint is used when the page type is unknown, and will attempt to identify the page type and send it to the appropriate extraction API. termsOfService: https://www.diffbot.com/terms/ contact: email: support@diffbot.com version: 1.1.0 servers: - url: https://api.diffbot.com/v3 tags: - name: Article paths: /article: get: tags: - Article summary: Article description: Automatically extract clean article text and other data from news articles, blog posts and other text-heavy pages. operationId: article parameters: - name: url in: query description: Target URL to extract required: true schema: type: string default: https://www.technologyreview.com/2020/09/04/1008156/knowledge-graph-ai-reads-web-machine-learning-natural-language-processing/ - name: fields in: query description: Specify optional fields to be returned from any fully-extracted pages (e.g. `fields=querystring,links`) schema: type: string enum: - links - extlinks - meta - querystring - breadcrumb - quotes - name: timeout in: query description: Sets a value in milliseconds to wait for the retrieval/fetch of content from the requested URL. The default timeout for the third-party response is 30 seconds (30000). schema: type: integer format: int32 - name: callback in: query description: Use for jsonp requests. Needed for cross-domain ajax. schema: type: string - name: proxy in: query description: 'Specify an IP address of a [custom proxy](https://docs.diffbot.com/reference/using-proxies#how-to-use-proxies) that will be used to fetch the target page. (Ex: `&proxy` or `&proxy=0.0.0.0`)' schema: type: string default: '' - name: proxyAuth in: query description: 'Used to specify the authentication parameters that will be used with a custom proxy specified in the &proxy parameter. (Ex: `proxyAuth=username:password`)' schema: type: string - name: useProxy in: query description: Set to `default` to use [Diffbot's datacenter proxy](https://docs.diffbot.com/reference/using-proxies#how-to-use-proxies) for this request. `none` will instruct Extract to not use proxies, even if proxies have been enabled for this particular URL globally. schema: type: string - name: paging in: query description: Pass `paging=false` to disable automatic concatenation multiple-page articles. schema: type: boolean - name: maxTags in: query description: 'Set the maximum number of automatically-generated tags to return. (Default: 10)' schema: type: integer format: int32 - name: tagConfidence in: query description: Set the minimum relevance `score` of tags to return, between 0.0 and 1.0. By default only tags with a `score` equal to or above 0.5 will be returned. schema: type: number format: float - name: categoryConfidence in: query description: Set the minimum relevance `score` of categories to return, between 0.0 and 1.0. By default only categories with a `score` equal to or above 0.5 will be returned. schema: type: number format: float - name: discussion in: query description: Pass `discussion=false` to disable automatic extraction of article comments. schema: type: boolean - name: naturalLanguage in: query description: 'Run extracted text and title through the Diffbot Natural Language API. Example: `&naturalLanguage=entities,facts,categories,sentiment`.' schema: type: string enum: - entities - sentiment - summary - facts - openFacts - records - categories - sentences - language - name: summaryNumSentences in: query description: 'Sets the maximum number of sentences for summary generation when using naturalLanguage=summary (Default: 3).' schema: type: integer format: int32 responses: '200': description: Successful API Response content: application/json: schema: type: object properties: request: type: object properties: pageUrl: type: string api: type: string version: type: integer humanLanguage: type: string objects: type: array items: type: object properties: date: type: string sentiment: type: number images: type: array items: type: object properties: naturalHeight: type: integer width: type: integer diffbotUri: type: string url: type: string naturalWidth: type: integer height: type: integer author: type: string estimatedDate: type: string publisherRegion: type: string icon: type: string diffbotUri: type: string siteName: type: string type: type: string title: type: string tags: type: array items: type: object properties: score: type: number sentiment: type: number count: type: integer label: type: string uri: type: string rdfTypes: type: array items: type: string publisherCountry: type: string humanLanguage: type: string authorUrl: type: string pageUrl: type: string html: type: string categories: type: array items: type: object properties: score: type: number name: type: string id: type: string text: type: string authors: type: array items: type: object properties: name: type: string link: type: string type: type: string title: type: string example: request: pageUrl: https://www.technologyreview.com/2020/09/04/1008156/knowledge-graph-ai-reads-web-machine-learning-natural-language-processing/ api: article version: 3 humanLanguage: en objects: - date: Fri, 04 Sep 2020 00:00:00 GMT sentiment: 0.153 images: - naturalHeight: 869 width: 654 diffbotUri: image|3|1663647584 url: https://wp.technologyreview.com/wp-content/uploads/2022/03/Flower-Trip-style.jpeg?resize=1006,640 naturalWidth: 1366 height: 418 author: Will Douglas Heaven estimatedDate: Fri, 04 Sep 2020 00:00:00 GMT publisherRegion: North America icon: https://www.technologyreview.com/static/media/favicon.1cfcdb44.ico diffbotUri: article|3|973247980 siteName: MIT Technology Review type: article title: This know-it-all AI learns by reading the entire web nonstop tags: - score: 0.998680055141449 sentiment: 0 count: 10 label: artificial intelligence uri: https://diffbot.com/entity/E_lYDrjmAMlKKwXaDf958zg rdfTypes: - http://dbpedia.org/ontology/Skill - http://dbpedia.org/ontology/Activity - score: 0.9686350226402283 sentiment: 0.889 count: 7 label: Diffbot uri: https://diffbot.com/entity/EYX1i02YVPsuT7fPLUYgRhQ rdfTypes: - http://dbpedia.org/ontology/Organisation - score: 0.9306924939155579 sentiment: 0 count: 2 label: Michigan uri: https://diffbot.com/entity/E2eIrTt0jPUmGmuV6N2O3KQ rdfTypes: - http://dbpedia.org/ontology/Place - http://dbpedia.org/ontology/PopulatedPlace - http://dbpedia.org/ontology/State - score: 0.9025880098342896 sentiment: 0 count: 1 label: Paul Katsen uri: https://diffbot.com/entity/EqUim_ci0ObmrK2gZM3UfNA rdfTypes: - http://dbpedia.org/ontology/Person - score: 0.8933213353157043 sentiment: 0.48 count: 4 label: Katy Perry uri: https://diffbot.com/entity/E_6rhi_PEOD6vGencwOxd2A rdfTypes: - http://dbpedia.org/ontology/Person - score: 0.8848651051521301 sentiment: 0 count: 4 label: Mike Tung uri: https://diffbot.com/entity/ESGMaGV9uP0SuTmfPTtNEoA rdfTypes: - http://dbpedia.org/ontology/Person - score: 0.8562507629394531 sentiment: 0 count: 4 label: Google uri: https://diffbot.com/entity/EUFq-3WlpNsq0pvfUYWXOEA rdfTypes: - http://dbpedia.org/ontology/Organisation - score: 0.7750672101974487 sentiment: 0 count: 2 label: Alaska uri: https://diffbot.com/entity/E4odwkG_xMNeZTbHrnNrojA rdfTypes: - http://dbpedia.org/ontology/Place - http://dbpedia.org/ontology/PopulatedPlace - http://dbpedia.org/ontology/State - score: 0.7653270959854126 sentiment: 0 count: 1 label: Zola uri: https://diffbot.com/entity/E0qGTA2o5NjaeezggjMsoVw rdfTypes: - http://dbpedia.org/ontology/Organisation - score: 0.7643865942955017 sentiment: 0.75 count: 3 label: GUID Partition Table uri: https://diffbot.com/entity/EReKbXuSJMYmoM8lawtgEsA rdfTypes: - http://dbpedia.org/ontology/Skill - http://dbpedia.org/ontology/Activity publisherCountry: United States humanLanguage: en authorUrl: https://www.technologyreview.com/author/will-douglas-heaven/ pageUrl: https://www.technologyreview.com/2020/09/04/1008156/knowledge-graph-ai-reads-web-machine-learning-natural-language-processing/ html: '
knowledge graph illustration

Back in July, OpenAI’s latest language model, GPT-3, dazzled with its ability to churn out paragraphs that look as if they could have been written by a human. People started showing off how GPT-3 could also autocomplete code or fill in blanks in spreadsheets.

In one example, Twitter employee Paul Katsen tweeted “the spreadsheet function to rule them all,” in which GPT-3 fills out columns by itself, pulling in data for US states: the population of Michigan is 10.3 million, Alaska became a state in 1906, and so on.

Except that GPT-3 can be a bit of a bullshitter. The population of Michigan has never been 10.3 million, and Alaska became a state in 1959.

Language models like GPT-3 are amazing mimics, but they have little sense of what they’re actually saying. “They’re really good at generating stories about unicorns,” says Mike Tung, CEO of Stanford startup Diffbot. “But they’re not trained to be factual.”

This is a problem if we want AIs to be trustworthy. That’s why Diffbot takes a different approach. It is building an AI that reads every page on the entire public web, in multiple languages, and extracts as many facts from those pages as it can.

Like GPT-3, Diffbot’s system learns by vacuuming up vast amounts of human-written text found online. But instead of using that data to train a language model, Diffbot turns what it reads into a series of three-part factoids that relate one thing to another: subject, verb, object.

Pointed at my bio, for example, Diffbot learns that Will Douglas Heaven is a journalist; Will Douglas Heaven works at MIT Technology Review; MIT Technology Review is a media company; and so on. Each of these factoids gets joined up with billions of others in a sprawling, interconnected network of facts. This is known as a knowledge graph.

Knowledge graphs are not new. They have been around for decades, and were a fundamental concept in early AI research. But constructing and maintaining knowledge graphs has typically been done by hand, which is hard. This also stopped Tim Berners-Lee from realizing what he called the semantic web, which would have included information for machines as well as humans, so that bots could book our flights, do our shopping, or give smarter answers to questions than search engines.

A few years ago, Google started using knowledge graphs too. Search for “Katy Perry” and you will get a box next to the main search results telling you that Katy Perry is an American singer-songwriter with music available on YouTube, Spotify, and Deezer. You can see at a glance that she is married to Orlando Bloom, she’s 35 and worth $125 million, and so on. Instead of giving you a list of links to pages about Katy Perry, Google gives you a set of facts about her drawn from its knowledge graph.

But Google only does this for its most popular search terms. Diffbot wants to do it for everything. By fully automating the construction process, Diffbot has been able to build what may be the largest knowledge graph ever.

Alongside Google and Microsoft, it is one of only three US companies that crawl the entire public web. “It definitely makes sense to crawl the web,” says Victoria Lin, a research scientist at Salesforce who works on natural-language processing and knowledge representation. “A lot of human effort can otherwise go into making a large knowledge base.” Heiko Paulheim at the University of Mannheim in Germany agrees: “Automation is the only way to build large-scale knowledge graphs.”

Super surfer

To collect its facts, Diffbot’s AI reads the web as a human would—but much faster. Using a super-charged version of the Chrome browser, the AI views the raw pixels of a web page and uses image-recognition algorithms to categorize the page as one of 20 different types, including video, image, article, event, and discussion thread. It then identifies key elements on the page, such as headline, author, product description, or price, and uses NLP to extract facts from any text.

Every three-part factoid gets added to the knowledge graph. Diffbot extracts facts from pages written in any language, which means that it can answer queries about Katy Perry, say, using facts taken from articles in Chinese or Arabic even if they do not contain the term “Katy Perry.”

Browsing the web like a human lets the AI see the same facts that we see. It also means it has had to learn to navigate the web like us. The AI must scroll down, switch between tabs, and click away pop-ups. “The AI has to play the web like a video game just to experience the pages,” says Tung.

Diffbot crawls the web nonstop and rebuilds its knowledge graph every four to five days. According to Tung, the AI adds 100 million to 150 million entities each month as new people pop up online, companies are created, and products are launched. It uses more machine-learning algorithms to fuse new facts with old, creating new connections or overwriting out-of-date ones. Diffbot has to add new hardware to its data center as the knowledge graph grows.

Researchers can access Diffbot’s knowledge graph for free. But Diffbot also has around 400 paying customers. The search engine DuckDuckGo uses it to generate its own Google-like boxes. Snapchat uses it to extract highlights from news pages. The popular wedding-planner app Zola uses it to help people make wedding lists, pulling in images and prices. NASDAQ, which provides information about the stock market, uses it for financial research.

Fake shoes

Adidas and Nike even use it to search the web for counterfeit shoes. A search engine will return a long list of sites that mention Nike trainers. But Diffbot lets these companies look for sites that are actually selling their shoes, rather just talking about them.

For now, these companies must interact with Diffbot using code. But Tung plans to add a natural-language interface. Ultimately, he wants to build what he calls a “universal factoid question answering system”: an AI that could answer almost anything you asked it, with sources to back up its response.

Tung and Lin agree that this kind of AI cannot be built with language models alone. But better yet would be to combine the technologies, using a language model like GPT-3 to craft a human-like front end for a know-it-all bot.

Still, even an AI that has its facts straight is not necessarily smart. “We’re not trying to define what intelligence is, or anything like that,” says Tung. “We’re just trying to build something useful.”

NLP maps hallucinogenic experience
Demis Hassabis
' categories: - score: 0.962 name: Technology & Computing id: iabv2-596 - score: 0.962 name: Artificial Intelligence id: iabv2-597 text: 'Back in July, OpenAI’s latest language model, GPT-3, dazzled with its ability to churn out paragraphs that look as if they could have been written by a human. People started showing off how GPT-3 could also autocomplete code or fill in blanks in spreadsheets. In one example, Twitter employee Paul Katsen tweeted “the spreadsheet function to rule them all,” in which GPT-3 fills out columns by itself, pulling in data for US states: the population of Michigan is 10.3 million, Alaska became a state in 1906, and so on. Except that GPT-3 can be a bit of a bullshitter. The population of Michigan has never been 10.3 million, and Alaska became a state in 1959. Language models like GPT-3 are amazing mimics, but they have little sense of what they’re actually saying. “They’re really good at generating stories about unicorns,” says Mike Tung, CEO of Stanford startup Diffbot. “But they’re not trained to be factual.” This is a problem if we want AIs to be trustworthy. That’s why Diffbot takes a different approach. It is building an AI that reads every page on the entire public web, in multiple languages, and extracts as many facts from those pages as it can. Like GPT-3, Diffbot’s system learns by vacuuming up vast amounts of human-written text found online. But instead of using that data to train a language model, Diffbot turns what it reads into a series of three-part factoids that relate one thing to another: subject, verb, object. Pointed at my bio, for example, Diffbot learns that Will Douglas Heaven is a journalist; Will Douglas Heaven works at MIT Technology Review; MIT Technology Review is a media company; and so on. Each of these factoids gets joined up with billions of others in a sprawling, interconnected network of facts. This is known as a knowledge graph. Knowledge graphs are not new. They have been around for decades, and were a fundamental concept in early AI research. But constructing and maintaining knowledge graphs has typically been done by hand, which is hard. This also stopped Tim Berners-Lee from realizing what he called the semantic web, which would have included information for machines as well as humans, so that bots could book our flights, do our shopping, or give smarter answers to questions than search engines. A few years ago, Google started using knowledge graphs too. Search for “Katy Perry” and you will get a box next to the main search results telling you that Katy Perry is an American singer-songwriter with music available on YouTube, Spotify, and Deezer. You can see at a glance that she is married to Orlando Bloom, she’s 35 and worth $125 million, and so on. Instead of giving you a list of links to pages about Katy Perry, Google gives you a set of facts about her drawn from its knowledge graph. But Google only does this for its most popular search terms. Diffbot wants to do it for everything. By fully automating the construction process, Diffbot has been able to build what may be the largest knowledge graph ever. Alongside Google and Microsoft, it is one of only three US companies that crawl the entire public web. “It definitely makes sense to crawl the web,” says Victoria Lin, a research scientist at Salesforce who works on natural-language processing and knowledge representation. “A lot of human effort can otherwise go into making a large knowledge base.” Heiko Paulheim at the University of Mannheim in Germany agrees: “Automation is the only way to build large-scale knowledge graphs.” Super surfer To collect its facts, Diffbot’s AI reads the web as a human would—but much faster. Using a super-charged version of the Chrome browser, the AI views the raw pixels of a web page and uses image-recognition algorithms to categorize the page as one of 20 different types, including video, image, article, event, and discussion thread. It then identifies key elements on the page, such as headline, author, product description, or price, and uses NLP to extract facts from any text. Every three-part factoid gets added to the knowledge graph. Diffbot extracts facts from pages written in any language, which means that it can answer queries about Katy Perry, say, using facts taken from articles in Chinese or Arabic even if they do not contain the term “Katy Perry.” Browsing the web like a human lets the AI see the same facts that we see. It also means it has had to learn to navigate the web like us. The AI must scroll down, switch between tabs, and click away pop-ups. “The AI has to play the web like a video game just to experience the pages,” says Tung. Diffbot crawls the web nonstop and rebuilds its knowledge graph every four to five days. According to Tung, the AI adds 100 million to 150 million entities each month as new people pop up online, companies are created, and products are launched. It uses more machine-learning algorithms to fuse new facts with old, creating new connections or overwriting out-of-date ones. Diffbot has to add new hardware to its data center as the knowledge graph grows. Researchers can access Diffbot’s knowledge graph for free. But Diffbot also has around 400 paying customers. The search engine DuckDuckGo uses it to generate its own Google-like boxes. Snapchat uses it to extract highlights from news pages. The popular wedding-planner app Zola uses it to help people make wedding lists, pulling in images and prices. NASDAQ, which provides information about the stock market, uses it for financial research. Fake shoes Adidas and Nike even use it to search the web for counterfeit shoes. A search engine will return a long list of sites that mention Nike trainers. But Diffbot lets these companies look for sites that are actually selling their shoes, rather just talking about them. For now, these companies must interact with Diffbot using code. But Tung plans to add a natural-language interface. Ultimately, he wants to build what he calls a “universal factoid question answering system”: an AI that could answer almost anything you asked it, with sources to back up its response. Tung and Lin agree that this kind of AI cannot be built with language models alone. But better yet would be to combine the technologies, using a language model like GPT-3 to craft a human-like front end for a know-it-all bot. Still, even an AI that has its facts straight is not necessarily smart. “We’re not trying to define what intelligence is, or anything like that,” says Tung. “We’re just trying to build something useful.”' authors: - name: Will Douglas Heavenarchive page link: technologyreview.com/author/will-douglas-heaven type: article title: This know-it-all AI learns by reading the entire web nonstop | MIT Technology Review '500': description: Internal Server Error content: application/json: schema: type: object properties: errorCode: type: integer error: type: string example: errorCode: 500 error: Internal Server Error security: - tokenscheme: [] components: securitySchemes: tokenscheme: type: apiKey name: token in: query x-readme: explorer-enabled: true proxy-enabled: true samples-enabled: true