{"cells":[{"cell_type":"markdown","metadata":{},"source":["\n"," \n","\n"]},{"cell_type":"markdown","metadata":{},"source":["
index.html
\". If your request is successful, the server will send the object to the client in an **HTTP response**. This includes information like the type of the **resource**, the length of the **resource**, and other information.\n","\n","\n","The figure below represents the process. The circle on the left represents the client, the circle on the right represents the Web server. The table under the Web server represents a list of resources stored in the web server. In this case an HTML
file, png
image, and txt
file .\n","
\n","The HTTP protocol allows you to send and receive information through the web including webpages, images, and other web resources. In this lab, we will provide an overview of the Requests library for interacting with the HTTP
protocol. \n","
http://
www.ibm.com
and www.gitlab.com
/images/IDSNlogo.png
GET
method, this is an HTTP
method. Also the location of the resource /index.html
and the HTTP
version. The Request header passes additional information with an HTTP
request:\n"]},{"cell_type":"markdown","metadata":{},"source":["HTTP
request is made, an HTTP
method is sent, this tells the server what action to perform. A list of several HTTP
methods is shown below. We will go over more examples later.\n"]},{"cell_type":"markdown","metadata":{},"source":["HTTP/1.0
, a status code (200) meaning success, followed by a descriptive phrase (OK). The response header contains useful information. Finally, we have the response body containing the requested file, an HTML
document. It should be noted that some requests have headers.\n"]},{"cell_type":"markdown","metadata":{},"source":["HTTP/1.1
requests easily. We can import the library as follows:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["import requests"]},{"cell_type":"markdown","metadata":{},"source":["We will also use the following libraries:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["import os \n","from PIL import Image\n","from IPython.display import IFrame"]},{"cell_type":"markdown","metadata":{},"source":["You can make a GET
request via the method get
to [www.ibm.com](http://www.ibm.com/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0101ENSkillsNetwork19487395-2022-01-01):\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["url='https://www.ibm.com/'\n","r=requests.get(url)\n"]},{"cell_type":"markdown","metadata":{},"source":["We have the response object r
, this has information about the request, like the status of the request. We can view the status code using the attribute status_code
.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["r.status_code"]},{"cell_type":"markdown","metadata":{},"source":["You can view the request headers:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["print(r.request.headers)"]},{"cell_type":"markdown","metadata":{},"source":["You can view the request body, in the following line, as there is no body for a get request we get a None
:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["print(\"request body:\", r.request.body)"]},{"cell_type":"markdown","metadata":{},"source":["You can view the HTTP
response header using the attribute headers
. This returns a python dictionary of HTTP
response headers.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["header=r.headers\n","print(r.headers)"]},{"cell_type":"markdown","metadata":{},"source":["We can obtain the date the request was sent using the key Date
\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["header['date']"]},{"cell_type":"markdown","metadata":{},"source":["Content-Type
indicates the type of data:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["header['Content-Type']"]},{"cell_type":"markdown","metadata":{},"source":["You can also check the encoding
:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":[" r.encoding"]},{"cell_type":"markdown","metadata":{},"source":["As the Content-Type
is text/html
we can use the attribute text
to display the HTML
in the body. We can review the first 100 characters:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["r.text[0:100]"]},{"cell_type":"markdown","metadata":{},"source":["You can load other types of data for non-text requests, like images. Consider the URL of the following image:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Use single quotation marks for defining string\n","url='https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0101EN-SkillsNetwork/IDSNlogo.png'"]},{"cell_type":"markdown","metadata":{},"source":["We can make a get request:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["r=requests.get(url)"]},{"cell_type":"markdown","metadata":{},"source":["We can look at the response header:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["print(r.headers)"]},{"cell_type":"markdown","metadata":{},"source":["We can see the 'Content-Type'
\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["r.headers['Content-Type']"]},{"cell_type":"markdown","metadata":{},"source":["An image is a response object that contains the image as a bytes-like object. As a result, we must save it using a file object. First, we specify the file path and\n","name\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["path=os.path.join(os.getcwd(),'image.png')\n","path"]},{"cell_type":"markdown","metadata":{},"source":["We save the file, in order to access the body of the response we use the attribute content
then save it using the open
function and write method
:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["with open(path,'wb') as f:\n"," f.write(r.content)"]},{"cell_type":"markdown","metadata":{},"source":["We can view the image:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["Image.open(path) "]},{"cell_type":"markdown","metadata":{},"source":[" wget
wget
function to retrieve content from the web server as shown below. Write the python code to perform the same task. The code should be the same as the one used to download the image, but the file name should be 'Example1.txt'
.\n"]},{"cell_type":"markdown","metadata":{},"source":["!wget -O /resources/data/Example1.txt
\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["/get
, this indicates we would like to preform a GET
request. This is demonstrated in the following table:\n"]},{"cell_type":"markdown","metadata":{},"source":["[http://httpbin.org/](http://httpbin.org/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0101ENSkillsNetwork19487395-2022-01-01)
is a simple HTTP Request & Response Service. The URL
in Python is given by:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["url_get='http://httpbin.org/get'"]},{"cell_type":"markdown","metadata":{},"source":["A query string is a part of a uniform resource locator (URL), this sends other information to the web server. The start of the query is a ?
, followed by a series of parameter and value pairs, as shown in the table below. The first parameter name is name
and the value is Joseph
. The second parameter name is ID
and the Value is 123
. Each pair, parameter, and value is separated by an equals sign, =
.\n","The series of pairs is separated by the ampersand &
.\n"]},{"cell_type":"markdown","metadata":{},"source":["payload
to the params
parameter of the get()
function:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["r=requests.get(url_get,params=payload)"]},{"cell_type":"markdown","metadata":{},"source":["We can print out the URL
and see the name and values\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["r.url"]},{"cell_type":"markdown","metadata":{},"source":["There is no request body\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["print(\"request body:\", r.request.body)"]},{"cell_type":"markdown","metadata":{},"source":["We can print out the status code\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["print(r.status_code)"]},{"cell_type":"markdown","metadata":{},"source":["We can view the response as text:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["print(r.text)"]},{"cell_type":"markdown","metadata":{},"source":["We can look at the 'Content-Type'
.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["r.headers['Content-Type']"]},{"cell_type":"markdown","metadata":{},"source":["As the content 'Content-Type'
is in the JSON
format we can use the method json()
, it returns a Python dict
:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["r.json()"]},{"cell_type":"markdown","metadata":{},"source":["The key args
has the name and values:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["r.json()['args']"]},{"cell_type":"markdown","metadata":{},"source":["GET
request, a POST
is used to send data to a server, but the POST
request sends the data in a request body. In order to send the Post Request in Python, in the URL
we change the route to POST
:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["url_post='http://httpbin.org/post'"]},{"cell_type":"markdown","metadata":{},"source":["This endpoint will expect data as a file or as a form. A form is convenient way to configure an HTTP request to send data to a server.\n"]},{"cell_type":"markdown","metadata":{},"source":["To make a POST
request we use the post()
function, the variable payload
is passed to the parameter data
:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["r_post=requests.post(url_post,data=payload)"]},{"cell_type":"markdown","metadata":{},"source":["Comparing the URL from the response object of the GET
and POST
request we see the POST
request has no name or value pairs.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["print(\"POST request URL:\",r_post.url )\n","print(\"GET request URL:\",r.url)"]},{"cell_type":"markdown","metadata":{},"source":["We can compare the POST
and GET
request body, we see only the POST
request has a body:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["print(\"POST request body:\",r_post.request.body)\n","print(\"GET request body:\",r.request.body)"]},{"cell_type":"markdown","metadata":{},"source":["We can view the form as well:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["r_post.json()['form']"]},{"cell_type":"markdown","metadata":{},"source":["There is a lot more you can do. Check out Requests for more.\n"]},{"cell_type":"markdown","metadata":{},"source":["Joseph Santarcangelo
A Data Scientist at IBM, and holds a PhD in Electrical Engineering. His research focused on using Machine Learning, Signal Processing, and Computer Vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD.