{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Kivy指南-8-着色器app\n", "\n", "> 利用OpenGL的特性做一个射击游戏\n", "\n", "- toc: true \n", "- badges: true\n", "- comments: true\n", "- categories: [jupyter,Kivy,Android,iOS]\n", "- image: kbpic/8.9.screenshot.png" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "恭喜你Hold到现在！最后两章我们准备讨论一些OpenGL的底层细节，与Kivy完全不同的内容，比如**OpenGL着色器语言（OpenGL Shading Language，GLSL）**。这个语言可以让我们轻松写出高性能的代码。\n", "\n", "我们将通过一个屏幕保护程序来介绍OpenGL的特性，然后做一个射击游戏（shoot-em-up game，shmup）app。本章的代码是准备工作，和其他项目都是一章不同，射击游戏在下一章做。\n", "\n", "\n", "\n", "本章会讨论很多复杂的问题，不过由于篇幅不能面面俱到，建议阅读OpenGL最新文档，因为OpenGL是一个标准，更新很快，新特性不断被加入，希望你在阅读的时候及时留意OpenGL相关的进展。\n", "\n", "首先要讨论的是高性能着色器方法，尽管和普通的Kivy代码差别不大，而且两者大部分都相互兼容，在很多部件中都可以互相替代。因此，这种方法主要是用GLSL实现于资源消耗很大、性能要求很高的部分，消除性能的瓶颈。\n", "\n", "## OpenGL简介\n", "\n", "现在让我们简单介绍下OpenGL。OpenGL是一个底层的图像API，其标准被广泛采用。桌面系统和移动系统均支持（在移动系统上是OpenGL ES，**Embedded Systems，嵌入式系统**）。现代浏览器也支持OpenGL ES的一个分支版本，就是大名鼎鼎的WebGL。\n", "\n", "OpenGL的广泛采用使其具有良好的跨平台能力，尤其在视频游戏和图形编程方面。Kivy的跨平台着色器同样依赖于OpenGL。\n", "\n", "### 并行方式\n", "\n", "OpenGL可以操纵基本的图形元素，如单个的点和屏幕上的每个像素。我们可以画三个点，然后构成一个三角形，计算每个像素的应该填充的颜色。你可能认为这种做法是非常麻烦的。这正是像kivy这样的高级图形框架出现的原因：它们都隐藏了OpenGL的管线（pipeline）的一堆细节，提供组件和布局来简化操作。复杂底层的管线功能如下图所示：\n", "\n", "![pipeline](kbpic/8.1.pipeline.png)\n", "\n", "这个复杂的图形包括以下四个方面：" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- 应用给OpenGL一个点数组（vertices），可以重用这些点的索引数组（indices），以及其他数值（uniforms）\n", "- 对每个点进行点着色器（vertex shader），根据需要做一系列计算。然后输出到片段着色器（fragment shader）\n", "- 对每个像素进行分段着色器（也叫像素着色器，pixel shader），计算像素的颜色。有时除了点着色器的输出结果，还要考虑一个颜色常量\n", "- 像素被着色器到屏幕上，其他的一些任务被完成，这些任务我们暂不讨论\n", "\n", ">上面使用的一组点称为模型（model）或网格（mesh）。它们不一定是连续的，也可能是离散的多边形；这些模型的理论基础后面会介绍。\n", "\n", "OpenGL超快速度的背后是大量并行计算方法的使用。上面提到的点着色器和像素着色器可能不会自动飞快运行，但是当通过GPU加速的瞬间，着色器造成的延迟并不会随着着色复杂度的增加呈指数级增长；在正常的硬件上其增速基本是线性的。\n", "\n", "对应到现实中，我们讨论的，就是今天用2-16核CPU的电脑进行多任务、并行编程的个人电脑。一般的中档显卡，也有几千个GPU核心，可以并行处理更多的计算。\n", "\n", "每个任务都是独立运行的，不像一般程序里面的线程，着色器不能等待其他任务完成再运行，这样会出现阻塞且严重影响性能，除了使用管线命令的时候（就像前面提到的，先进行点着色器，再运行像素着色器）。这种限制在你刚刚接触GLSL的时候，可能有点难理解。\n", "\n", "这就是为什么有些算法可以在GPU上面非常高效。现代加密算法像**bcrypt**就是用来限制这种高并发性能的——通过限制暴力破解能力来保障安全性。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 性能得失" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "并非只有原始的OpenGL才可以获得极快的性能。很多时候，像Kivy这样的高级框架也可以如此。比如，当你在屏幕上着色器多边形的时候，下面的一系列动作就会发生：\n", "\n", "- 用python定义多边形的形状和位置\n", "- 点，索引数组和相关的资源（如花纹）都上传到图像接口\n", "- 调用点着色器功能，进行定位、着色、缩放等变换\n", "- 最后，调用像素着色器功能，返回一个栅格图像显示到屏幕上\n", "\n", "无论你是要Kivy部件还是用原始的OpenGL命令和GLSL着色器，两种方式的性能都是一样的。这是因为Kivy底层和OpenGL类似。\n", "\n", "也就是说，底层优化其实没多大必要，这就是由几个矩形构成的*Kivy Bird*游戏，可以直接通过高级接口实现的原因。基本上，我们可以在*Kivy Bird*里优化一两个部件，但是其性能很难被察觉。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 改善性能" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "那么，性能到底如何改善呢？答案就是减少Python代码，精简着色器内容。假如我们要着色9,000个相似的多边形（比如秋天的落叶或漫天的行程）。如果每个多边形都用一个Kivy部件，那我们就要做很多个Python对象，要被单独序列化成OpenGL指令。另外，每个部件都有点集合相关的图像接口，必然产生大量的API调用，还有许多类似的匹配（mesh）。\n", "\n", "我们可以做两件事来优化：\n", "\n", "- 避免大量Python类实例，把它们放在一个数值中即可。如果储存成一个适合OpenGL处理的格式中，就可以免掉序列化的步骤\n", "- 把所有的图形组合到一起作为一个单独模型处理，这样可以减少API调用。批处理通常是很好的优化方法，因为它让OpenGL以并行方式更好的运行。\n", "\n", "在本章结束的时候我们会实现这些方法。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### GLSL介绍" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "GLSL作为一门语言，与C语法类似，是一个静态强类型。如果你不熟悉C语言，下面是参考。C与Python不同，不在乎缩进，句末要用分号，逻辑代码块要用括号。\n", "\n", "GLSL同时支持C和C++的注释方式：\n", "\n", "```C\n", "/* ANSI C-style comment */\n", "// C++ one-line comment\n", "```\n", "\n", "变量声明方式为`[type] [name] [= optional value];`：\n", "\n", "```C\n", "float a; // this has no direct Python equivalent\n", "int b = 1;\n", "```\n", "\n", "函数定义方式为`[type] [name] ([arguments]) { [body of function] }`：\n", "\n", "```C\n", "float pow2(float x)\n", "{\n", " return x * x;\n", "}\n", "```\n", "\n", "控制结构形式为：\n", "\n", "```C\n", "if (x < 9.0)\n", "{\n", " x = 9.0;\n", "}\n", "```\n", "\n", "差不多就这些了，即使没有学过C，你现在也可以读GLSL代码了。\n", "\n", "着色器代码的起点是`main()`函数。后面，我们把点着色器和像素着色器放在一个文件中，因此，在一个文件里面会有两个`main()`函数。其结构如下：\n", "\n", "```C\n", "void main(void)\n", "{\n", " // code\n", "}\n", "```\n", "\n", "这里的`void`类型表示函数没有返回值，和Python的`NoneType`不一样，你不能把变量声明为`void`类型。这里的两个`main()`函数返回值和参数都被忽略了，所以写成`void main(void)`。着色器会根据需要把结果写到内部变量，`gl_Position`，`gl_FragColor`和其他变量里面，不需要返回数据结果，输入参数为空也是同理。\n", "\n", "GLSL类型系统完全反映了它的作用。不像C语言，它为点和矩阵准备了专门的类型，这些类型支持数学运算（所以你可以直接对矩阵变量进行乘法`mat1 * mat2`，多简单啊！C一个试试就知道C多麻烦了）。在计算机图形学里，矩阵计算不可或缺，后面会介绍。\n", "\n", "下面，我们就来写几个GLSL的例子。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 在Kivy中自定义着色器" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "首先我们需要用Python代码实现窗口、加载着色器等等。代码如下：" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from kivy.app import App\n", "from kivy.base import EventLoop\n", "from kivy.graphics import Mesh\n", "from kivy.graphics.instructions import RenderContext\n", "from kivy.uix.widget import Widget\n", "\n", "\n", "class GlslDemo(Widget):\n", " def __init__(self, **kwargs):\n", " Widget.__init__(self, **kwargs)\n", " self.canvas = RenderContext(use_parent_projection=True)\n", " self.canvas.shader.source = \"basic.glsl\"\n", " # Set up geometry here.\n", "\n", "\n", "class GlslApp(App):\n", " def build(self):\n", " EventLoop.ensure_window()\n", " return GlslDemo()\n", "\n", "\n", "if __name__ == \"__main__\":\n", " GlslApp().run()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "这么我们创建了一个部件`GlslDemo`；用来管理所有的渲染。`RenderContext`是`Canvas`的子类用来轻松替换着色器。`basic.glsl`文件包含点着色器和像素着色器，后面会实现。\n", "\n", "注意我们没用Kivy语言，因为没有布局要使用，所以这里不是`glsl.kv`文件，我们通过`GlslApp.build()`方法配置根部件。\n", "\n", "`EventLoop.ensure_window()`是必须的，因为我们需要接入OpenGL特性，包括在运行`GlslDemo.__init__()`方法时，调用GLSL编译器。如果此时没有应用窗口（更重要的是，没有相关的OpenGL内容），程序就会崩溃。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 建立几何图形" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "写着色器之前，我们还需要渲染一些东西——一系列点模型。我们用两个等斜边的直角三角形构成一个简单的矩形（这么分是因为基本多边形是三角形）。\n", "\n", ">Kivy更偏向于二维图形，所以在二维设计中不会强加任何限制。而OpenGL是源自三维图形，所以你可以用生动的模型来创建视频游戏，也可以结合Kivy的部件来创建UI。本书不做介绍，但是两者底层的机制是一样的。\n", "\n", "这里需要升级`GlslDemo`部件里的`__init__()`方法：" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def __init__(self, **kwargs):\n", " Widget.__init__(self, **kwargs)\n", " self.canvas = RenderContext(use_parent_projection=True)\n", " self.canvas.shader.source = \"basic.glsl\"\n", "\n", " fmt = ((b\"vPosition\", 2, \"float\"),) # Step 1\n", "\n", " vertices = ( # Step 2\n", " 0,\n", " 0,\n", " 255,\n", " 0,\n", " 255,\n", " 255,\n", " 0,\n", " 255,\n", " )\n", "\n", " indices = (0, 1, 2, 2, 3, 0) # Step 3\n", "\n", " with self.canvas:\n", " Mesh(fmt=fmt, mode=\"triangles\", indices=indices, vertices=vertices) # Step 4" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "方法解释如下：\n", "\n", "- 写`OpenGL`代码时需要注意点对象没有标准的格式，因此，我们需要定义一个。简单的做法就是用点的位置`vPosition`。我们的矩形是二维的，因此我们就传递两个坐标，默认是浮点型数值。因此，定义是`(b'vPosition', 2, 'float')`\n", "- 确定了点的格式之后，我们把这些点放到一个数组里，如`vertices = (...)`行所示。这个元组只有一层，后面会单独定义记录的格式，然后再把所有值聚合在一起，没有用分隔符或其他类似好记的名称。这是C语言结构体的经典方式\n", "- 索引数组用来复用那些顶点。通常，一个顶点被多个三角形同时使用。我们使用索引数组里的index来重复使用这些顶点——这样顶点数增加时内存占用相对更少\n", "- 有了这些数据结构之后，我们就用类似Kivy画布指令的`Mesh`把它们组合起来。它按照通常部件渲染的方式进行渲染，和其他Kivy部件有很好的兼容性。GLSL代码可以用来连接所有的部分\n", "\n", ">这章提到了一个C语言概念，`array（数组）`——存放同类数据的连续内存区域。Python数据结构里也有，不过，Python通常用`tuple（元组）`或`list（列表）`。" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "#### 解释索引" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "要理解OpenGL的索引（index），让我们举个例子。用前面代码里面的顶点，是按照`(x, y)`存放的：" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "vertices = (\n", " 0,\n", " 0,\n", " 255,\n", " 0,\n", " 255,\n", " 255,\n", " 0,\n", " 255,\n", ")" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "一个索引就是顶点列表里面的顶点数据，原点是`(0, 0)`。如下图所示：\n", "\n", "![openglindex](kbpic/8.2.openglindex.png)\n", "\n", "现在顶点还没有连起来，所以显示出来就是一些点，而不是一个多边形。我们定义一个`indices`列表来组合它们。三个点两组构成两个三角形：" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "indices = (\n", " 0,\n", " 1,\n", " 2, # 三个点构成一个三角形\n", " 2,\n", " 3,\n", " 0, # 另一个三角形\n", ")" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "这两个三角形中，第一个是由顶点0，1，2构成，第二个是由订单2，3，0构成。如下图所示，颜色是展示用的，我们还没有设置颜色，后面会补上。\n", "\n", "![triangles](kbpic/8.3.triangles.png)\n", "\n", "OpenGL代码里面的索引就是这样使用的。\n", "\n", ">OpenGL数据结构内存优化方法中很少是专门用来减少RAM的——很多时候视频接口吞吐量是性能的瓶颈，所以优化的目标都是每帧传递更多的内容，而不是通过压缩数据来节省内存。这点自始至终没有改变过。" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "### 写GLSL代码" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "下面我们就来写可以在GPU上执行的GLSL代码，它和C语言一样快。\n", "\n", "Kviy要求点着色器和像素着色器代码在一个文件里，代码用`'---vertex'`和`'---fragment'`分割，`$HEADER$`语句由Kivy指定，这并非任何标准，只在这里用：\n", "\n", "```C\n", "---vertex\n", "$HEADER$\n", "\n", "void main(void)\n", "{\n", " // vertex shader\n", " gl_Position = ...\n", "}\n", "\n", "---fragment\n", "$HEADER$\n", "\n", "void main(void)\n", "{\n", " // fragment shader\n", " gl_FragColor = ...\n", "}\n", "```\n", "\n", "为了节省篇幅，后面的代码会忽略这些样本代码，希望你看代码的时候记得它们是存在的。\n", "\n", "`$HEADER$`宏变量是上下文相关的，表示不同类型的着色器。\n", "\n", "在点着色器里面，`$HEADER$`是下面代码：\n", "\n", "```C\n", "varying vec4 frag_color;\n", "varying vec2 tex_coord0;\n", "attribute vec2 vPosition;\n", "attribute vec2 vTexCoords0;\n", "uniform mat4 modelview_mat;\n", "uniform mat4 projection_mat;\n", "uniform vec4 color;\n", "uniform float opacity;\n", "```\n", "\n", "在像素着色器里面，`$HEADER$`是下面代码：\n", "\n", "```C\n", "varying vec4 frag_color;\n", "varying vec2 tex_coord0;\n", "\n", "uniform sampler2D texture0;\n", "```\n", "\n", "有点啰嗦，以后的Kivy版本应该会简化它们。" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "#### 存储类与类型" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "在前面的代码里，不仅有数据类型，还有一个存储类：\n", "\n", "- 存储类（Storage classes）\n", " - `attribute`：由点数据格式指定对应每个点的属性，由应用传递\n", " - `uniform`：GLSL全局变量，同样由应用传递，但是不会随点变化\n", " - `varying`：有点着色器传递到像素着色器\n", "- 常用数据类型（Commonly used data types）\n", " - `float`：浮点类型\n", " - `vec2, vec3, vec4`：长度为2，3，4的元组类型，内部值为浮点类型。可以表示点、颜色等等\n", " - `mat2, mat3, mat4`：规模为 2 × 2，3 × 3，4 × 4的矩阵类型，\n", " - `sampler2D`：表示一个用来装饰的纹理（texture）类型" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "#### 基本着色器" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "现在，让我们来写一个简单的着色器：\n", "```C\n", "void main(void)\n", "{\n", " vec4 pos = vec4(vPosition.xy, 0.0, 1.0);\n", " gl_Position = projection_mat * modelview_mat * pos;\n", "}\n", "```\n", "\n", "把每个点的坐标值转换成Kivy系统的坐标值，其原点在左下角。\n", "\n", ">这里不演示坐标变换的细节，作为入门教程有点复杂。其实也没必要完全理解这些细节，或者读完整本书。\n", ">如果你很感兴趣，可以去看看[OpenGL的坐标空间和矩阵用法](http://www.learnopengles.com/understanding-opengls-matrices/)\n", "\n", "最简单的像素着色器就是一个返回固定颜色的函数：\n", "\n", "```C\n", "void main(void)\n", "{\n", " gl_FragColor = vec4(1.0, 0.0, 0.5, 1.0);\n", "}\n", "```\n", "\n", "这里为每个点返回一个RGBA颜色值`#FF007F`。\n", "\n", "如果你运行程序，你会看到输出结果如下图所示：\n", "\n", "![basicshaders](kbpic/8.4.basicshaders.png)\n", "\n", "聊胜于无，折腾这么久，终于有点儿成果了，下面我们调整一下看看变化。" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "#### 程序的颜色" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "除了显示一种颜色外，颜色还有可以通过另一个方法让像素着色器显示出来。\n", "\n", "假设我们想要计算每个像素的RGB颜色：\n", "\n", "- `R`通道值与`x`轴坐标成正比\n", "- `G`通道值与`y`轴坐标成正比\n", "- `B`通道值等于`R`与`G`均值\n", "\n", "对应最简单像素着色器的算法如下：\n", "\n", "```C\n", "void main(void)\n", "{\n", " float r = gl_FragCoord.x / 255.0;\n", " float g = gl_FragCoord.y / 255.0;\n", " float b = 0.5 * (r + g);\n", " gl_FragColor = vec4(r, g, b, 1.0);\n", "}\n", "```\n", "`gl_FragCoord`变量包括相对于应用窗口的像素坐标值（并非实际屏幕的坐标值）。这里除以`255.0`是为了简化计算，让所有的值都落在[0，1]区间内。\n", "\n", "效果图像如下：\n", "\n", "![computingcolor](kbpic/8.5.computingcolor.png)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "#### 彩色顶点" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "类似的彩色效果可以通过给点增加颜色数据来实现。因此，我们需要扩展点的数据格式，增加一个带颜色的属性`vColor`：" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "fmt = (\n", " (b\"vPosition\", 2, \"float\"),\n", " (b\"vColor\", 3, \"float\"),\n", ")\n", "\n", "vertices = (\n", " 0,\n", " 0,\n", " 0.462,\n", " 0.839,\n", " 1,\n", " 255,\n", " 0,\n", " 0.831,\n", " 0.984,\n", " 0.474,\n", " 255,\n", " 255,\n", " 1,\n", " 0.541,\n", " 0.847,\n", " 0,\n", " 255,\n", " 1,\n", " 0.988,\n", " 0.474,\n", ")\n", "\n", "indices = (0, 1, 2, 2, 3, 0)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "更新之后，每个点都有5个维度，后面三个是RGB颜色值。对应的`vertices`也要做修改。\n", "\n", "`vColor`属性是一个RGB颜色值，而点着色器用的是RGBA颜色，这里并没有每个点的alpha通道值，我们需要对点着色器做一些调整：\n", "\n", "```C\n", "attribute vec3 vColor;\n", "\n", "void main(void)\n", "{\n", " frag_color = vec4(vColor.rgb, 1.0);\n", " vec4 pos = vec4(vPosition.xy, 0.0, 1.0);\n", " gl_Position = projection_mat * modelview_mat * pos;\n", "}\n", "```\n", "\n", ">在GLSL中，`vColor.rgb`和`vPosition.xy`表示法叫混合（swizzling）。它们可以有效的获取矢量的一部分，类似于Python的切片（slice）。\n", "\n", ">这里的`vColor.rgb`表示“取矢量的前三个值”，在Python里面就是`vColor[:3]`。还可以颠倒顺序，如`vColor.bgr`，甚至重复，如`vColor.ggg`（取三个`G`通道值），样式很灵活。\n", "\n", ">同理，可以取矢量的四个值，如`.xyzw`，`.rgba`或`.stpq`。\n", "\n", "那么，像素着色器就很简单了：\n", "\n", "```C\n", "void main(void)\n", "{\n", " gl_FragColor = frag_color;\n", "}\n", "```\n", "\n", "让点与点之间的颜色插值计算，就呈现平滑渐变的效果了，这就是OpenGL的工作方式，如下图所示：\n", "\n", "![passingcolor](kbpic/8.6.passingcolor.png)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "#### 纹理映射" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "下面再给我们的矩形增加一些纹理。我们还要扩展点数据格式的定义，给每个点增加纹理坐标值：" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "fmt = (\n", " (b\"vPosition\", 2, \"float\"),\n", " (b\"vTexCoords0\", 2, \"float\"),\n", ")\n", "\n", "vertices = (\n", " 0,\n", " 0,\n", " 0,\n", " 1,\n", " 255,\n", " 0,\n", " 1,\n", " 1,\n", " 255,\n", " 255,\n", " 1,\n", " 0,\n", " 0,\n", " 255,\n", " 0,\n", " 0,\n", ")" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "纹理坐标值通常都在[0,1]区间内，原点在左上角，这和Kivy的左下角原点是不同的。使用过程中，需要注意这个差别。\n", "\n", "下面是Python加载纹理并传递给渲染的过程：" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from kivy.core.image import Image\n", "\n", "with self.canvas:\n", " Mesh(\n", " fmt=fmt,\n", " mode=\"triangles\",\n", " indices=indices,\n", " vertices=vertices,\n", " texture=Image(\"kivy.png\").texture,\n", " )" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "这样就把当前文件夹内的`kivy.png`文件转换成纹理了。如下图所示：\n", "\n", "![texture](kbpic/8.7.texture.png)\n", "\n", "这和前面的着色器没啥不同。点着色器传递纹理的坐标值：\n", "\n", "```C\n", "void main(void)\n", "{\n", " tex_coord0 = vTexCoords0;\n", " vec4 pos = vec4(vPosition.xy, 0.0, 1.0);\n", " gl_Position = projection_mat * modelview_mat * pos;\n", "}\n", "```\n", "\n", "像素着色器用修改过的`tex_coord0`坐标值来装饰在`texture0`位置的纹理，于是返回对应的颜色：\n", "\n", "```C\n", "void main(void)\n", "{\n", " gl_FragColor = texture2D(texture0, tex_coord0);\n", "}\n", "```\n", "\n", "把代码放到一起就看可以到如下效果：\n", "\n", "![texturemapping](kbpic/8.8.texturemapping.png)\n", "\n", "通过着色器的实现经历，你可以去做一些相关程序了。如果有的地方不明白不用郁闷，GLSL确实比较复杂，系统的学习它得费一番功夫。\n", "\n", "但是，它可以让你更清楚底层的工作细节。即使你日常工作中没写过底层代码，你也可以掌握这些知识，以避免性能瓶颈，改善程序的架构。" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## 做满天星app" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "学习了一堆GLSL的知识后，我们来做一个满天星屏保的app，星星会从屏幕的中央向四边飞来，给人一种穿越星空的快感。\n", "\n", ">动态效果用图片没法展示，你可以直接运行实例代码看看效果。\n", "\n", "每颗星都会做如下动作：\n", "\n", "1. 在屏幕中央随机生成\n", "2. 从屏幕中央向四周飞去直到看不见为止\n", "3. 然后从屏幕中央重新生成\n", "\n", "我们还会让星星的尺寸在飞行中不断变大。如下图所示：\n", "\n", "![screenshot](kbpic/8.9.screenshot.png)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "### 应用架构" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "新应用类的架构借用前面章节的内容即可。这里同样不用Kivy语言描述部件层级，所以没有`starfield.kv`。Python代码如下：" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from kivy.base import EventLoop\n", "from kivy.clock import Clock\n", "\n", "\n", "class StarfieldApp(App):\n", " def build(self):\n", " EventLoop.ensure_window()\n", " return Starfield()\n", "\n", " def on_start(self):\n", " Clock.schedule_interval(self.root.update_glsl, 60 ** -1)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "这里`build()`方法创建并返回根部件`Starfield`；它将控制所有的数学和渲染应用中的内容。\n", "\n", "`on_start()`事件handler在程序启动后，让根部件通过`update_glsl()`方法实现每秒更新60次。\n", "\n", "`Starfield`类还分成两部分：`__init__()`方法用来创建数据结构，`update_glsl()`方法呈现图像（计算每颗星的位置）并渲染星星。" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "### 数据结构与初始化" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "初始化代码如下：" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from kivy.core.image import Image\n", "from kivy.graphics.instructions import RenderContext\n", "from kivy.uix.widget import Widget\n", "\n", "NSTARS = 1000\n", "\n", "\n", "class Starfield(Widget):\n", " def __init__(self, **kwargs):\n", " Widget.__init__(self, **kwargs)\n", " self.canvas = RenderContext(use_parent_projection=True)\n", " self.canvas.shader.source = \"starfield.glsl\"\n", "\n", " self.vfmt = (\n", " (b\"vCenter\", 2, \"float\"),\n", " (b\"vScale\", 1, \"float\"),\n", " (b\"vPosition\", 2, \"float\"),\n", " (b\"vTexCoords0\", 2, \"float\"),\n", " )\n", "\n", " self.vsize = sum(attr[1] for attr in self.vfmt)\n", "\n", " self.indices = []\n", " for i in range(0, 4 * NSTARS, 4):\n", " self.indices.extend((i, i + 1, i + 2, i + 2, i + 3, i))\n", "\n", " self.vertices = []\n", " for i in range(NSTARS):\n", " self.vertices.extend(\n", " (\n", " 0,\n", " 0,\n", " 1,\n", " -24,\n", " -24,\n", " 0,\n", " 1,\n", " 0,\n", " 0,\n", " 1,\n", " 24,\n", " -24,\n", " 1,\n", " 1,\n", " 0,\n", " 0,\n", " 1,\n", " 24,\n", " 24,\n", " 1,\n", " 0,\n", " 0,\n", " 0,\n", " 1,\n", " -24,\n", " 24,\n", " 0,\n", " 0,\n", " )\n", " )\n", "\n", " self.texture = Image(\"star.png\").texture\n", "\n", " self.stars = [Star(self, i) for i in range(NSTARS)]" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "`NSTARS`是星星的总数，调整它会改变星星的密度。一个带Intel集成显卡的中档电脑可以轻松支持几千个星星。再高档点的显卡带几万个星星也没问题。\n", "\n", "和前面的例子不同，这次我们不在最后实现索引和点的数组。在初始化阶段我们就设置好，方便后面`update_glsl()`方法更新。\n", "\n", "`vfmt`顶点格式包括四个属性，如下表所示：\n", "\n", "|**属性**|**功能**|\n", "|:--:|:--:|\n", "|`vCenter`|星星中点在屏幕上的坐标值|\n", "|`vScale`|星星的尺寸，1表示（48 × 48 像素）|\n", "|`vPosition`|每个顶点与星星中点的相对位置|\n", "|`vTexCoords0`|纹理坐标值|\n", "\n", "还有个属性`vsize`是顶点数组`vfmt`里的单个顶点的总长度。它计算顶点格式`vfmt`中间列的总和。\n", "\n", "`vertices`列表包含需要的数据；但它没有层次，不方便操作。这里就做了一个`Star`辅助类来实现，它把细节封装起来，在顶点数组外加了一个类，这样可以省不少事儿。\n", "\n", "`Star`类还会持续跟踪不属于顶点数据格式的一些属性，极坐标（以中心为原点的`angle`角度和`distance`距离）和不断增大的`size`。\n", "\n", "`Star`类定义如下：" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import math\n", "from random import random\n", "\n", "\n", "class Star:\n", " angle = 0\n", " distance = 0\n", " size = 0.1\n", "\n", " def __init__(self, sf, i):\n", " self.sf = sf\n", " self.base_idx = 4 * i * sf.vsize\n", " self.reset()\n", "\n", " def reset(self):\n", " self.angle = 2 * math.pi * random()\n", " self.distance = 90 * random() + 10\n", " self.size = 0.05 * random() + 0.05" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "`base_idx`是星星点数组的第一个顶点，还要加一个引用`sf`，方便`Starfield`实例连接`vertices`。\n", "\n", "`reset()`调用之后把星星的属性复原。" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "### 推动图像" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "`Starfield.update_glsl()`方式实现了星星运动的算法，被`on_start()`事件handler的Kivy时钟程序频繁调用。源代码如下：" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from kivy.graphics import Mesh\n", "\n", "\n", "def update_glsl(self, nap):\n", " x0, y0 = self.center\n", " max_distance = 1.1 * max(x0, y0)\n", "\n", " for star in self.stars:\n", " star.distance *= 2 * nap + 1\n", " star.size += 0.25 * nap\n", "\n", " if star.distance > max_distance:\n", " star.reset()\n", " else:\n", " star.update(x0, y0)\n", "\n", " self.canvas.clear()\n", "\n", " with self.canvas:\n", " Mesh(\n", " fmt=self.vfmt,\n", " mode=\"triangles\",\n", " indices=self.indices,\n", " vertices=self.vertices,\n", " texture=self.texture,\n", " )" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "首先星星从屏幕中央产生之后，我们计算最大距离`max_distance`。然后，我们重复星星列表，让它们运动起来，然后不断放大。超出屏幕的星星被重置。\n", "\n", "函数的最后看着很熟悉，那是前面讲过的渲染技巧。调用`canvas.clear()`是必须的，但是每次调用都增加一个新的匹配（Mesh），极快的推送到显卡上进行处理。\n", "\n", "代码的最后内容是`Star.update()`方法。它更像星星的四个顶点，把新的坐标值写到`vertices`数值中：" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def iterate(self):\n", " return range(self.j, self.j + 4 * self.sf.vsize, self.sf.vsize)\n", "\n", "\n", "def update(self, x0, y0):\n", " x = x0 + self.distance * math.cos(self.angle)\n", " y = y0 + self.distance * math.sin(self.angle)\n", " for i in self.iterate():\n", " self.sf.vertices[i : i + 3] = (x, y, self.size)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "`iterate()`辅助函数用来提高代码可读性，看着有点多，其实读起来更方便。\n", "\n", "为了完成程序迭代，整个内存映射进程会把序列化每一帧里大量的对象作为一个重要目标来处理，这会提高性能。" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "### 写GLSL代码" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "着色器代码和前面类似。点着色器代码：\n", "\n", "```C\n", "attribute vec2 vCenter;\n", "attribute float vScale;\n", "\n", "void main(void)\n", "{\n", " tex_coord0 = vTexCoords0;\n", " mat4 move_mat = mat4\n", " (1.0, 0.0, 0.0, vCenter.x,\n", " 0.0, 1.0, 0.0, vCenter.y,\n", " 0.0, 0.0, 1.0, 0.0,\n", " 0.0, 0.0, 0.0, 1.0);\n", " vec4 pos = vec4(vPosition.xy * vScale, 0.0, 1.0) * move_mat;\n", " gl_Position = projection_mat * modelview_mat * pos;\n", "}\n", "```\n", "\n", "我们通过`vScale`因子与顶点坐标值相乘来等比例放大，然后用`vCenter`属性把它们变换成位置。`move_mat`矩阵是变换矩阵，线性代数里的一种放射变换方法。\n", "\n", "像素着色器代码：\n", "\n", "```C\n", "void main(void)\n", "{\n", " gl_FragColor = texture2D(texture0, tex_coord0);\n", "}\n", "```\n", "\n", "最终的效果如下图所示：\n", "\n", "![star](kbpic/8.10.star.png)\n", "\n", "这样，满天星app就完成了，感受一下穿越的乐趣吧。" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## 总结" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "这一章我们介绍了底层的OpenGL硬件加速方式，用GLSL实现了点，索引和着色器。\n", "\n", "GPU直接编程是一个极其强大的概念，这种强大也需要大量付出。着色器比普通的Python代码要难得多，调试工作就更复杂了，也没有一个像Python的REPL那样方便的交互开发环境。也就是说，做应用的时候，写原始的GLSL代码是否必要并没有答案，只能因地制宜。\n", "\n", "这章的例子只是一个简单教程，并不是能力测试。因为 GLSL非常难学，大量的书籍和在线教程都已经介绍过它，这里的内容不足以完整的介绍OpenGL的内容，若感兴趣可以看看。\n", "\n", "现在，我们算是入门了。下一章我们利用这章的代码来做一个更好玩的应用，一个酷炫的射击游戏。" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.7" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": false, "sideBar": false, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": false, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 1 }