{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Wirting A Lexer In Nim\n", "\n", "- 2018/11/28 \n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Table of Contents\n", "\n", "1. Overview\n", "2. Why Nim?\n", "3. Wat is an Interpreter?\n", "3. What is a Lexer?\n", "4. What I have learned\n", "5. Impression" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Overview\n", "\n", "![](https://www.oreilly.co.jp/books/images/picture_large978-4-87311-822-2.jpeg)\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Writing An Interpreter In Go\n", "\n", "- Make a Programming Language called \"Monkey\" \n", "- Make a REPL\n", " - Lexer\n", " - Parser\n", " - Evaluator(?)\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Why Nim ?\n", "\n", "- Not Use\n", " - Go (same as this book)\n", " - Rust (very difficult!)\n", " - C, C++ (don't like!)\n", "- Very Fast\n", " - [benchmarking](https://mrsekut.site/?p=1591)\n", "- Low Learning Cost\n", "- Interesting\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# What is an Interpreter ?\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Language Processing System\n", "\n", "- Compiler\n", "> translate source code from a high-level programming language to a lower level language\n", "- Interpreter\n", "> 1. parse the source code and perform its behavior directly; \n", "> 2. translate source code into some efficient intermediate representation and immediately execute this; \n", "> 3. explicitly execute stored precompiled code made by a compiler which is part of the interpreter system.\n", "\n", " - ex. 1: Lisp, Brainfuck, ..\n", " - ex. 2: Python, Ruby, ..\n", " - ex. 3: Java, Basic, ..\n", "\n", "- [Compiler - Wikipedia](https://en.wikipedia.org/wiki/Compiler)\n", "- [Interpreter (computing) - Wikipedia](https://en.wikipedia.org/wiki/Interpreter_(computing))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Interpreter\n", "\n", "- make AST and run\n", "\n", " \n", "![](https://1.bp.blogspot.com/-BmT3MCbTUfw/V2f0Zu9VsyI/AAAAAAAAFuI/FoBiF336ZmwHL6mZYqWB0j8dpkH_jYaVACLcB/s640/6c4385fbe3d8471982c9b2a030106d38.png)\n", "- by [Positive Technologies - learn and secure : Theory and Practice of Source Code Parsing with ANTLR and Roslyn](http://blog.ptsecurity.com/2016/06/theory-and-practice-of-source-code.html)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## REPL?\n", "\n", "- Read, Eval, Print, Loop\n", "- Ptyhon, JavaScript, Haskell, etc.\n", "\n", "### ex\n", "\n", "- `> python`: REPL\n", "- `> python hoge.py`: no REPL" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# What is a Lexer?\n", "\n", "- Create Tokens from Input\n", "- Token is the samllest unit of program" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Example \n", "\n", "#### Input\n", "\n", "```\n", "let five = 5\n", "let ten = 10\n", "```" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "#### Output\n", "\n", "```\n", "[\n", " LET,\n", " IDENT(\"five\"),\n", " ASSIGN,\n", " INT(5),\n", " LET,\n", " IDENT(\"ten\"),\n", " ASSIGN,\n", " INT(10),\n", "]\n", "```\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Examples of Other Tokens\n", "\n", "- PAREN: `(`,`)`\n", "- BRASE: `{`,`}`\n", "- LT, GT: `<`,`>`\n", "- PROC: `proc`\n", "- IF: `if`\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## use Case..of \n", "\n", "```\n", "proc nextToken*(self: Lexer): token.Token =\n", " var tok: token.Token\n", " self.skipWhiteSpace()\n", "\n", " case self.ch\n", " of ':':\n", " tok = newToken(COLON, self.ch)\n", " of '(':\n", " tok = newToken(LPAREN, self.ch)\n", " of ')':\n", " tok = newToken(RPAREN, self.ch)\n", " of ',':\n", " tok = newToken(COMMA, self.ch)\n", " of '+':\n", " tok = newToken(PLUS, self.ch)\n", " else:\n", " if isLetter(self.ch):\n", " let l = self.readIdentifier()\n", " let t = LookUpIdent(l)\n", " return Token(Type: t, Literal: l)\n", " elif isDigit(self.ch):\n", " let t = token.INT\n", " let l = self.readNumber()\n", " return Token(Type: t, Literal: l)\n", " else:\n", " tok = newToken(token.ILLEGAL, self.ch)\n", "\n", " self.readNextChar()\n", " tok\n", "```" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# What I have learned\n", "\n", "- two character token need to peek\n", " - `=` and `==`\n", " - `!` and `!=`\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### use IF in Case..of\n", "\n", "```\n", "case self.ch\n", "of '=':\n", " if self.peekChar() == '=':\n", " let ch = self.ch\n", " self.readNextChar()\n", " let l = $ch & $self.ch # make `==`\n", " tok = Token(Type: EQ, Literal: l)\n", " else:\n", " tok = newToken(ASSIGN, self.ch) # `=`\n", "```" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Impression\n", "\n", "### Test\n", "\n", "- test first\n", "\n", "### Types\n", "\n", "- Which is more difficult than Dynamic or Static typing\n", "\n", "### Grammar\n", "\n", "- not use `{`,`}`\n", "- a nest with indents like Python or Nim\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Next Step...\n", "\n", "- make a Parser!!\n", "- need to learn Nim's ptr or interface" ] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.1" }, "toc": { "nav_menu": {}, "number_sections": false, "sideBar": true, "skip_h1_title": false, "toc_cell": false, "toc_position": {}, "toc_section_display": "block", "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }