% NGSLANG(1) NGS User Manual % Ilya Sher % 2015 # NAME ngslang - Next Generation Shell language reference. # What is NGS? NGS is an alternative shell. At its core is a domain-specific language that was specifically designed to be a shell language. NGS is under development. The language part is already good enough to write some useful scripts. The CLI still doesn't exist and will be written using the same language. # Running NGS scripts **ngs** *script_name.ngs* You can put the following line as first line of your script: #!/usr/bin/env ngs If you do that, you can run the script as `./script_name.ngs` or `/full/path/to/script_name.ngs` (you must make your script executable, by running `chmod 755 script_name.ngs` ). See more about running NGS in [NGS(1)](ngs.1.md). # Terminology * type - Built-in or user-defined data type, similar to Python, Ruby and other languages. * object - Instance of a type, similar to Python, Ruby and other languages. The phrase "MyType object" refers to an Instance of "MyType". * field - A named slot of an object, similar to field in Python, Java, etc. * meta - Slot for auxiliary data on most types of objects (the ones that are references, i.e. not Int,Bool,Null), typically a `Hash` or `null`. Meta information is not fields and is accessed differently (using `meta` method). * method - Built-in or user-defined function. User defined methods can be closures. Methods are also called functions in several places in the documentation. * multimethod - A MultiMethod object containing ordered list of methods. When called, the appropriate method is selected from the list to perform the computation. # Language principles This section is about principles behind NGS language design. ## Systems engineers' language NGS is a domain-specific language. It is aimed to solve common system tasks in a convenient manner. Some examples: * `fetch('myfile.json')` will decode the JSON and return the data structure. (Use `read()` to get raw contents). * The ```` ``my_command`` ```` will decode the command output (JSON for example) and return the data structure. Note that ```` ``aws ...`` ```` will be parsed even further (not just JSON) to return more usable data structures. * `my_array.my_prop` returns an array of `.my_prop` properties of each element. ## Uniformity NGS tries to be uniform wherever possible to minimize surprises. The idea here is to match between expected and actual behaviour of given method plus parameters. ## Power As a rule, trade-offs between power and not allowing to shoot yourself in the foot are resolved in favor of the power solution. The language is aimed at experienced engineers which use their own judgement. The language should be powerful enough to shoot all feet in the building at once. ## Simple methods naming for less guess work For example, the multimethod `+`: * `1 + 2` adds the numbers * `arr1 + arr2` adds (concatenates) arrays * `hash1 + hash2` produces merged hash. ## Extensibility * `fetch('your_file.super-format')` can be extended to decode your format. * `read`, which reads from a file, can be extended to support HTTP or S3. * Most of the syntax (for example `my_var.my_field` or `my_var[my_index]`) is just sugar for calling methods. This behaviour lets you, the user, define any operator for existing or your custom types. ## Simplicity Very small number of core concepts in the language: * Types with a simple type system, geared only toward multiple dispatch. No classes. * Multimethods, which allow using same method name for operations on different types, as in the `+` example above. * Closures * Exceptions ## Familiarity Many concepts and syntax constructs come from other languages. # Naming convention * `var_name` * `CONST_NAME` - Currently NGS does not support constants, it's only a naming convention * `method_name` (multimethod name) * `TypeName` * `TransformationName` - example: `Strs` (converts to array of strings), `Argv` (constructs command line arguments array), `ExitCode` (converts anything to integer exit code). Reasoning behind `TransformationName`: * Transforms data into something else, like many other constructors * The output data might get it's own data type some day # The two syntaxes NGS has two syntaxes: **commands syntax** and **code syntax**. Motivation: I can not imagine any syntax for a "real" programming language that I would like which is built on top of the **commands syntax**, which is similar to bash. That's why NGS has **code syntax** for "real" programming and **commands syntax** for running external programs and i/o redirection, where this syntax is very convenient. ## Commands syntax This is the resembles-bash syntax geared towards running external programs and i/o redirection. Command syntax is the syntax at the top level of every NGS script. The most simple NGS scripts might look very similar to bash scripts. Commands are separated by either newlines or by semicolon (`;`). cat a.txt; touch myfile echo mystr >myfile ls | wc -l ## Code syntax Code syntax resembles other high-level languages such as Python or Ruby. Example: 1 + 2 * 3; %[abc def ghi].without('ghi').each(echo) Expressions are separated by either newlines or by semicolon (`;`). It is also permitted to have trailing semicolon in the end of a line: my_func(); my_other_func() **code syntax** is the syntax of **-e**, **-E**, **-p**, **-pi**, **-pl**, **-pj**, **-pjl**, and **-pt** switches to `ngs` interpreter. Example: $ ngs -p '(1...10).filter(F(num) num % 2 == 0)' # Output: [2,4,6,8,10] ## Switching between commands and code syntaxes In **commands syntax** it is possible to switch to **code syntax** in one of the following ways: ls { code syntax here } ls ${ code that computes the file name and returns a string, spaces don't matter, expanded into single argument of ls } # Expands to zero or more positional arguments to ls ls $*{ code that computes the files names and returns array of strings. Spaces don't matter, each element is expanded into single argument of ls } In **code syntax** it is possible to switch to **commands syntax** in one of the following ways: # Capture output out = `commands syntax` my_var = "mystring\n" + `my other command` # Capture output and decode it parsed_data_structure = ``commands syntax`` n = ``curl https://example.com/myservice/stats``.number_of_items_in_storage # Get reference to a process my_process = $( commands syntax ) # Comments Comments are allowed in both commands and code syntax. At the beginning of a line, a comment starts with a hash sign (`#`). A comment that starts in a middle of the line starts with space and a hash sign (`#`) and continues to end of the line. # comment till end of line run_external_program --input my_file # COMMENTS HERE ARE NOT IMPLEMENTED YET { a = 1 # comment till end of line } Also, lines that start with "TEST " (note the trailing space) are considered to be comments. They are ignored. The purpose is to allow an external testing script to extract and run these tests. The common pattern of marking code sections with comments got it's own syntax. It conveys semantic meaning of sections to the language and IDEs. That will allow additional related functionality in the future. Section does *not* introduce a new scope for variables/methods. section "Workaround for API stupidity" { if result is Null { result = [] } if result is Str { result .= split(',') } } # Variables As in other languages, variables are named locations that can store values. Variables' names should consist of ASCII letters (a-z, A-Z) and numbers (0-9). Variable name must start with a letter. Assignment to variables looks the same in both commands and code syntax. Referencing a variable in the code syntax is just the variable's name while referencing it in commands syntax or inside string interpolation is `$my_var` (not recommended) or `${my_var}` (recommended). Advanced topic: more precisely, the naming restrictions for the variables mentioned above are naming restrictions on identifiers. NGS can have variables that are named not by the rules above. Note that it is not recommended to use names which are not valid identifiers. As an exception to this rule, it's OK for methods which correlate with NGS syntax to have names which are not valid identifiers. Example of such methods are binary operators (`+`, `-`, etc.). See more about methods' naming in "Methods and multimethods" section below. Assigning to a variable works in both commands and code syntax. # Assigning value to a variable. # Note that syntax to the right of = is code syntax even if the surrouning syntax is commands syntax. a = 1 + 2 # Referencing a variable in code syntax. # Not that arguments to method call are in code syntax even if the surrouning syntax is commands syntax. echo(a) # Output: # 3 my_file = '1.txt' # Referencing a variable inside commands syntax. $(ls $my_file) $(ls ${my_file}) # Referencing a variable inside string interpolation. echo("A is ${a}") # Output: # A is 3 # In-place assignment ( -=,+=,*=,/=,%= ) a += 100 # The above is exactly the same as a = a + 100 Referencing undefined variable will cause `GlobalNotFound` or `UndefinedLocalVar` exceptions. ## Variables' scoping rules Variables scoping is similar to Python's. Variables scope types: * `local` - local to a method * `upvar` - local to enclosing method * `global` - global In a method, any variable that is not assigned to inside the method is looked up as an `upvar` (enclosing methods) and as `global`. a = 1 F f() { echo(a) } f() # Output: 1 a = 1 F f() { a = 2 F g() { echo(a) } g() } f() # Output: 2 In a method, any variable that is mentioned in any of the enclosing methods is automatically `upvar` - references the variable in the outer scope. a = 1 F f() { a = 2 F g() { a = 10 } g() echo(a) } f() echo(a) # Output: 10 # Output: 1 In a method, any variable that is assigned to (including the `i` in construct `for(i;10) ...`) in the method is automatically `local` unless it's an `upvar` as described above. Inner methods are also `local` by default. a = 1 F f() { a = 2 echo(a) F inner_func() "something" } f() echo(a) # Output: 2 # Output: 1 # Can not call inner_func from outside f() ## Modifying variables' scoping You can modify default scoping using the `global` and `local` keywords. a = 1 F f() { a = 2 F g() { # local instead of upvar local a a = 3 } g() echo(a) } f() # Output: 2 a = 1 F f() { a = 2 F g() { # global instead of upvar global a a = 3 } g() echo(a) } f() # Does not work yet due to a bug, "a" stays upvar # Output: 3 ## Planned features for variables Destructuring assignment. Something like the following: [a, b, *rest] = my_arr Syntax for `Hash` destructuring is not clear yet: ... = my_hash Destructuring should also be available in `for`: # Hash iterator gives Arr of two elements each iteration. # Destructuring would place the two elements in k and v varaibles. for [k, v] in my_hash { ... } # Methods and multimethods Methods' names are composed of letters (a-z, A-Z), digits (0-9) and the following characters: `_`, `-`, `|`, `=`, `!`, `@`, `?`, `<`, `>`, `~`, `+`, `*`, `/`, `%`, `(`, `)`, `$`, `.`, ```` `` ````, `"`, `:`, (space), `[`, `]` . ## Multi-dispatch Each value in NGS has a type, similar to many other languages. One of the main features of NGS is choosing the correct method of a multimethod, based on types of the arguments. This feature is called multi-dispatch. Let's start with the following example: F +(a:Int, b:Int) { ... } F +(s1:Str, s2:Str) { ... } { 1 + 1 # 2 'a' + 'b' # 'ab' } The `+` is a multimethod. It has a few methods. You can see definitions of two of the methods in the example above. One method adds numbers. Another method concatenates strings. How NGS knows which one of them to run? The decision is made based on arguments' types. NGS scans the methods list backwards and invokes the method that matches the given arguments (this matching process is called multiple dispatch). When a `MultiMethod` is called, the multimethod's methods list is searched from last element to first element. The method which parameters' types match is executed. This model is different from many other languages, where the most specific method is called. In NGS, if two methods' parameters both match the arguments, that one that was defined last is called. Typically, methods for more specific types are defined later in code so the behaviour is similar to other languages. If matched method is executed and a `guard` (see below) fails, the search is continued as if the method did not match. Consider the following types: { type Vehicle type Car(Vehicle) } Typical MultiMethod definition: F park(v:Vehicle) { ... } F park(c:Car) { ... } park(Vehicle()) # calls the first method park(Car()) # calls the second method Leveraging the NGS MultiMethod call behaviour: one can for example do debugging by adding the third method to the `park` MultiMethod: F park(v:Vehicle) { debug("Parking vehicle ${v}") super(v) # Execute one of the two methods defined above } ## Calling a method - simple syntax This syntax is available in both commands and code syntax. echo('Hello world') # Output: # Hello world ## Calling a method - object-style syntax This syntax is available in code syntax only. As a syntactic sugar, method call `f(a, b, c)` can be written as `a.f(b, c)`. { 'Hello world'.echo() } # Output: # Hello world ## Methods for operators In NGS, binary operators are just syntactic sugar for calling methods. The variety of permitted characters in methods' names allow using same method name as the operator. # Define the + operator for Fun objects (Fun objects are callable objects: methods and types) F +(f:Fun, g:Fun) { F composed_function(*args) { f(g(*args)) } } ... F reject(h:Hash, predicate:Fun) h.filter(not + predicate) Current methods with special names are listed below. # Output wrapped manually for your convenience. # All of them correspond to a syntax, mostly binary operators. $ ngs -pl 'globals().filterv(Fun).keys().filter(/[^a-zA-Z0-9_]/)' | sort | column -x -c 40 != !== !~ "$*" $() % %() * + +? - -> . .. ... .= / // :: ::= < <= == === =~ > >= ? [] []= \ `` ```` is not not in ~ ~~ ## Binary operators (methods) and precedence Higher numbers mean higher precedence. Note that short-circuit operators can not and are not implemented as method calls - that would require eager evaluation of the arguments. tor 40 "Try ... or", short-circuit questionable_code tor default_value tand 50 "Try ... and", short-circuit (not sure when it's needed, don't use it) or 60 Logical or, short-circuit and 65 Logical and, short-circuit in 70 Value-in-container check 1 in [1, 2, 3] "a" in {"a": 1} not in 70 Value-not-in-container check 10 not in [1, 2, 3] "b" not in {"a": 1} is 90 Instance-of check 1 is Int is not 90 Not-instance-of check 1 is not Str | 120 "Pipe", currenty not used === 130 "Same as" v = [1, 2]; v === v !== 130 "Not same as", [1, 2] !== [1, 2] == 130 "Equals", [1, 2] == [1, 2] != 130 "Not equals" [1, 3] != [1, 2] =~ 130 "Matches" {"a": 1, "b": 2} =~ {"a": 1} # true {"a": 1, "b": 2} =~ {"a": Any} # true !~ 130 "Does not match" {"a": 1, "b": 2} !~ {"z": 1} # true <= 150 "Less than or equals" < 150 "Less than" >= 150 "Greater or equals" > 150 "Greater" ~ 150 "Match" "a1b2c" ~ /[0-9]/ # MatchSuccess ~~ 150 "Match all" "a1b2c" ~~ /[0-9]/ # Arr[MatchSuccess] ... 160 "Inclusive range" 0...5 # 0,1,2,3,4,5 .. 160 "Exclusive range" 0..5 # 0,1,2,3,4 + 190 "Plus" +? 190 "Plus maybe" "a" + "b" # "ab" null + "b" # null "a" + null # null - 190 "Minus" * 200 "Multiply" or "repeat" 3 * 5 # 15 "ab" * 3 # "ababab" EmptyBox * 2 # two values of EmptyBox type % 200 "Modulus" or "each" 3 % 2 # 1 ['a', 'b'] % echo # Outputs a and b on different lines / 200 "Divide" or "map" 10 / 5 [1, 2, 3] / F(x) x * 2 ? 200 "Filter" [1, 2, 3] ? F(x) x > 1 \ 200 "Call" [1, 2, 3] \ echo ## Defining a method Method definition works in both commands and code syntax. When defining a named method, NGS automatically creates a `MultiMethod` with the given name (if it does not exist) and appends the new method to the multimethod's list of methods. F my_method() { echo("wassup?") echo("my method is running") } Method with optional parameters F mysum(a:Int, b:Int=100) a+b echo(mysum(5)) # Output: 105 echo(mysum(5, 200)) # Output: 205 Method with "rest" parameter F print_with_prefix(prefix:Str, *strings) { strings.each(F(s) { echo("$prefix$s") }) echo("Printed ${strings.len()} lines with prefix") } print_with_prefix('-> ', 'abc', 'def') # Output: # -> abc # -> def # Printed 2 lines with prefix Method with "rest keywords" parameter F print_properties(separator:Str, **kw_args) { kw_args.each(F(name, value) { echo("$name$separator$value") }) } print_properties(' => ', a=10, b=20) # Output: # a => 10 # b => 20 ## Method guards Method guards restrict method execution to specific conditions, typically the conditions are functions of parameters' values. If the guard condition evaluates to `false`, NGS considers the method as if it did not match the arguments and continues to search for other methods to run in the multimethod's methods list. F gg(i:Int) { echo("First gg active") echo(i*10) } # gg is now MultiMethod with one method gg(1) gg(5) # Output: # First gg active # 10 # First gg active # 50 F gg(i:Int) { echo("Second gg checking guard") guard i > 3 echo("Second gg active") echo(i*100) } # gg is now MultiMethod with two methods gg(1) gg(5) # Output: # Second gg checking guard # First gg active # 10 # Second gg checking guard # Second gg active # 500 ## Calling super methods Super methods are methods higher in the multimethod's list of methods. F sup(x) x+1 F sup(x) super(x) * 10 echo(sup(5)) # Output: 60 ## Anonymous function (method) literal Syntactically, anonymous method is `F` not followed by a name. f = F(item) { echo("Item: $item") } echo("F is $f") # Output: F is at 1.ngs:1> each([10,20,30], f) # Output: # Item: 10 # Item: 20 # Item: 30 # Anonymous method as map() parameter echo([1,2,3].map(F(x) x*5)) # Output: [5,10,15] ## Anonymous function (method) literal using magical X, Y or Z variables Three special variables can be used as a syntactic sugar for creating anonymous methods. A method call which uses one of the special variables X, Y, or Z is wrapped in anonymous function as follows: `F(X=null, Y=null, Z=null) { method_call(....) } `. Operators are just syntactic sugar for method calls so `X*5` for example is also processed as described. # X*5 is same as F(X=null, Y=null, Z=null) { X*5 } which in out case is roughly equivalent to F(X) X*5 echo([1,2,3].map(X*5)) # Output: [5,10,15] # "Key:" + X is same as F(X=null, Y=null, Z=null) { "Key:" + X } echo({"a": "one", "b": "two"}.map("Key:" + X)) # Output: ['Key:a','Key:b'] echo({"a": "one", "b": "two"}.map("Val:" + Y)) # Output: ['Val:one','Val:two'] echo([1,2,3,11,12].count(X>10)) # Output: 2 If the special variables X, Y, or Z are found in a string interpolation, such as `"Key $X, Value $Y"`, the string is wrapped in anonymous function as described above. echo({"a": 1, "b": 2}.map("Key $X, Value $Y")) # Output: ['Key a, Value 1','Key b, Value 2'] ## Anonymous function (method) literal using magical A, B or C variables Additional syntactic sugar for anonymous function is `{ ... }`. This is equivalent to `F(A=null, B=null, C=null) { ... }`. # X*5 + 1 would not work as X*5 itself would be anonymous function echo([1,2,3].map({ A*5 + 1 })) # Output: [6,11,16] ## Returning values from methods Method evaluates to the last expression, as in Lisp and Ruby. Additionally following expressions allow immediate return from a method. * `return` - returns `null` * `return EXPR` - returns the value of EXPR * `COND returns` - if COND evaluates to true, returns `null` * `COND returns EXPR` - if COND evaluates to true, returns the value of EXPR Examples of `return`, `returns` and last expression value as result of a method. F flow_ret(x) { if x < 0 { unrelated_calculation = 1 return "negative" } x == 0 returns "zero" "positive" } echo(flow_ret(-1)) echo(flow_ret( 0)) echo(flow_ret( 1)) # Output: # negative # zero # positive ## Returning from inner method Use `block NAME BODY` syntax to return from inner methods. F find_the_one(haystack:Arr, needle) block b { haystack.each(F(elt) { if elt == needle then b.return("Found it!") }) "Not found" } echo([10,20].find_the_one(20)) echo([10,20].find_the_one(30)) # Output: # Found it! # Not found ## Referencing specially named methods Since identifiers are alphanumeric (with underscores), `my_method` can be referenced simple by name, while `+` for example can not. Here is example of referencing specially named methods: # (+) references the method + F sum(something:Eachable1) something.reduce(0, (+)) # Types NGS is a dynamically typed language: values (and not variables) have types. a = 1 a = "ok" # 'a' had value of one type and then value of another type NGS is a "strongly typed" language: values are not implicitly converted to unrelated types. This makes the language more verbose in some cases but helps catch certain type of bugs earlier. echo(1+"2") # ... Exception of type MethodNotFound occurred ... # That means that NGS has no method that "knows" how to add an Int and an Str echo(1+Int("2")) # Output: 3 NGS has several pre-defined types. A programmer can define additional types. Types roughly correspond to classes in other languages (Python, Ruby, Java) but without defined fields and methods. ## Creating object of given type Creating object of a given type is calling the type. Syntactically, it's type name followed by parenthesis which optionally contain arguments. # Create object of type Path p = Path("/") # Create EmptyBox object eb = EmptyBox() ## Defining your own types Type definition is only supported in code syntax. # Switch to code syntax inside { ... }. "type" currently does not work in command syntax { # Define type type Vehicle # Define sub-type type Car(Vehicle) type RedThing # Multiple inheritance type RedCar([Car, RedThing]) } echo(Vehicle) echo(Car) echo(RedCar.parents) # Output: # # # [,] ## Type constructors and object creation * When type definition (`type T`) is executed, NGS defines the `.constructors` MultiMethod that includes one method. This method creates object of type `T`. The type of this method is `NormalTypeConstructor`. * After defining a type `T`, one can add methods to `.constructors` using `F T(...) {...}` definitions. If you add your method to the `.constructors` field, please make sure it returns an object of type `T` or object of a type that is a subtype of `T`. Otherwise, it will be surprising for the callers. * When you call a user defined type, its `.constructors` MultiMethod is called. * In case that no user defined constructor is defined, the default constructor is invoked. * If no user defined constructor matches the arguments * If the default constructor added by NGS (method of type `NormalTypeConstructor`) gets executed, it does the following: * Creates object of type `T` * Runs `init()` MultiMethod with the new object as the first argument and the arguments of the call to `T(...)` as second and on arguments to `init(...)`. * Note that calling `init()` might trigger `MethodNotFound` exception. It is ignored if and only if the call to `T()` had no arguments. Examples: # Customizing Box object creation (code from stdlib) F Box(x) FullBox(x) F Box(n:Null) EmptyBox() F Box(a:Arr, idx) { ... } F Box(h:Hash, k) { ... } # FullBox initialization, called when FullBox(x) is called F init(b:FullBox, val) b.val = val # Note that there is no init(eb:EmptyBox) as there is nothing to initialize Box(1) # FullBox with 1 as value Box(null) # EmptyBox ## Defining type fields Currently, there is no way to define fields of a type, similar to earlier versions of JavaScript and Python. Assigning to arbitrary field of an object is possible. type MyType t = MyType() t.anything = "blah" # works echo(t) # ## Defining type methods In NGS, a method does not "belong" to a type. Therefore, "defining type methods" is not applicable. You can define a (multi)method to work with your type. type MyType F my_method(t:MyType, i:Int) { ... } F my_method(s:Str, t:MyType) { ... } F my_method(t1:MyType, t2:MyType) { ... } If you would like to be able to call your method as `t.my_method(...)` you should define a method where the first parameter is of type `MyType`. type MyType F my_method(t:MyType, i:Int) { ... } t = MyType() t.my_method(10) my_method(t, 10) # Same as above This works because NGS supports [UFCS](https://en.wikipedia.org/wiki/Uniform_Function_Call_Syntax) and therefore `t.my_method(...)` is syntactically equivalent to `my_method(t, ...)`. # Loading and running additional code ## require The `require` method reads, compiles (to bytecode) and executes the given file. Currently absolute paths and paths relative to current directory are supported. In future, paths relative to current directory will not be supported but rather paths relative to the file that does `require()` call will be supported, similar to Node.js. This needs work which was not done yet, there was never intention to support paths relative to current directory; it's just wrong. require('/home/someone/my_module.ngs') `require` returns the value of last expression in the file. This feature plays nice with namespaces (see below) in cases when the whole file is an `ns { ... }` expression. ## Auto-loading When an undefined global variable is referenced, `global_not_found_handler` is called. This handler tries to load a module (currently from the `lib/autoload` directory) that will define the referenced variable. Here are some modules that are autoloaded and additional variables/methods they define. * AWS * IP (IPAddr, IPNet) * Iter (ArrIter, ConstIter, HashIter, RangeIter, ...) # Namespaces Namespaces are used to prevent names collisions and hide methods. One of the main usages for namespaces is in modules. Let's consider a file named `my_ns.ngs`: ns { # This method is only visible to other methods in the namespace F _private_helper_method() { ... } F accessible_method() { ... _private_helper_method() ... } global global_method F global_method() { ... _private_helper_method() ... } } Another file, `example.ngs` could be using the above namespace utilizing the `require` method: arbitrary_ns_name = require('my_ns.ngs') global_method() # :: is namespace member access operator arbitrary_ns_name::accessible_method() ## How namespace definition works Namespace definition works in both commands and code syntax. The `ns { ... }` definition is equivalent to the following code: F() { _exports = {} .... _exports }() # Call as soon as the method is defined Inside the `F() { ... }`, all variables and methods which do not start with underscore (`_`) are automatically added to the `_exports` hash. The underscore exclusion prevents from `_exports` itself to be added to the hash. This implementation also allows arbitrary manipulations of the returned value. # The simple case ngs -pi 'ns { F func() 7; }' Hash of size 1 [func] = # Prefix all exported methods and variables with "my_" ngs -pi 'ns { F func() 7; _exports .= mapk("my_$X") }' Hash of size 1 [my_func] = # Make "ns" return arbitrary value $ ngs -pi 'ns {_exports=7}' Int: 7 # Show exclusion of underscore prefixed items $ ngs -pi 'ns { F visible() 7; F _invisible() 7; }' Hash of size 1 [visible] = # Not only methods but also variables are exported $ ngs -pi 'ns { visible=1; _invisible=2; }' Hash of size 1 [visible] = 1 ## Different names for external variables in a namespace Namespaces also support additional syntax `ns(PARAMETERS) { ... }`. This syntax is converted to `F(PARAMETERS) { ... }()`. This allows referencing external variables by another names. Inside the namespace one can now define variables and methods which would shadow the global (or upvar) variables and still access them. Here is example from `lib/autoload/AWS.ngs` : ns(GlobalRes=Res, GlobalResDef=ResDef, global_test=test) { ... doc AWS resource type Res(GlobalRes) ... doc AWS Resource definition type ResDef(GlobalResDef) ... } Note that when `ns(PARAMETERS) { ... }` syntax is converted to `F(PARAMETERS) { ... }`, the method is called without any arguments so all parameters must have default values or an exception will occur: $ ngs -pi 'ns(a) { 7 }' ... Exception of type ArgsMismatch occurred ## Planned future changes to namespaces The `ns { ... }` construct will probably not return a `Hash` object but object of a new type, `Namespace` which will probably be subtype of `Hash`. In any case, the `::` operator will be defined in such way that `my_namspace::my_method()` will continue to work. # Frequently used types and methods This section is not a reference but rather an overview. It includes the most important types and methods. For full documentation regarding types and methods see the generated documentation. ## Null `null` signifies absence of data. `null` is the only value of type `Null`. echo("abcd".pos("c")) echo("abcd".pos("x")) # Output: # 2 # null if null echo("if null") # No output, null converts to false ## Bool The type `Bool` represents a boolean. The values `true` and `false` are the only values of type `Bool`. if true echo("if true") if false echo("if false") # Output: # if true echo(1 == 1) # Output: # true echo(1 == 2) # Output: # false echo(true and false) # Output: # false echo(true or false) # Output: # true When a boolean value is needed, such as in `if EXPR {...}`, the `EXPR` is converted to boolean by calling `Bool(EXPR)`. NGS defines `Bool` multimethod (technically `Bool` type constructors), with methods for many types. These methods cause the following values to be converted to false: * 0 * null * false * empty array * empty hash * empty string * an EmptyBox object, for example `[1,2,3].Box(10)` (which boxes the element at index 10) * a Failure object, for example `Result({ 1 / 0 })` * regular expression that did not match, for example `"abc" ~ /XYZ/` Unless specified otherwise, all other values are converted to `true`. To define how your user-defined types behave as boolean, define `Bool(x:YOUR_TYPE)` method. ## Int `Int` represents integer numbers echo(1 + 2 * 3) # Output: # 7 { 3.each(echo) } # Output: # 0 # 1 # 2 { 3.map(X*2).each(echo) } # Output: # 0 # 2 # 4 ## Str `Str` represents string of bytes. Strings - string interpolation a = 1 echo("A is now $a") echo('A is now $a') # Output: # A is now 1 # A is now $a echo("Calculation result A: ${10+20}") echo("Calculation result B: ${ [1,2,3].map((*), 10).join(',') }") # Output: # Calculation result A: 30 # Calculation result B: 10,20,30 Strings - some basic methods that operate on strings echo("abc" + "abc") # Output: # abcabc echo("abc:def:ggg".split(":")) # Output: # ['abc','def','ggg'] echo("abc7def8ggg".split(/[0-9]/)) # Output: # ['abc','def','ggg'] echo("abc7def8ggg".without(/[0-9]/)) # Output: # abcdefggg if "abc" ~ /^a/ { echo("YES") } else { echo("NO") } # Output: # YES m = "abc=120" ~ /=/ if m { echo("Name=${m.before} Value=${m.after}") } # See RegExp type # Output: # Name=abc Value=120 ## Arr `Arr` represents an ordered list which can be accessed by zero-based index. Currently `Arr` is implemented as consecutive memory locations. Arrays - basics x = ["first", "second", "third", "fourth"] echo(x) # Output: # ['first','second','third','fourth'] echo(x.len()) # Output: # 4 echo('first' in x) # Output: # true echo('fifth' in x) # Output: # false echo(x[1]) # Output: # second echo(x[1..3]) # Output: # ['second','third'] echo(x == %[first second third fourth]) # Output: # true x = [ "blah" 2 'some text' ] echo(x) # Output: # ['blah',2,'some text'] echo(x[10]) # ... Exception of type IndexNotFound occurred ... Arrays - some basic methods that operate on arrays { [1,2].each(echo) } # Output: # 1 # 2 { [1,2].each(F(item) { echo("Item: $item") }) } # Output: # Item: 1 # Item: 2 echo([1,2].map(X * 10)) # See "Anonymous function literal using magical X, Y or Z variables" # Output: # [10,20] echo(Hash([["a",1],["b",2]])) # Converts array consisting of pairs into Hash # Output: # {a=1, b=2} ## Hash `Hash` represents a mapping between keys and values. `Hash` is implemented as a hash table. Hashes - basics x = {"a": 1, "b": 2} echo(x) # Output: # {a=1, b=2} echo(x['a']) echo(x.a) # Output: # 1 # 1 { x.b = 20 } echo(x.len()) # Output: # 2 echo(x.keys()) # Output: # ['a','b'] echo(x.values()) # Output: # [1,20] x = { "c": 1 "d": 2 } echo(x) # Output: # {c=1, d=2} echo('c' in x) # Output: # true echo(1 in x) # Output: # false echo(x.get('d')) # Output: # 2 echo(x.get('e')) # Output: # null echo(x.get('e', 'my_default')) # Output: # my_default echo(x.e) # ... Exception of type KeyNotFound occurred ... x = %{akey avalue bkey bvalue} echo(x) # Output: # {akey=avalue, bkey=bvalue} Hashes - some basic methods that operate on Hashes { h = {"a": 1, "b": 2} h.each(F(k, v) { echo("$k = $v") }) } # Output: # a = 1 # b = 2 h = {"a": 1, "b": 2} echo(h.map(F(k, v) "key-$k value-$v")) # Output: # ['key-a value-1','key-b value-2'] h = {"a": 1, "b": 2} echo(h.mapk("key-${X}")) # Output: # {key-a=1, key-b=2} h = {"a": 1, "b": 2} echo(h.mapv(X*10)) # Output: # {a=10, b=20} h = {"a": 1, "b": 2} echo(h.mapkv({ ["key-$A", B*100] })) # Output: # {key-a=100, key-b=200} h = {"a": 1, "b": 2} echo(Arr(h)) # Converts Hash into array consisting of pairs # Output: # [['a',1],['b',2]] # Higher-order functions (methods) This section is an overview. The purpose is to give overall impression of how higher-order functions feel in NGS. For full documentation regarding methods (including higher-order functions) see the generated documentation. Higher order functions are functions that get another function as a parameter. echo([1,2,3].all(Int)) # Output: # true echo([1,2,3,"a","b"].all(Int)) # Output: # false echo([1,2,3,"a","b"].any(Str)) # Output: # true echo([1,2,11,12,3].none(X>100)) # Output: # true echo([1,2,"a","b",3].filter(Int)) # Output: # [1,2,3] echo([1,2,"a","b",3].reject(Int)) # Output: # ['a','b'] echo([1,2,11,12,3].reject(X>10)) # Output: # [1,2,3] echo([1,2,11,12,3].count(X>10)) # Output: # 2 # Exceptions Similar to other languages (such as Python, Ruby, Java, etc), NGS has exceptions which can be thrown and caught. { type MyError(Error) try { # "e1 throws e2" is same as "if e1 { throw(e2) }" 1 == 2 throws Error("This can't be!") throw MyError("As usual, very helpful message") } catch(e:MyError) { echo("[Exceptions] This error was expected: $e") } catch(e:Error) { echo("[Exceptions] Unexpected error: $e") throw e } # Output: [Exceptions] This error was expected: ... } The `catch` clauses are implemented using multimethod. If an exception occurs, the `catch` multimethod is called with a single argument, the exception. Unlike all other situations, the search of a method that matches the exception is done from the beginning to the end of methods list. Implementation of `catch` using multimethod prevents using `return` inside `catch` clauses in the expected way. That's why this implementation might change in the future. ## Ignoring exceptions using "try" without "catch" myhash = {"a": 1, "b": 2} v = try myhash.a echo(v) # Output: 1 v = try myhash.c echo(v) # Output: null v = try { unrelated = 1+2; myhash.c } echo(v) # Output: null ## Default value for expression that causes exception The short-circuit binary operator `tor` (try-or) allows specifying default value for an expression that might throw an exception. # .backtrace might not exist, .frames might not exist # Accessing non-existent fields using the . notation will cause KeyNotFound exception parent_frames = parent.backtrace.frames tor [] `EXPR tor null` is functionally equivalent to `try EXPR`. ## Alternative mechanism for handling exceptions See generated documentation for the `Result` type. # Flow control ## If If works in both commands and code syntax. if my_var > 10 { a += 100 b = "x" } else { b = "y" } result = if my_var then "xyz" else "ww". result = if my_var 10 20 # result is now either 10 or 20 In `if`, `while`, and `for`, where `{...}` code block is expected, if the block consists of only a single expression, curly braces are optional. In `if`, the `then` and `else` keywords are optional. Note that `if` is an expression. `if` without else where the condition is equivalent to `false`, evaluates to `null`. ## Loops Loops work in both commands and code syntax. for(i=0; i<5; i+=1) { if i == 3 { continue } echo("Regular loop, iteration $i") } # Output: # Regular loop, iteration 0 # Regular loop, iteration 1 # Regular loop, iteration 2 # Regular loop, iteration 4 for(i;5) { i == 3 continues echo("Shorthand loop, iteration $i") } # Output: # Shorthand loop, iteration 0 # Shorthand loop, iteration 1 # Shorthand loop, iteration 2 # Shorthand loop, iteration 4 for i in [1,5,10,20,50] { echo(i) } # Output: # 1 # 5 # 10 # 20 # 50 # See "Iterators" i = 0 while i<10 { echo("While loop, iteration $i") i += 1 # Same as "if i == 2 { break }" i == 2 breaks } # Output: # While loop, iteration 0 # While loop, iteration 1 ## Switch and switch-like expressions Switch and switch-like expressions work in both commands and code syntax. * The first match activates the related code, there is no fall-through. * The value resulting from executing the code is the result of the switch-like expression. * If there is no match, `switch`, `match` and `cond` return `null`. * If there is no match, `eswitch`, `ematch` and `econd` throw `SwitchFail` exception. Switch and switch-like expressions examples. a = 10 result = switch a { 10 "ten" 20 "twenty" 30 { more_code(); "thirty" } } echo("Switch result for $a is $result") # Output: Switch result for 10 is ten a = "my_string" result = match a { Int "an integer" /^my_/ "special string" Str { more_code(); "a string" } Any "not sure" } echo("Match result for $a is $result") # Output: Match result for my_string is special string a = 12 result = cond { a > 10 "Excellent" a > 5 { more_code(); "Good enough" } a > 3 "so so" } echo("Cond result for $a is $result") # Output: Cond result for 12 is Excellent # SwitchFail exception will be thrown F will_throw_exception1() { a = "bad value" result = eswitch a { 1 "one" 2 "two" } } # SwitchFail exception will be thrown F will_throw_exception2() { a = true result = ematch a { Int "an integer" Str "a string" } } # SwitchFail exception will be thrown F will_throw_exception3() { a = 10 result = econd { a > 15 "one" a > 20 "two" } } ## Block The feature was inspired by `block` in Lisp ("structured, lexical, non-local exit facility") and adapted to fit NGS. The `block` evaluates to the first value provided by `return`. If no `return` occurs, the `block` evaluates to the last expression inside the `block`. my_result = block b { if i_know_the_answer_by_now() { b.return(42) } if maybe_now() { b.return(7) } 0 } # Regular expressions myregex = /^begin/ echo(myregex) # Output: mymatch = "beginABC" ~ myregex echo(mymatch) # Output: echo(mymatch.matches[0]) # Output: begin echo(mymatch.after) # Output: ABC all_matches = "1a2bcd3efg" ~~ /([0-9])(.)/ each(all_matches, F(match) { echo("The character after the digit ${match.matches[1]} is ${match.matches[2]}") }) # Output: # The character after the digit 1 is a # The character after the digit 2 is b # The character after the digit 3 is e # Collector facility Collector facility assists in building lists, hashes and other data structures. Collector facility should be used where `map` and other functional facilities are not meeting the requirements or when using collector is cleaner. Typically, the code of the following forms is replaced by collector (or some mix or something similar to the following forms): # --- 1 --- my_arr = [] for ... { my_arr.push(my_something_one) if ... { my_arr.push(my_something_two) } } # --- 2 --- my_arr = [] my_arr.push(my_something_one) for ... { if ... { my_arr.push(my_something_two) } } my_arr.push(my_something_three) The `collector` keyword wraps the following expression (or expressions if they are grouped using `{...}` syntax) in a function with single argument, `collect`. `collector` keyword can be followed by slash (`/`) and initial expression, which defaults to an empty array. mylist = collector { collect("HEADER") [10,20].each(collect) collect("FOOTER") } echo(mylist) # Output: ['HEADER',10,20,'FOOTER'] myhash = collector/{} { collect("first", -1) {"a": 1, "b": 2, "c":100}.each(F(k, v) { if v < 100 { collect("($k)", v*10) } }) collect("last", -2) } echo(myhash) # Output: {first=-1, (a)=10, (b)=20, last=-2} mysumm = collector/0 [1,10,100].each(collect) echo(mysumm) # Output: 111 # Code from stdlib: F flatten(arr:Arr) collector arr.each(F(subarr) { subarr.each(collect) }) ## Customizing collector facility `collector` keyword is implemented as a call to `collector` method with two arguments: the initial value (which defaults to empty array) and the wrapped code, which is passed as a method. By defining `collector` method, the programmer can customize the behaviour. Here is implementation of collector for `Arr`. # Code from stdlib: F collector(a:Arr, body:Fun) { body(F(elt) a.push(elt)) a } Here `a` gets the initial value, `body` is the wrapped code that comes after the `collector` keyword, the anonymous function that is passed to `body` becomes the `collect` function inside the wrapped code. Here is `collector` for `Hash` and it's usage example: # Code from stdlib: F collector(h:Hash, body:Fun) { body(F(k, v) h[k] = v) h } F mapk(h:Hash, mapper:Fun) collector/{} h.each(F(k, v) collect(mapper(k), v)) # Assignment shortcuts These are syntactically equivalent expressions: a = a + 1 a += 1 a = a - 1 a -= 1 a = a * 1 a *= 1 a = a / 1 a /= 1 a = a % 1 a %= 1 a = a.f() a .= f() a = f(a) a = a.f(b) a .= f(b) a = f(a, b) # Language gotchas This section will be expanded as I get feedback :) ## NGS should not be your first language I do not recommend NGS as your first language. Python, for example, would be a much better choice. Programming experience is assumed prior to using NGS. ## Watch the version NGS is under development. Breaking changes can happen. If you do anything important with NGS, it's preferable to note the git revision you are using for reproducible installation. The plan is to stop breaking NGS when it reaches version 1.0.0. From that version, the behaviour will be common - patch level increase for fixes, minor for improvements, major for breaking changes. ## Keyword arguments gotchas Keyword arguments implementation is preliminary so: # Keyword arguments are silently ignored if corresponding positional argument is passed. kwargs = {"a": 10} F f(a, **kw) a; f(1, **kwargs) == 1 # Keyword arguments for existing named parameters cause parameters not to match kwargs = {"a": 10} F f(a) a; f(1, **kwargs) == 1 # **kwargs silently override (consistent with literal hash) previous named arguments kwargs = {"a": 10} F f(a=1) a; f(a=10, **kwargs) == 1 # For speed and implementation simplicity reasons, the **kw parameter has all keys, # even if some of them matched and used for parameters. # This is somewhat likely to change in the future. kwargs = {"a": 10} F f(a=1, **kw) [a, kw]; f(**kwargs) == [10, {"a": 10}] ## Namespaces gotchas Since by default all variables are local, the following example will not add method to `some_global_name` multimethod but will create namespace-local `some_global_name` multimethod. myns = ns { F some_global_name() ... } # Only available as myns::some_global_name(...) # Global some_global_name(...) will not be able to access the # above implementation. The correct version to add method to a global multimethod is ns { global some_global_name F some_global_name() ... } # Available as global some_global_name(...) ## Comments syntax gotchas Comments syntax is implemented in many places but not everywhere. If you get syntax error regarding comment, move it to somewhere nearby. ## Mutable default parameter gotchas The code below will probably not do what was intended. Note that the default parameter value is only computed once, at method definition time. F f(x, a:Arr=[]) { a.push(x) a } echo(f(10)) # [10] echo(f(20)) # [10,20] Same happens in Python and hence already described: http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments ## Type system is only used for multi-dispatch Consider the code: F f(x:Int=null) { ... } The `x:Int` only says that when calling `f`, the parameter `x` must be of type `Int`. It does *not* mean that `x` must be `Int` inside `f`. Even the default value can be of different type. # Handlers Handlers are called by NGS when a certain condition occurs. **A handler** is a `MultiMethod`. Like with any other multimethod, you can override what it does by defining your own method with the same name further down in the code. Since standard handlers are defined in **stdlib.ngs** which is typically loaded first, your own methods will be "further down". ## method_not_found_handler `method_not_found_handler` is called when a multimethod was called but no method matched the arguments. Use `F method_not_found_handler(callable:Fun, *args) ...` to add your behaviours. ## global_not_found_handler `global_not_found_handler` is called on attempt to read from an undefined global variable. Uses `NGS_PATH` when looking up the appropriate file. # Hooks **A hook** is a `Hook` object. Some hooks are called by NGS when a certain condition occurs. You are free to create and use your own hooks. When called, it executes all registered methods. The main difference vs calling a `MultiMethod` is that using hook you get accumulative behaviour instead of overriding behaviour. User-defined hook example: { h = Hook() h.push({ echo("A") }) h.push({ echo("B") }) h() } # Outputs one per line: A, B Another way is to add named hook handlers (also a practical example): exit_hook['cleanup_temp_files'] = F(exit:Exit) { # remove my temp files } ## exit_hook `exit_hook` is called when NGS is about to exit. Typical cases are: * All of the given code was executed. * `throw` was invoked and no matching `catch` was found. Method signature: `exit_hook(exit:Exit)`. `Exit` currently has two keys: `exit_code` (`Int`) and `exceptions` (`Arr`). **stdlib.ngs** defines one standard exit hook. $ ngs -pi 'exit_hook.Hash()' Hash of size 1 [print_exception] = (exit:Exit) at /usr/local/lib/ngs/stdlib.ngs:5345> * `print_exception` prints exception details if an exception occurred. # Patterns Methods such as `filter`, `filterk`, `filterv`, etc take a pattern argument, somewhat similar to Ruby. Traditionally, such methods (functions) took a method (function) as the predicate argument instead. A predicate function is a function with one parameter and that returns a boolean. Using pattern as opposed to predicate allows more dense (meaning per characters of code) expressions. NGS higher-order methods (functions) that have pattern parameters compare items to patterns using the `=~` operator: F filter(e:Eachable1, pattern=Bool.constructors) { t = Type(e) ret = t() e.each(F(elt) { if elt =~ pattern ret.push(elt) }) ret } This behaviour allows defining how any given type behaves as a pattern. Several built-in types have `=~` method defined for them. Here are some examples of how patterns shorten the code: # Same as {...}.filterk(F(k) k ~ /^b/) {"a1": 1, "a2": 2, "b1": 10}.filterk(/^b/) # {"b1": 10} # Same as {...}.filter(F(elt) elt.x == 7) [{"x": 10, "y": 20}, {"x": 7, "y": 30}].filter({"x": 7}) # [{x=7, y=30}] # Same as {...}.filter(F(t) t is Str) ["abc", 1, "def"].filter(Str) # [abc,def] One can easily define how a type behaves as a pattern: define appropriate `=~` method. Here is the `=~` method that allows `.filter(SomeType)` such as `.filter(Str)` example above: F =~(x, t:Type) x is t Note that using a method as pattern is still possible: [1,2,11].filter(F(x) x > 10) # [11] That is because `=~` method uses the given function as predicate: F =~(x, f:Fun) f(x) # Iterators Iterators give more flexibility and control as opposed to `each` iteration. ## The iterator protocol Iterators must implement the following methods: * `Bool(x:YOUR_ITER_TYPE)` which tells whether there are more values * `next(x:YOUR_ITER_TYPE)` which returns the next value (or throws `NoNext` exception) When `for x in EXPR { BODY }` syntax is used, it is equivalent to the following: _hidden_iter = Iter(EXPR) while _hidden_iter { x = _hidden_iter.next() BODY } ## Referencing the iterator in the BODY Since `Iter(x:Iter)` is defined as `x`, you can use the following solution if you want to use the `for` syntax and have access to the iterator: for x in my_iter=Iter(EXPR) { BODY # can manipulate my_iter for advanced control } The above works as follows: `my_iter=Iter(EXPR)` is an assignment which evaluates to an `Iter` object. `for` uses `Iter` on that value to get an iterator. `Iter(i:Iter)` is defined as `i`. Therefore, the `Iter` used by for and `my_iter` is the same iterator. ## Built-in iterators See `Iter` type documentation to see which iterators are available in NGS. # Executing External Programs ## Syntax - Basics * In **commands syntax** (default for the top level of an NGS file) just go ahead and `my_prog my_arg ...` * In **code syntax** to be able to get into **commands syntax**, use one of the options below: * `$(my_prog my_arg ...)` - run the process and get the reference to it, a `ProcessesPipeline` object. * `%(my_prog my_arg ...)` - construct the `CommandsPipeline` but do not run it. Useful for passing ready-to-run commands around: `t=%(ls /); ...; $($t)` * `` `my_prog my_arg ...` `` - capture the output as a string, an `Str` object. * ```` ``my_prog my_arg ...`` ```` - capture and decode the output, an arbitrary structured data. ## Syntax - Substitution of Command Line Arguments ls $fname # $var_name - expands to exactly one argument ls ${ EXPR } # ${ EXPR } - expands to exactly one argument ls $*fnames # $*var_name - expands to zero or more arguments ls $*{ EXPR } # $*{ EXPR } - expands to zero or more arguments `EXPR` above can be a single expression or any number expressions, new-line or `;` separated, value of the last one is used for the substitution (consistent with the rest of the language). ## Syntax - Options Rationale: it is sometimes desirable to alter the behaviour of execution of an external program. NGS provides syntax for passing arbitrary key-value pairs to augment `CommandsPipeline` or `Command`. ``` option_name:option_value my_prog my_arg ... # opts for my_prog option_name: my_prog my_arg ... # opts for my_prog option_name::option_value command_1 | command_2 # options for the whole pipeline option_name:: command_1 | command_2 # options for the while pipeline ``` Note that no spaces allowed after `:` or `::` and before the value of the option. Missing value defaults to `true`. Available options follow. It is possible to extend NGS to handle user defined options. The values for the options are not a special syntax, they are just **expressions syntax** which is used in many other parts of the language. Providing exit codes which will *not throw an exception* ``` ok: my_prog my_arg ... # any exit code is ok ok:1 my_prog my_arg ... # only exit code 1 is ok ok:[0, 1] my_prog my_arg ... # only exit codes 0 and 1 are ok ok:0..5 my_prog my_arg ... # only exit codes between 0 and 5 (exclusive) are ok ok:0..5 my_prog my_arg ... # only exit codes between 0 and 5 (inclusive) are ok ``` Do not capture the output of the command. This is the default behaviour of the top-level **commands syntax**: the stdout connected to the stdout of the NGS process: ``` top_level:: my_prog my_arg ... ``` Return the first line of the output, without the new-line characters ``` line: my_prog my_arg ... # Usage: t = `line: my_prog my_arg ...` ``` Log the command including arguments and redirects before running it: ``` log: my_prog my_arg ... ``` Internally, the `&` in `$(my_prog my_arg &)` sets the `&` option to `true`. This mechanism can be used to modify the behaviour of `CommandsPipeline` prepared by `%(...)`. ngs -pi '%(ls / &)' CommandsPipeline options: Hash of size 1 options: [&] = true command[0]: **WARNING**: unrecognized options are silently ignored. This is considered a bug and should be fixed - https://github.com/ngs-lang/ngs/issues/28 ## Syntax - Input/Output Redirections Redirections are represented by the `CommandRedir` type. When a command is parsed and converted to `CommandsPipeline`, each `Command` gets a (possibly empty) list of redirections (in the `.redirects` property) Redirections syntax (based on bash syntax): ``` my_prog my_arg ... >myfile # overwrite my_prog my_arg ... >myfile # append my_prog my_arg ... 2>myfile # overwrite, from stderr my_prog my_arg ... 2>>myfile # append, from stderr my_prog my_arg ... 2>${true} # capture stderr ``` Running external programs examples: $ ngs -pi '$(ok: ls xxx)' ls: xxx: No such file or directory ProcessesPipeline cp: CommandsPipeline cp: command[0]: pipe_in: null pipe_out: null processes[0]: Process processes[0]: command = processes[0]: pid = 77460 processes[0]: exit_code = 1 processes[0]: exit_signal = null processes[0]: redirects[0]: > processes[0]: output on fd 1, stdout (0 lines): processes[0]: output on fd 2, stderr (0 lines): $ ngs -pi '$(ok: ls xxx 2>${true})' ProcessesPipeline cp: CommandsPipeline cp: command[0]: true>] argv=[ls,xxx]> pipe_in: null pipe_out: null processes[0]: Process processes[0]: command = true>] argv=[ls,xxx]> processes[0]: pid = 77477 processes[0]: exit_code = 1 processes[0]: exit_signal = null processes[0]: redirects[0]: > processes[0]: redirects[1]: datum=> processes[0]: output on fd 1, stdout (0 lines): processes[0]: output on fd 2, stderr (1 lines): processes[0]: ls: xxx: No such file or directory ## Internals At runtime, a `CommandsPipeline` object is created for all the syntaxes mentioned above. Then depending on the specific syntax, appropriate method is called with the `CommandsPipeline` argument. The methods are: `$()`, ```` `` ````, and ```````` ```` ````````. External commands are represented with type `CommandsPipeline`, which defines what should be run. In the simple case, there is exactly one command. In more complex cases `CommandsPipeline` has several programs and pipes between them, like for `$(my_prog_1 | my_prog_2 | my_prog_3)` . When `CommandsPipeline` is started being executed, a new object of type `ProcessesPipeline` is created using the appropriate `CommandsPipeline` as a blueprint. `ProcessesPipeline` has a `cp` field, which references the corresponding `CommandsPipeline`. $ ngs -pi '$(ls | wc -l)' ProcessesPipeline cp: CommandsPipeline cp: command[0]: cp: pipe[1]: cp: command[1]: pipe_in: null pipe_out: null processes[0]: Process processes[0]: command = processes[0]: pid = 77369 processes[0]: exit_code = 0 processes[0]: exit_signal = null processes[0]: redirects[0]: > processes[0]: output on fd 1, stdout (0 lines): processes[0]: output on fd 2, stderr (0 lines): processes[1]: Process processes[1]: command = processes[1]: pid = 77370 processes[1]: exit_code = 0 processes[1]: exit_signal = null processes[1]: redirects[0]: > processes[1]: redirects[1]: > processes[1]: output on fd 1, stdout (1 lines): processes[1]: 75 processes[1]: output on fd 2, stderr (0 lines): ## Decoding the Output Let's look on what's behind the described above syntax: * ```` ``my_prog my_arg ...`` ```` - capture and decode the output, an arbitrary structured data. Internally, the following code is executed: ``` p.stdout.decode({'process': p, 'pipeline': cp}) ``` The code above calls the multimethod defined as `F decode(s:Str, hints:Hash={})` . Out of the box NGS implements several methods that are able to parse JSON, JWT, output of `aws` and `find` commands. In case you need to decode a format not known to NGS, you can implement your own parser by defining yet another `decode()` method with appropriate `guard` statement to make sure that `s` is the expected input. It should look roughly like the following (real example from stdlib): ``` doc Parse the output of "find" command which does not use "-printf". Handles "-print0". doc hints - hints.process.command.argv[0] must be 'find' doc %RET - Arr of Str F decode(s:Str, hints:Hash) { guard try hints['process'].command.argv[0] == 'find' argv = hints['process'].command.argv guard '-printf' not in argv zero_sep = '-print0' in argv if zero_sep then s.split(chr(0))[0..-1] else s.lines() } ``` Before writing your own parser, check whether you can leverage the `jc` project. It can parse many formats and output the information as JSON. In turn, JSON can be easily parsed by NGS. Integration with NGS is as simple as the following: ``` $ ngs -pl '``jc ifconfig``.name.sort()' # prints interfaces names bridge0 en0 utun0 ... ``` `jc` project is at https://github.com/kellyjonbrazil/jc ## Exit codes handling when running external programs Immediately after an external program finishes, it's exit code (and in the future, possibly other aspects) is checked using `finished_ok` multimethod. The multimethod returns a `Bool`. If it's `false`, an exception is thrown. For unknown programs, `finished_ok` returns `false` for non-zero exit code and hence an exception is thrown. This behaviour can be modified with the `ok:` option of the command. NGS knows about some of external programs (`false`, `test`, `fuser`, `ping`) which might return non-zero exit codes which do not indicate an error. These commands do not cause exceptions for exit code one. If there is a particular program that you are using which also returns non-zero exit codes which should not cause an exception, you should add your own method `finished_ok` which handles that program. Please consider contributing this addition back to NGS. ## External program boolean value When converting to `Bool`, for example in the expression `if $(test -f my_file) { ... }`, all processes in the `ProcessesPipeline` must exit with code zero in order to get `true`, otherwise the result of converting to `Bool` is `false`. ## Constructing command line arguments While it's feasible to construct an array with command line arguments for a program, `Argv` makes this task much easier by supporting common cases such as omitting a command line switch which has no corresponding value to use. Relatively basic example of using the facility follows. Note that `get()` methods might return `null`. In this case, the corresponding `--switch` will not be present. ``` args = Argv({ '--destination-cidr-block': route.DestinationCidrBlock '--gateway-id': route.get('GatewayId') '--instance-id': route.get('InstanceId') '--network-interface-id': route.get('NetworkInterfaceId') '--vpc-peering-connection-id': route.get('VpcPeeringConnectionId') '--nat-gateway-id': route.get('NatGatewayId') }) $(aws ec2 create-route --route-table-id ... $*args) ``` See `Argv` multimethod documentation - https://ngs-lang.org/doc/latest/generated/Argv.html ## Globbing Globbing is not implemented yet. ## Debugging Use `DEBUG` environment variable to show debug information about processes that NGS runs. Example: ``` $ DEBUG=process,process2 ngs -e '$(ls)' [DEBUG process2 42167 main] cp.pipes.len()=2 [DEBUG process 42167 main] Parsed command: [ls] [DEBUG process 42167 main] [find_in_path] got ls [DEBUG process 42167 main] [find_in_path] will search [DEBUG process 42167 main] [find_in_path] ls found at [DEBUG process 42167 main] PID after fork: 42168 [DEBUG process 42168 main] PID after fork: 0 [DEBUG process2 42167 main] PIPES FROM REDIRECTS [>] REDIRECTS [>] COMMAND {command=, executable=, pid=null, exit_code=null, exit_signal=null, outputs={1=null, 2=null}, stdout=null, stderr=null, reading_threads=, writing_threads=, redirects=[>]} [DEBUG process 42167 main] Reading all output of the child process [DEBUG process 42167 main] Creating thread process-42168-output-fd-1-reader [DEBUG process 42167 process-42168-output-fd-1-reader] Thread set up done, will run thread code [DEBUG process2 42167 process-42168-output-fd-1-reader] READING [DEBUG process2 42168 main] PIPES FROM REDIRECTS [>] REDIRECTS [>] COMMAND {command=, executable=, pid=null, exit_code=null, exit_signal=null, outputs={1=null, 2=null}, stdout=null, stderr=null, reading_threads=, writing_threads=, redirects=[>]} [DEBUG process2 42168 main] CHILD CLOSING FDS 3 TO 100 EXCEPT FOR [4] [DEBUG process2 42167 main] $() returns process {command=, executable=, pid=42168, exit_code=null, exit_signal=null, outputs={1=null, 2=null}, stdout=null, stderr=null, reading_threads=>, writing_threads=, redirects=[>]} [DEBUG process2 42168 main] CHILD REDIRECT PIPE 1 -> 4 [DEBUG process 42168 main] will execve() [DEBUG process 42167 main] [wait] joining reading_threads: > [DEBUG process 42167 main] Joining thread [DEBUG process2 42167 process-42168-output-fd-1-reader] READ <> [DEBUG process2 42167 process-42168-output-fd-1-reader] DATUM *** [DEBUG process2 42167 process-42168-output-fd-1-reader] CLOSED READING END OF PIPE, REDIRECT > [DEBUG process 42167 process-42168-output-fd-1-reader] Read all output of the child process for descriptor 1, closing reading/parent end [DEBUG process 42167 main] [wait] joined threads [DEBUG process 42167 main] [wait] joining writing_threads: [DEBUG process 42167 main] [wait] joined threads [DEBUG process 42167 main] [wait] will waitpid(42168) [DEBUG process 42167 main] [wait] waitpid(42168) -> [42168,0] ``` In most cases `DEBUG=process` should be enough to understand what is happening. `process2` shows lower level details. # Accessing Environment Variables On NGS startup, the global variable `ENV` is populated with the environment variables. `ENV` is a `Hash` object. echo(ENV.HOME) echo(ENV["HO" + "ME"]) `ENV` is also passed to external programs. { ENV.HOME = "/home/fake" $(my_prog) } Accessing non-existent environment variable throws an exception: ``` $ ngs -e ENV.X [ERROR 2021-12-16 09:31:09 IST] +--------------------------------------+ [ERROR 2021-12-16 09:31:09 IST] | Environment variable 'X' is not set. | [ERROR 2021-12-16 09:31:09 IST] +--------------------------------------+ ... ``` # NGS exit codes This section describes exit codes returned by NGS scripts written on exit. Both regular results of running a program and exceptions are converted to exit code using `ExitCode` MultiMethod. Built in methods of `ExitCode` do the following conversion: * `true` - 0 * `false` - 1 * `Int` object - the number itself * `Exception` objects - 240 * `FullBox` object - 0 * `EmptyBox` object - 1 * `Success` object - 0 * `Failure` object - 1 * `FatalError` objects - using the `exit_code` field of the exception, which if not given to `FatalError` constructor defaults to 1. * `ProcessesPipeline` / `Process` objects - the exit code of the external command * `MatchY` object - 0 * `MatchN` object - 1 * All others - 0 Special cases: * An exception and additional exception when converting the first exception using `ExitCode` - 241 * An exception and `exit_hook` failure - 242 * An exception, additional exception when converting the first exception using `ExitCode`, and `exit_hook` failure - 243 * Non-Exception was thrown and then caught by C code - 245 * NGS could not find the bootstrap file - 246 * NGS could not open bootstrap file - 247 * NGS could not parse bootstrap file - 248 ## Customizing translation of your types/objects to exit code Define `F ExitCode(t:MyType) {...}` method that returns `Int` to customize the exit code. # Declarative primitives Some of NGS libraries (currently only AWS) implement declarative primitives approach. Declarative primitives approach is to expose functions to manipulate resources in an idempotent way. Similar to configuration management systems, exposed resources are manipulated by declaring their desired state. Unlike configuration management systems, this approach does not deal with dependencies and order resolution. Declarative primitives approach provides convenient scripting facility, not a configuration management system. ## Related methods **Constructors** Resources references are created using constructors, such as `AWS::Vpc`. Resources references are of type which is a subtype of `ResDef`. my_vpc = AWS::Vpc() # Reference to all VPCs that can be fetched using AWS CLI echo(my_vpc) # Output: echo(AWS::Vpc().Type()) # Output: echo(AWS::Vpc().Type().parents) # Output: [] **find** `find` - find the specified resources. Almost never should be called manually. `expect`, `converge`, `each` and other methods run `find` automatically if needed (if it did not run previously on the reference). my_vpc = AWS::Vpc().find() echo(my_vpc) # Output: # Empty result is not an exception for `find`. **expect** `expect` - check that referenced resources exist. Throw an exception otherwise. Used to assert assumptions about the system. If you write a script that for example counts on exactly one VPC that matches your description to exist, I strongly recommend to be explicit and state your expectation as `.expect(1)`. my_vpc = AWS::Vpc().expect() # If no VPCs exist - throws exception echo(my_vpc) # Output: # # Find VPCs with name tag "XX" my_vpc = AWS::Vpc({"Name": "XX"}).expect() # If no VPCs exist - throws exception # ... Exception of type AssertFail occurred ... # Find VPCs with the given tag my_vpc = AWS::Vpc({"aws:cloudformation:stack-name": "edge-vpc"}).expect(1) # Would throw exception if there was not exactly one resources matching the description echo(my_vpc) # Output: # **converge** `converge` - Creates or updates resources if needed in order to match given properties. Example from production code, shortened for brevity: # Library that describes ELB creation. The system has several ELBs which are similarly configured. # The library contains this common configuration. ns(c=converge) { F converge(env, role, dns=null) { edge_vpc_anchor = {'aws:cloudformation:stack-name': 'edge-vpc'} vpc = AWS::Vpc(edge_vpc_anchor).expect(1) tags = {'env': env, 'role': role, 'creator': 'system/images/elb.ngs'} # Anchor: name + vpc_id elb_sg = AWS::SecGroup("https-server", vpc) elb = AWS::Elb("${env}-${role}") # ---> Converge happens here <--- # Load balancer is either created or updated here elb.c( Tags = tags + if dns { {'DNS': dns} } else { {} }, ListenerDescriptions = ..., Subnets = AWS::Subnet(edge_vpc_anchor).expect(2), SecurityGroups = elb_sg, HealthCheck = ..., Instances = AWS::Instance({'env': env, 'role': role}).expect() ) } } **delete** `delete` deletes referenced resources. # Delete referenced resources, in this case one specific load balancer AWS::Elb("prod-mobile-redirect").delete() # Delete all unused load balancers AWS::Elb().reject(X.Instances).delete() ## Anchor Anchors describe which resources are referenced. As you probably have noticed, resource references, which are created using constructors, get some parameters. These parameters are called "Anchor". Note that all parameters passed to the constructors are called "Anchor" collectively. Examples of Anchors: * Tags hash: `AWS::Instance({'env': env, 'role': role})` * Explicit tags: `AWS::Instance(Tags={'env': 'prod', 'role': 'ssh-sandbox'})` * Resources specific properties: `AWS::Vpc(IsDefault=true).expect(1)` * Tags and resource-specific properties: `AWS::Vpc(IsDefault=false, Tags=...).expect(1)` Any combination of explicit tags and resource-specific properties is possible. ## Properties Properties describe the desired state of resources. The parameters to `converge` are properties. See the `converge` example above. ## Planned future changes * `regions` in AWS library will probably be renamed to `_Region` or something else starting with underscore to emphasize it's a syntactic property, not something that correlates to AWS CLI output. * Currently there is no way to know whether `converge` has created resources or updated them. There will be a way to tell. # Debugging This section is planned to grow over time. ## MethodNotFound exception $ ngs -pi 'map(1,2,3)' ... Exception of type MethodNotFound occurred ... # Means the parameters did not match any of the existing methods. # See which methods are defined: ngs -pi map MultiMethod Methods Array of size 7 [0] = [1] = [2] = [3] = [4] = [5] = [6] =