REF PROGLIST                                        John Gibson Oct 1992

       COPYRIGHT University of Sussex 1993. All Rights Reserved.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<                             >>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<    THE PROGLIST VARIABLE    >>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<                             >>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Poplog provides the global  permanent variable proglist  for use as  the
input item stream by many system modules, including the Pop-11 compiler.
This  file  describes  the  procedures  and  variables  associated  with
proglist.

         CONTENTS - (Use <ENTER> g to access required sections)

  1   Introduction

  2   Reading Items from Proglist

  3   Reading Items with Macro Expansion

  4   Proglist-Related Permanent Macros

  5   Including a File in the Current Compilation Stream

  6   Proglist State Variables



---------------
1  Introduction
---------------

A standard requirement for  compilers and other  utilities in Poplog  is
the ability to read source text in itemised (or 'tokenised') form,  that
is, to have  an input  character stream lexically  analysed into  words,
numbers, strings, etc. This  process can be  performed, for example,  by
the item repeater procedures constructed by * incharitem from  character
repeaters (see REF * CHARIO and REF * ITEMISE).

    For programs which wish to process such an input item stream, it  is
particularly convenient to do so via a dynamic list constructed from the
item repeater (see REF * LISTS for  a description of dynamic lists).  An
item can then be  obtained simply by  taking the head  of the list,  and
saved there until the next item is required (at which point the list can
be tailed). Another advantage of this method is that extra items can  be
inserted into the stream when necessary, by concatenation onto the front
of the list (e.g. for macro expansion, see below).

    Because many parts of programs may require access to the item stream
currently in use, it is also useful to define a standard location  where
the input list can be found; for this reason, Poplog provides the global
permanent variable proglist.  The list  (dynamic or  otherwise) in  this
variable is  used as  the  input item  stream  by many  system  modules,
including the Pop-11 compiler (see REF * POPCOMPILE).

    Although proglist can be processed directly, it is conventional (and
usually  more  convenient)  to  take  input  items  from  it  using  the
item-reading procedures described below.  Of these procedures,  itemread
and nextitem also provide a macro expansion facility.




------------------------------
2  Reading Items from Proglist
------------------------------

poplastitem -> item                                           [variable]
item -> poplastitem
        Contains the  last  item read  from  proglist with  readitem  or
        itemread.


readitem() -> item                                           [procedure]
        Removes the next item from proglist and returns it. If  proglist
        is null, then <termin> is  returned. The item returned is  saved
        in poplastitem.


nextreaditem() -> item                                       [procedure]
        Returns the next item from proglist, but without removing it (so
        that another  call of  nextreaditem will  return the  same  item
        again). <termin> is returned if proglist is null.


readtill(end_item) -> (item_1, item_2, ..., end_item)        [procedure]
readtill(end_list) -> (item_1, item_2, ..., end_item)
        Reads items with readitem until an item either equal to end_item
        (first  form)  or  a  member   of  end_list  (second  form)   is
        encountered. All items read, including the last, are returned.


readline() -> list                                  [procedure variable]
        The  standard  procedure  in  this  variable  reads  items  from
        proglist (using readitem) until a newline is encountered.  (This
        is done by setting popnewline true, so that newline is  returned
        as a word.) The result is a list containing all the items  read,
        excluding the newline.


pop_readline_prompt -> string                                 [variable]
string -> pop_readline_prompt
        The item in this variable (default '?\s') is locally assigned to
        popprompt during readline. See  REF * SYSIO for possible  values
        of popprompt.


requestline(prompt) -> list                                  [procedure]
        Same as readline,  but with pop_readline_prompt  locally set  to
        prompt.


rdstringto(end_item) -> string                               [procedure]
rdstringto(end_list) -> string
        Reads items with readitem until an item either equal to end_item
        (first  form)  or  a  member   of  end_list  (second  form)   is
        encountered.

        The result is  the string formed  by concatenating together  the
        printed representations  of all  the items  read, EXCLUDING  the
        last (the items are concatenated with sys_><, see REF * PRINT).


readstringline() -> string                                   [procedure]
        Reads a string of characters string up to (but not  including) a
        newline or <termin>, in the following way:

        If   proglist   is    an   unexpanded    dynamic   list,    then
        nextchar(readitem) is used to  get the characters until  newline
        or <termin> is encountered  (see REF * ITEMISE). Leading  spaces
        are ignored; a  backslash \  character may be  used to  escape a
        following newline, backslash, or leading  space when this is  to
        be included in the string. Trailing spaces are also ignored.

        Otherwise, if proglist actually contains some items already, the
        result is

                rdstringto([^newline ^termin])

        (with popnewline set true).


yesno(item) -> bool                                          [procedure]
        Prints item (using  =>) then  uses readline  to read  a line  of
        input; if the input is [yes] or [y] it returns true, if [no]  or
        [n] it returns false, otherwise it prompts for input again.


proglist_read_by(read_p) -> list                             [procedure]
        Given any  procedure read_p  which  reads items  from  proglist,
        executes read_p and returns a separate list of the items it read
        (which are removed from proglist).

        pop_syntax_only is locally set  true while read_p is  executing.
        Thus if read_p is a procedure that compiles (e.g.) Pop-11  code,
        the only effect  will be to  return the list  of items it  read,
        without actually planting any Poplog VM code.


exprread() -> list                                           [procedure]
        Returns a list of Pop-11 items constituting an expression,  read
        from proglist, for use  with LIB * FORM. See HELP * FORM.  (This
        procedure is just proglist_read_by(%pop11_comp_expr%).)


impread() -> list                                            [procedure]
        Returns a  list  of  Pop-11  items  constituting  an  imperative
        sequence separated by commas, read  from proglist, for use  with
        the form macro. See HELP * FORMS.


impseqread() -> list                                         [procedure]
        Returns a  list  of Pop-11  items  constituting an  sequence  of
        imperatives, separated  by  ",",";","=>", or  "==>",  read  from
        proglist, for use with the form macro. See HELP * FORMS.


listread() -> item                                           [procedure]
        Reads either  a  single  item from  proglist,  or  a  bracketted
        sequence of them. The outer (and nested) brackets may be  either
        [ ... ] or { ... }. (Note that this is not the same as a  Pop-11
        list or vector  constructor expression, in  that the  'evaluate'
        keywords %, ^, and ^^ are not recognised.) See HELP * LISTREAD.


numberread() -> number                                       [procedure]
        Reads and returns the next  item from proglist, which must  be a
        number.


readimp(bool) -> list                                        [procedure]
        Returns a list  of Pop-11 items  constituting an imperative  (if
        bool is true), or a sequence of imperatives (if bool is  false),
        read from proglist.


varread() -> word                                            [procedure]
        Returns a word read from proglist, for use with LIB * FORMS.




-------------------------------------
3  Reading Items with Macro Expansion
-------------------------------------

A macro is a word which is currently declared as a lexical or  permanent
identifier with identprops "macro" (see REF * WORDS, * IDENT). When such
a word is read from proglist using the procedures itemread and nextitem,
it undergoes macro expansion, that is, the word is replaced in  proglist
by a sequence  of items derived  from the value  of the identifier.  The
item actually returned by the call  of itemread or nextitem is then  the
first of these (unless  this is again a  macro, in which case  expansion
proceeds recursively until the first item is not a macro). Thus a  macro
can substitute itself in the input stream with any desired items.

    The sequence of items into which a macro is expanded depends on  its
identifier value, which may be either a procedure, a list, or any  other
item (except an undef record):

    # For a procedure, the expansion is  all the items left on the  user
      stack by calling  it (i.e. the  topmost item on  the stack is  the
      last in the  sequence). The  procedure is called  with the  next N
      items from proglist (read with itemread) as its arguments, where N
      is its * pdnargs.

    # For a  list,  the  expansion  is all  the  elements  of  the  list
      (possibly none).

    # For any other  item, the  expansion is  that item.  A mishap  will
      result if the item is an undef record (on the assumption that this
      means the macro's value is undefined).

Expansion is actually performed by collecting the items in a list  list,
and then adding them to proglist with

        list <> proglist -> proglist;

after removal of  the macro  word itself  and any  arguments. (Thus  the
original list pairs constituting proglist are not updated, which can  be
important in some contexts -- see proglist_macro_pair below.)

    Note that before checking  for a macro,  nextitem and itemread  will
attempt to  autoload any  word which  is not  currently declared  as  an
identifier (by  calling  * sys_autoload). This  permits  autoloading  of
(permanent) macros, and is also the mechanism by which autoloading takes
place in  Pop-11. (When  unwanted, this  can be  turned off  by  locally
assigning false to * pop_autoload.)

    If a macro word is required  to be read from proglist without  macro
expansion, it must be preceded by the word "nonmac" (see below).


itemread() -> item                                           [procedure]
        After  expanding  macros  (and  possibly  autoloading  undefined
        words),  returns  and  removes  the  next  non-macro  item  from
        proglist. <termin> is  returned if  proglist is  null. The  item
        returned is saved in poplastitem.


nextitem() -> item                                           [procedure]
        As for  itemread, but  does not  remove the  returned item  from
        proglist (so that another call of nextitem will return the  same
        item again).


proglist_macro_pair -> list                                   [variable]
list -> proglist_macro_pair
        As described  above, macro  expansion  always creates  new  list
        pairs to add to proglist, and does not update the original ones.
        This variable provides a means for macro procedures to  actually
        update the original proglist if they wish to.

        When a macro procedure is applied, proglist_macro_pair holds the
        list pair on proglist that actually contained the macro word (if
        the macro has no  arguments, the next pair  on proglist will  be
        the tl  of this  one).  Thus for  example, a  macro  producing a
        single item  can, as  well as  returning it  as a  result,  also
        assign it directly to

            hd(proglist_macro_pair)

        (alternatively,  it   can  do   the  latter   and  then   assign
        proglist_macro_pair to proglist, returning no result).

        Of particularly relevance in this  context are macros that  read
        characters from  the input  stream using  * nextchar, which  can
        fail to work properly with certain compilers (e.g. Pop-11)  that
        expect to be able to read  sections of proglist more than  once.
        The reason for this  is that on a  second or subsequent  reading
        the original  proglist still  contains the  macro word,  but  no
        longer has available the characters required by the macro (since
        it has been itemised ahead of  that point). Macros of this  type
        should therefore use  proglist_macro_pair to replace  themselves
        directly, so that  re-reading of the  item stream will  function
        correctly.




------------------------------------
4  Proglist-Related Permanent Macros
------------------------------------

nonmac                                                           [macro]
        Escapes the next word on proglist as a macro, so that when  read
        with nextitem or itemread the word will not be expanded. E.g. if
        proglist is

                    [nonmac foo ...]

        where "foo" is a macro, then itemread() will produce "foo".  (On
        the other hand, note that readitem() would produce "nonmac".)


#_<                                                              [macro]
>_#                                                              [macro]
        #_< implements on-the-fly macros by allowing the evaluation of a
        sequence of Pop-11 statements in the input stream. Its usage is

                    #_< statement-sequence >_#

        The statement sequence up to >_# is compiled and evaluated,  and
        any items left on  the stack in so  doing are spliced back  onto
        proglist in  place  of  the whole  construct.  For  example,  if
        proglist is

                    [ #_< 3 + 4 * 5 >_# ...]

        then itemread() will return 23. Note that >_# is simply a  macro
        that produces <termin>,  so that #_<  can compile the  statement
        sequence to <termin>; in fact, this macro is just defined as

           pop11_exec_compile(termin, pop11_comp_stmnt_seq_to, true) ->


#_IF                                                             [macro]
#_ELSEIF                                                         [macro]
#_ELSE                                                           [macro]
#_ELSE_ERROR                                                     [macro]
#_ENDIF                                                          [macro]
        These macros enable the conditional reading or skipping of items
        in proglist. Their usage is

                #_IF expression
                    item-sequence
                #_ELSEIF expression
                    item-sequence
                #_ELSEIF expression
                    item-sequence
                    ...
                #_ELSE (or #_ELSE_ERROR)
                    item-sequence
                #_ENDIF

        where the #_ELSEIF  and #_ELSE clauses  are optional, and  where
        each Pop-11  expression  is  terminated  by  a  newline  as  for
        readline (qv). The item-sequence may be empty.

        Starting from the #_IF, each  expression occurring after a  #_IF
        or #_ELSEIF is evaluated until one is found that returns a  true
        result, item sequences  following a false  result being  skipped
        without macro expansion. The sequence following the true result,
        or that following a #_ELSE  if no expression evaluated to  true,
        is then 'allowed through' in the sense that the current call  of
        itemread etc and  subsequent calls will  return these items  (if
        there is no true clause or #_ELSE then whatever item follows the
        #_ENDIF will result).

        #_ELSE_ERROR may be used instead of  a #_ELSE: in this case,  if
        no expression  in the  corresponding  #_IF or  #_ELSEIF  clauses
        evaluates to  true, the  mishap

            NO CLAUSE SATISFIED IN #_IF

        results.

        These macros may be  nested to any depth,  i.e. any of the  item
        sequences can contain further #_IF constructs.


#_TERMIN_IF                                                      [macro]
        This macro is followed by an expression like #_IF, i.e.

                #_TERMIN_IF expression

        If expression evaluates to true,  [] is assigned to proglist  to
        terminate the current item stream.

        It also clears  the internal stack  associated with #_IF,  which
        means that  no  #_IF-associated  errors  will  result  from  the
        premature end-of-file (but note  that this is NOT so for  Pop-11
        closing brackets, e.g.

            section;
            #_TERMIN_IF true
            endsection;

        would produce an error).


DEF                                                              [macro]
DEFV                                                             [macro]
        These macros enable  the status  of an identifier  to be  tested
        with (for example) #_IF.

        DEF takes an  identifier name  (either as  a word  or a  section
        pathname) and  returns  true  if  the  identifier  is  currently
        declared and has a non-false value. Otherwise it returns  false.
        E.g.

            #_IF DEF UNIX

        will satisfy the #_IF if UNIX  is declared as an identifier  and
        has  a  non-false  value.  The  test  for  being  declared  uses
        * sys_current_ident, and so works for both lexical and permanent
        identifiers.

        DEFV is similar,  but combines the  test for declaration  with a
        comparison of the identifier's value against a given number.  It
        takes three  arguments:  the first  is  an identifier  name  (as
        before), the second is one of the comparison operator names

                =   /=   <   <=   >   >=

        and the third is a (non-complex) number. It returns true if  the
        same conditions as for  DEF hold, and in  addition the value  of
        the identifier (which must be numeric) satisfies the  comparison
        with the number. E.g.

            #_IF DEFV VMS >= 5.3

        will satisfy the #_IF if VMS is declared and has a numeric value
        >= 5.3.

        (Note, however, that before doing the comparison both values are
        multiplied by 1000  and then rounded  to integers. This  enables
        floating-point values -- which may  not be quite accurate --  to
        be compared  sensibly. The  principal use  of this  macro is  in
        comparing floating-point version numbers.)

        INCLUDE * SYSDEFS defines a set of system-specific macros  (such
        as UNIX, VMS, etc) which can be used in conjunction with DEF and
        DEFV    for    controlling    conditional    compilation.    See
        HELP * SYSDEFS.




-----------------------------------------------------
5  Including a File in the Current Compilation Stream
-----------------------------------------------------

#_INCLUDE                                                        [macro]
        This macro enables the stream of  items from a given file to  be
        spliced into proglist at its current point.

        It takes  one  argument,  a  string  or  a  word  representing a
        filename, which is passed to discin to open a character repeater
        CHAR_REP for the file, e.g.

            #_INCLUDE 'foo.ph'
            #_INCLUDE baz

        Note that discin  appends pop_default_type to  a word  argument;
        #_INCLUDE locally adds an 'h' to  its current value, so that  if
        e.g. pop_default_type  is  '.p', then  the  default for  a  word
        argument to #_INCLUDE will be '.ph'.

        The  current  values  of  proglist,  cucharin,  popfilename  and
        poplinenum are then saved on a stack (-pop_#_include_stack-, see
        below), and the latter 3  variables are set to have  appropriate
        values for the new character repeater CHAR_REP. proglist is then
        set with

            pdtolist(incharitem(CHAR_REP)) -> proglist

        so that the  item returned  by the itemread,  etc which  invoked
        this macro  will  be the  first  item  of the  file.  When  this
        proglist is  exhausted, the  previously  saved values  of  the 4
        variables are unstacked  and restored,  and the  next item  read
        will be the item following the #_INCLUDE argument.

        Note that if isincharitem(itemread)  is true before proglist  is
        set, i.e.  the existing  proglist was  producing items  from  an
        incharitem repeater (see REF * ITEMISE), then the item  repeater
        constructed from CHAR_REP is  given the same itemiser  character
        types as the existing  one. Moreover, any  local changes to  the
        character types of the new one are copied back when the original
        proglist is restored. This makes #_INCLUDEing files  transparent
        to local character type changes.

        The macro include allows a search list of directories to be used
        in connection with #_INCLUDE. See HELP * INCLUDE.


include                                                          [macro]
        This macro takes one argument, i.e.

            include FILENAME

        It searches through the directories named in popincludelist  for
        an occurrence of a file with the given name, after appending, if
        necessary, an appropriate file-type  such as '.ph' as  explained
        in connection with * #_INCLUDE. include compiles the first  such
        file found, as part  of the current  compilation stream so  that
        top level lexically  scoped declarations are  accessible in  the
        file containing the include  command. This makes it  unnecessary
        to specify the full  path name of the  file to be included.  See
        HELP * INCLUDE and REF * LIBRARY for further detail.


popincludelist -> list                                        [variable]
list -> popincludelist
        This user-assignable variable is used  by the macro include.  It
        holds a list  of directories  in which to  search for  "include"
        files. See * #_INCLUDE, * include, LIB * INCLUDE.


loadinclude                                                      [macro]
        This macro behaves  exactly as  include, except  that the  given
        file is compiled in  a separate compilation  stream (as if  with
        compile), rather than being included in the current one.




---------------------------
6  Proglist State Variables
---------------------------

proglist has a number of associated global variables, some of which  are
accessible to the user, some of which are not. The complete state of the
input stream represented by proglist is returned by the active  variable
proglist_state.  A  complete  state  for  a  new  input  source  can  be
constructed with the procedure proglist_new_state, and then assigned  to
proglist_state.


proglist -> list                                              [variable]
list -> proglist
        The input list  of items  (possibly dynamic)  used by  readitem,
        nextread, itemread and nextitem, etc.


cucharin() -> char                                  [procedure variable]
        Contains the character  repeater procedure currently  associated
        with proglist. That is, if proglist  is a dynamic list which  is
        producing items from a character repeater, then cucharin is that
        repeater. When  proglist  is  not dynamic,  cucharin  will  be a
        procedure that just returns <termin> when called.

        Compilers set up  this variable when  setting up proglist  for a
        new input  stream,  e.g. with  proglist_new_state  (see  below).
        Changing this variable afterwards generally has no effect, since
        compiler procedures do not reference it again.


popfilename -> false_or_string                                [variable]
        If cucharin is a  character repeater taking  input from a  named
        file or  device,  then  this variable  contains  the  associated
        filename. Otherwise, it contains false.

        Compilers set up  this variable when  setting up proglist  for a
        new input stream, e.g. with proglist_new_state (see below).

        It is  used  in particular  by  sysprmessage, which  assumes  it
        contains the name of the file currently being compiled, and  (if
        not false), includes the line

                FILE: popfilename LINE NUMBER: poplinenum

        in mishap messages.


poplinenum -> int                                             [variable]
        If cucharin is a character repeater taking input from a file  or
        device, then  this variable  contains  the current  line  number
        within the file. It is incremented by discin character repeaters
        and charin each time a newline is read (see REF * CHARIO).

        Compilers set  this  variable  to 1  (or  as  appropriate)  when
        setting  up  proglist  for  a   new  input  stream,  e.g.   with
        proglist_new_state (see below).

        Assumed by sysprmessage  to contain the  line number within  the
        current file being compiled (cf popfilename above).


pop_#_include_stack -> stack                        [protected variable]
        This variable holds  the stack used  by #_INCLUDE, as  described
        above.

        false represents an  empty stack;  when the  stack is  non-empty
        (i.e. when the current proglist  comes from a #_INCLUDEd  file),
        its value  is  a  vector  containing the  saved  values  of  the
        proglist-related global variables, i.e.

            { ^proglist_state }

        (which includes the stack variable itself).

        Compiler startup procedures locally  set this variable to  false
        by assigning  a  proglist_new_state  for an  input  source  to a
        localised proglist_state, i.e.

            dlocal proglist_state = proglist_new_state(source);

        The  test  for  pop_#_include_stack  having  a  true  value  can
        therefore be used to determine whether the current file is being
        #_INCLUDEd or directly compiled.


proglist_state -> (item1, ..., itemN)                  [active variable]
(item1, ..., itemN) -> proglist_state
        This active variable produces all the variables associated  with
        proglist,  including  proglist  itself,  cucharin,  popfilename,
        poplinenum, pop_#_include_stack, as well  as the internal  stack
        associated with #_IF.

        NOTE that the number and  disposition of the values produced  by
        this variable are intended  to be opaque to  the user (i.e.  you
        should not assume there  are any particular  number of them,  or
        anything about  what the  values are).  The number,  meaning  or
        ordering of  the values  may change  in future  versions of  the
        system.

        proglist_state is intended to be used only in two ways,  neither
        of which conflicts with  the above: (1) to  save the state  in a
        vector, as in

                { ^proglist_state } -> save;

        (after which

                explode(save) -> proglist_state;

        will restore the state), or (2) locally to a procedure, with  or
        without an assignment from proglist_new_state, e.g.

            dlocal proglist_state = proglist_new_state(list);

        The above  is  the correct  way  to  locally assign  a  list  to
        proglist; just localising and assigning proglist itself, i.e.

                dlocal proglist = list;

        MAY work  some  of  the  time, but  will  not  necessarily  work
        correctly in  all  contexts (e.g  where  the #_INCLUDE  or  #_IF
        stacks are non-empty).


proglist_new_state(source) -> (item1, ..., itemN)            [procedure]
        For a new  input source source,  returns a set  of values to  be
        assigned to proglist_state.

        In this set, pop_#_include_stack is false and the internal stack
        associated with #_IF  is empty. The  state values for  proglist,
        cucharin, popfilename and poplinenum are derived from the source
        argument, as follows:

        If source is a list (i.e. of items), then proglist is that list.
        In this  case,  cucharin  is  a  procedure  that  just  produces
        <termin> when called, and popfilename and poplinenum have  their
        current values.

        Otherwise, source may be  either a character repeater  procedure
        directly, or an argument to be  given to discin to produce  one.
        (I.e.  filename   string,  word,   or  device   record  --   see
        REF * discin. Note that  when given a  word, discin will  append
        pop_default_type to it for the resultant filename.)

        In either  case, the  character repeater  char_rep is  then  the
        value for cucharin, and the proglist value is the dynamic list

                pdtolist(incharitem(char_rep))

        The values for popfilename and  poplinenum are derived from  the
        pdprops of  char_rep as  follows: if  the props  are a  filename
        string (as will be the case  for a discin result), then this  is
        popfilename and poplinenum is 1; if  the props are a pair,  then
        the front is taken  as popfilename and  the back as  poplinenum;
        otherwise, popfilename is false and poplinenum is 0.

        As with proglist_state, the actual number and disposition of the
        values produced by this  procedure are opaque  to the user.  The
        state produced  should only  be assigned  to proglist_state  (or
        saved in a vector, etc).



--- C.all/ref/proglist
--- Copyright University of Sussex 1993. All rights reserved.