=encoding utf8 =head1 TITLE Synopsis 22: Distributions, Recommendations, Delivery and Installation =head1 VERSION Created: 15 March 2014 Last Modified: 27 September 2015 Version: 6 =head1 TERMINOLOGY Because many of the concepts used in this document may be overloaded by other concepts in the mind of the reader, it seems like a good idea to define some terminology first. Please note that these definitions only apply within the context of Perl 6. =head2 compilation unit A piece of code that can be compiled as a unit. It can either exist in a file (and be compiled using C or C), or be presented as a stream of characters for compilation with C. An example of a compilation unit in a file: lib/JSON/Fast.pm6 =head2 distribution A distribution is an archive of some form that can be used to install zero or more compilation units (each stored in a separate file), with any possibly associated files needed for execution. For example: lib/JSON/Fast.pm6 lib/JSON/PurePerl.pm6 It has a name for identification, which may or may not coincide with the compilation units in the distribution. An example of a distribution name: JSON-Fast It also has a version, to distinguish it from other distributions with the same name. For instance: 1.23 Which, together, are used to create the filename of the distribution, for instance: JSON-Fast.1.23.tar.gz Please note that by changing the C<::> from the module specification to a C<-> for the filename of the archive, we are effectively disallowing an owner to upload a distribution for "JSON-Fast" and "JSON::Fast" at the same time. This seems unlikely to become a problem. A Perl 6 distribution B contain a configuration file named C, containing JSON-encoded information about the contents of the distribution. =head2 owner The owner of a distribution is responsible for the development of a distribution. This can either be a single person, or a (semi-)official organisation. The owner of a distribution has a (mnemonic) name, e.g.: JRANDOM Please note that the owner is not necessarily the developer of a distribution, although if the owner is a single individual, this is pretty likely. =head2 content storage A service to which an owner can upload a distribution to and other people can download B distributions from. This is most likely some online web-service, but it does not need to be. It has a logical name that is essentially a URL. An example would be: cpan The (mnemonic) name of the owner usually coincides with the userid or login name used to upload distributions, but does not need to be. =head2 auth The C of a distribution, is a globally unique identifier for a person or organization. It can be constructed from the name of a content storage and the name of the owner, separated by a colon. For example: cpan:JRANDOM Please note that this is B an authority, merely an indication of the location where the distribution for that owner was obtained. Typically the auth of a distribution is used to try to load compilation units of already installed distributions, such as in: use JSON::Fast:auth; =head2 identity The identity of a distribution, is the combination of name of the content storage, the name of the owner, name and version of a distribution, separated by colons. For example: cpan:JRANDOM:JSON-Fast:1.23 There should really be only one unique distribution for a given identity in the world. The content storage should accept an identity and either directly return the archive for that distribution, or return a URL from which that distribution can be downloaded. =head2 recommendation manager A service that will translate a request for a compilation unit (with optional owner and/or version and/or content storage specification, like a C statement) into a list of zero or more identities of distributions that match the request. The recommendation manager is only used during the installation process of the distribution for a wanted compilation unit. A recommendation manager can be run by a community (like the current Perl 6 ecosystem or the packages list for Perl 5 on CPAN), or by company (for use inside the company itself), or by any reviewing / grading service (for use by anybody wanting to use that service), or by any other person willing to put in the effort. A request for: JSON::PurePerl would yield the identity: cpan:JRANDOM:JSON-Fast:1.23 because the compilation unit C is part of the C distribution. However, a request for: JSON::Fast:auth would not find anything, because it has the wrong content storage specification ("github" instead of "cpan"). Please note that a recommendation manager does not need to be bound to a single content storage. In fact, a recommendation manager would be best if being able to supply identities from the best of B worlds. And potentially be able to recommend identities responding to more natural language queries, but that is probably outside the scope of this specification. A bundle of distributions is basically just a collection of identities that listen to a name (such as C). A recommendation manager may provide bundles as part of its service. And packagers may use this information as the source for bundling distributions in their specific packaging system. =head1 DISTRIBUTION A Perl 6 distribution consists of an archive of some form (presumably a .tar.gz, .tar.bz2 or .zip file) which is expected to at least contain a file called "META6.json". The existence of this file indicates that this distribution is a Perl 6 distribution. This is important for those archive networks that also serve as a content-distribution system for other types of distributions (such as PAUSE / CPAN), so that they can adapt the processing of the contents, or decide to ignore any processing at all (such as CPAN-testers not being able to test Perl 6 distributions (yet)). =head2 META6.json The META6.json file is a JSON-file that must at least contain a C, C, C and C section. =head3 meta-version Required. The version of the META6.json schema in use, as a quoted Version literal (without the "v" prefix). This document describes version 1. =head3 perl Mandatory. The minimal perl version for which this distribution can be installed. Specified as a version string. So: "perl" : "6.d" would not allow installation on Perl version 6.c =head3 name Mandatory. The name identifying this distribution. This is usually the name of the main module that this distribution provides. An example of this would be: "name" : "JSON::Fast" =head3 version Mandatory. The version literal for this distribution. An example of this would be: "version" : "1.23" =head3 description Mandatory. A one line description of this distribution. So, for instance: "description" : "Providing fast JSON encoding/decoding" =head3 authors Optional. A list of (mnemonic) names of people who work / have worked on this distribution. For informational purposes only. An example: "authors" : [ "Janet Random", "Sam Helpedwithit" ] Please note that for identification purposes, only the owner (who uploaded the distribution to the content storage) should be used. =head3 provides Mandatory. A list of module - local filename pairs that this distribution makes available to be Cd. For example: "provides" : { "JSON::Fast" : "lib/JSON/Fast.pm6", "JSON::PurePerl" : "lib/JSON/PurePerl.pm6" } Please note that the filenames specified only indicate the names of the files in the B. The installer may need to mangle filenames to be able support file systems that do not support the file names given (e.g. when they contain Unicode characters where the file system does not support them). This also implies that any installer needs to keep a local database that is able to convert from the module names given, to the actual associated file). =head3 depends Optional. A list of run-time dependencies, specified as C strings. To indicate alternatives, it is possible to specify an object with an C key and a list of C strings as a value, instead of just a single use string. So: "depends" : [ "Sereal:auth:ver(1..*)", {"any": ["Archive::Compress", "Archive::Zlib"]} ] would indicate a dependency on Sereal, and either Archive::Compress or Archive::Zlib. An alternative itself may again be a list which in turn may again contain alternatives, i.e. the dependencies form a fully recursive tree. The dependencies may also be split between different phases, e.g. build time dependencies are only needed on the machine where a package is built, not on the one where it's installed. "depends": { "runtime": { "requires": [ "Sereal:auth:ver(1..*)", "JSON::Fast" ], "recommends": [ "JSON::Pretty" ] }, "build": { "requires": [ "System::Info" ] }, "test": { "requires": [ "File::Temp" ] } } An installer has the option to automatically install any dependencies, if the user has so indicated. Dependencies and alternatives should be tried in the order they are specified. In the case of alternatives, the first one for which the recommendation manager returns an identity should be installed. Failure of installation of an alternative may allow automatic attempts on other alternatives. Please note that the C strings of compilation units are specified. It is the responsibility of the recommendation manager to turn these into identities of distributions that can be downloaded. Dependencies need not be Perl 6 only. By using the C<:from> adverb on the C string, dependencies external to Perl 6 can be specified. These could be Perl 5 modules like C>. Two names are reserved: =head4 :from With C> one can specify on a C executable be available at the given stage. The installer may only check the availability of this dependency, or it may try to provide this, e.g. by installing the appropriate package to the system. =head4 :from A dependency like C> tells the installer, that the system library with the given name must be present. The name is expanded following the same rules as NativeCall uses, e.g. C will be expanded to C on UNIX-like systems and C on Windows. =head4 System specific values All information in C and C is subject to system specific collapsing, i.e. at any point one can specify an object with a C key and an object as the value. The "somethings" in the key may be C, C for checking environment variables or C, C, C, C for Raku's C<$*DISTRO>, C<$*KERNEL>, C<$*RAKU> and C<$*VM> objects respectively. The value object is structured like a C statement with possible values and/or a default value (the empty string) as the key and the value to replace the whole object with on a match. An example where the name of the only dependency is picked by the currently running VM: "depends" : { "name" : { "by-vm.name": { "moar": "Low::Level::Backend::MoarVM", "jvm": "Low::Level::Backend::JVM", "": "Low::Level::Backend::Generic", } } } Dynamically determined values may appear anywhere in the dependency specification. e.g. the whole C section may depend on the distro name. Or just the version of one of the dependencies may depend on the VM backend. =head4 Dependency hints The META data may provide hints to the installer for how to satisfy the dependency. So if there's a location where a library can be obtained from for a certain system, this information may be provided as a dependency hint: "depends": [ { "name": "archive:from", "hints": { "by-kernel.name": { "win32": { "url": "http://www.p6c.org/~jnthn/libarchive/libarchive.dll", "checksum": {"sha-256": "E6836E32802555593AEDAFE1CC00752CBDA"}, "target": "resources/libraries/" } "": { } } } } ] The installer would download the library from the specified C and store it in the C directory of the build tree. This resource will also automatically added to the C section of the meta data so it can be accessed through C<%?RESOURCES>. If the library needs to be built, additional build information can be supplied. The same infrastructure as for the distribution itself can be used. "depends": { "runtime": [ { "name": "svm:from", "hints": { "source": { "builder": "Distribution::Builder::MakeFromJSON", "build": { "src-dir": "src", "makefile-variables": { "VERSION": "3.22", "svm": {"resource": "libraries/svm"} } } } } ] } =head3 emulates Optional. A hash in which the key is the compunit (provided by the distribution) to be aliased, and the value is the use string that should match to get that compilation unit. An example of this would be: "emulates" : { "JSON::Fast" : "JSON::XS:auth" } If then later, a program would say: use JSON::XS; it would in fact load the C compunit, but make it appear as a C compunit, but only if there was no "real" C compunit installed that would match the C specification. In other words: if the real thing is available, then it will be used. If it is not, it will fall back to the indicated compilation unit. And it will look like you are using the thing you asked for. Conversely, if one would do a: use JSON::Fast; then later doing a: use JSON::XS:auth; in the same scope would become a no-op, just as if the compunit had already been loaded. Please note that it is responsibility of the emulating compunit to actually provide a compatible interface with emulated compunit. =head3 supersedes Optional. Has the same meaning as "emulates" for the L. But has additional meaning for external packagers: it indicates a potential superseding of the indicated compilation unit from the packagers point of view. See L. =head3 superseded-by Optional. Has the reverse meaning of "emulates" for the L. It is a hash in which the key is compunit provided by the distribution, and the value is the C string of the compunit it should be aliased to if that compunit is available. So in this case: "superseded-by" : { "JSON::Fast" : "SuperJSON:ver(v1.0 .. *)" } it would mean that if a program attempts to load the C compunit of this distribution, it should instead use any C compunit that is installed that has a version of 1.0 or higher. In other words: please don't use my compunit, unless you really have to. This tag has additional meaning for packagers: if a packager detects a valid C and C pair in its collection of distributions to be packaged, the packager may decide to only supply the distribution providing the C compilation unit. Please note that C has no meaning as a C, so an installer should probably not automatically install any C compunits. =head3 excludes Optional. A hash in which the key is a compilation unit provided by this distribution, and the value is a C string of all compilation units that will be disallowed when attempted to be loaded in the same lexical scope. An example of this would be: "excludes" : { "JSON::PurePerl" : "JSON::Slow:auth:ver(1..*)" } So, if a lexical scope loads C from this distribution, then attempting to load C will cause a Failure. Please note that this has B meaning for packagers: it is simply a way to provide a better error message if a collision of some sort will occur when both modules are loaded in the same lexical scope. =head3 build-depends Deprecated. A list of build-time dependencies, specified as C strings. An example of this would be: "build-depends" : [ "Archive::Zip" ] =head3 test-depends Deprecated. A list of test-time dependencies, specified as C strings. An example of this would be: "test-depends" : [ "Test:auth" ] =head3 resources Optional. A list of file names, each of them corresponding to a file in the "resources" directory in the distribution. At build time any not-yet existing files should be created. The installer will install all files into a location at its discretion (taking care of file-system case-insensitivity and Unicode-support issues), and make the files available through the L hash. =head3 support Optional. A hash of key-value pairs regarding support for this distribution. Possible names are: C, C, C, C, C, C. =over 4 =item email The email address of the owner of this distribution, if any. =item mailinglist The mailinglist of users of this distribution, if any. =item bugtracker The bugtracker of this distribution, if any. =item source The URL of the source of this distribution, if any. =item irc The URL of the IRC channel where this distribution can be discussed, if any. =item phone A fully qualified phone number (with potential cost indication) that can be used with queries about this distribution. =back =head3 production Optional. A Boolean to indicate whether or not this is a distribution intended to be used in production. For instance: "production" : true By default, a distribution is not ready for production. If a distribution is not ready for production, then it will never be recommended. Please note that this section is only to be used by B, giving them the opportunity to decide whether or not to install that distribution. Once a distribution is installed, it can be loaded just like any other distribution. =head3 license Recommended, even if the license status is unknown, or there is none (see below). The L license identifier that the package is distributed under. I<"By providing a short identifier, users can efficiently refer to a license without having to redundantly reproduce the full license".> ― spdx.org SPDX license identifiers are meant to be interoperable and are the most widely used standardized identifiers for licenses. "license" : "Artistic-2.0" If the license isn't on the SPDX standardized list, it is highly recommended to include a URL to the text of the license under the support key in addition to the name of the license: "license" : "Artistic-2.0", "support": { "license": "http://www.perlfoundation.org/artistic_license_2_0" }, In the case that it is known that the author has choosen no license, C should be used: "license" : "NONE" In the event that a distributor or some other person does not know the license, and therefore makes no assertion about whether the project is or is not licensed, C should be used: "license" : "NOASSERTION”" =head3 tags Optional. A list of general purpose tags. For instance: "tags" : [ "json", "perl6" ] It has no meaning other than the meaning B assign to it. =head2 Special directories A distribution may contain several directories that will be handled specially. =head3 bin Any file inside this directory, will be installed as a callable shell application. =head3 t All C<.t> files in this directory, will be tested in alphabetical order, possibly in parallel. =head3 hooks All files in this directory should contain executable Perl 6 code, to be executed at various stages of the install process of this distribution. =head2 Installation Distributions can be installed with an installer. This may be a command-line script, or be some kind of GUI. The installer needs the user to (implicitly) select the C in which the distribution should be installed, and the recommendation manager that should be used to convert requests for a compilation unit into an identity of a distribution to be downloaded. =head1 CLASSES =head2 CompUnit::Repository Base class (interface, really) for the object living in the L array. Used both for installing compunits, as well as finding a certain compunit by way of its from, longname, auth and ver information. =head3 Specifying a CompUnit::Repository The specification of a CompUnit::Repository can happen at various places: in a configuration file, on the command-line when starting (with C<-I>), and in code when trying to create a CompUnit::Repository object programmatically. Each specification consists of a CompUnit::Repository class specification (either explicitly, implicitly, or the short-hand form using the L identifier), optional named parameters and a location indicator (usually a path or a URL). Some examples (where CUR is short for CompUnit::Repository): /foo/bar simple CUR::FileSystem in /foo/bar file#/foo/bar (same) inst#/installed simple CUR::Installation in /installed inst#name#/installed (same) but also set %?CUSTOM_LIB inst#name[work]#/installed (same) but more CLI-friendly inst#name{work}#/installed (same) alternate CLI-friendly way CompUnit::Repository::Installation#/installed (same) but with full class name CompUnit::Repository::GitHub#masak/html-template get it from GitHub Multiple specifications may be concatenated with C<,>. If no class is specified on subsequent specifications, then the previous class specification will be assumed. So: /foo/bar,/foo/baz both use CUR::FileSystem inst#/installed,/also both use CUR::Installation /foo/bar,inst#/installed first CUR::FileSystem, second CUR::Installation =head3 new my $repo = CompUnit::Repository.new( $specification ); Create a new CompUnit::Repository-like object with the given specification, either for inclusion in L, or to install a distribution. Returns the instantiated object. Please note that for a given C, there should always only be one C object in a process. =head3 short-id say CompUnit::Repository::FileSystem.short-id; # "file" Returns a short C<\w+> identifier string (a C if you will) to identify the use of this repo, e.g. in strings provided with the C<-I> command line parameter. It should be unique for all of the CompUnit::Repository subclasses loaded. The following C's are pre-defined: file CompUnit::Repository::FileSystem inst CompUnit::Repository::Installation =head3 install my $installed = $repo.install( $dist ); Install the compilation units of a L in the appropriate way for this CompUnit::Repository-like object. May cause a fatal exception if this repository does not support installing. The parameter contains the L object to be installed. Returns True if the distribution was installed, or a C if it didn't. Of course, any implementation of L's C interface, may decide to accept additional meta-information and store this and make available for later introspection. =head3 uninstall my $removed = $repo.uninstall( $dist ); The opposite of L. =head3 candidates my @candidates = $repo.candidates( $longname, $auth?, $ver?, $from? ); Return L candidates given the matching credentials. =head2 CompUnit::Repository::FileSystem The simplest, default case for locating compilation units on a file system. The module name specified should directly map to a file, similar to how Perl 5 this does. Mainly intended to be used in development situations to allow developers to have modules to be available without them having to be installed. =head3 install Will fail unconditionally: installing compilation units is done by putting files in the right location. =head2 CompUnit::Repository::Installation The default case for installed compilation units. Similar to the way Perl 5 installs modules, but with meta-information per object, rather than globally in a C. This should be used by installers such as C. =head3 install my $installed = $repo.install( $dist ); Install the given L object in this repo. =head2 Distribution The class for installing distributions using L. Basically provides the API to access the C file of a distribution. It should at least contain the following methods: =head3 meta my $meta = $dist.meta; Return a Hash with the representation of the meta-data, constructed the same way as in the C specification. Please note that an actual META6.json file does not need to exist, just a representation in that format. =head3 content my $content = $dist.content( ); my $content = $dist.content( ); Return the octet-stream as specified by the given keys, navigating through the C hash. =head2 Distribution::Local::Tar An implementation of the C API for locally stored C<.tar> files. =head2 CompUnit The object that describes a compilation unit. Contains as much meta-data as possible, e.g. from the C object it came from. =head3 new my $compunit = CompUnit.new( $path ); Create a new CompUnit object with the given specification. Returns the instantiated object, or C if it could not find the C. Please note that for a given C, there should always only be one C object in a process. This to prevent reloading the same compilation unit more than once. =head3 load my $loaded = @candidates[0].load(...); Returns True if loading of the C was successful, or a C if something went wrong. =head3 loaded my $loaded = $compunit.loaded; Return whether the compunit has been loaded. =head3 precomp my $precompiled = $compunit.precomp; Create a pre-compiled version of the CompUnit. Optionally takes an output file argument (default: the same path as the source, with an extension identifying it is a precompiled version), an optional named argument C<:INC> (with default C<@?INC>) and an optional named argument C<:force> to force creation even if the output file already exists. =head1 SYSTEM VARIABLES Several dynamic variables are available. =head2 %?RESOURCES This hash provides compile and runtime access to files associated with the L of the current compilation unit. So, if the C file contains this C section: "resources" : [ "images/fido.png", "images/zowie.png", "libraries/inline_helper ] then the C is supposed to contain the files C and C. After installation, IO objects of such a file would be available through %?RESOURCES The absolute path could be obtained with: %?RESOURCES.absolute and a handle could be obtained with: %?RESOURCES.open without there being any guarantee that this path has anything to do with the path as specified in the distribution. Please also note that the installer will probably mangle filenames of actually installed files. This means you can have files with Unicode characters in upper/lower case, and still be able to access them when installed on a file system that does not support Unicode. =head2 CompUnit::RepositoryRegistry This class manages the repositories. Use C to get the C instance for the "site" repository. =head2 $*REPO This is the head of the linked list of C objects that will be queried in turn whenever an attempt should be made to load a compilation unit (be that at compile time with C, or at runtime with C. The C object pointed to by C<$*REPO> will either satisfy a given dependency itself, or hand the request off to it's C<.next-repo>. =head1 AUTHORS Elizabeth Mattijsen =for vim:set expandtab sw=4: