{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Loading ASL Results into a Model\n", "\n", "## Summary\n", "\n", "In this scripting example we break apart the work flow that occurs when a Pyomo model is solved using the ASL solver plugin. The ASL solver plugin is a generic interface designed for solvers that utilize the AMPL Solver Library. This library takes model input in the form of an NL file and provides a solver solution in the form of an SOL file. As such, it provides a single unifying framework for interacting with a wide array of optimization solvers.\n", "\n", "Pyomo includes separate tools for writing NL files and reading SOL files. In this example, we will show how to use these tools directly, as an alternative to calling the ASL solver plugin. In particular, we show how to save information about the symbol map created by the NL writer to a file so that it can be recovered at a later time. The symbol map that is recovered can be used to load a solution from the SOL file reader into any Pyomo model with component names that match those on the model used by the NL writer.\n", "\n", "## Solving With ASL\n", "\n", "Consider the case below where we solve a simple Pyomo model using Ipopt through the ASL solver plugin and then verify that the solver termination condition is optimal before loading the solution into the model. Note that this example assumes Pyomo version 4.1 or later is installed. Since Pyomo 4.1, the **_load_\\__solutions_** keyword must be assigned a value of _False_ when calling the _solve_ method on a solver plugin in order to prevent the solution from being automatically loaded into the model. This allows us to check the solver termination condition before manually loading the solution via the call to _model.solutions.load_\\__from_." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Objective: 0.9999999925059035\n" ] } ], "source": [ "# %load script.py\n", "from pyomo.environ import *\n", "from pyomo.opt import SolverFactory, TerminationCondition\n", "\n", "def create_model():\n", " model = ConcreteModel()\n", " model.x = Var()\n", " model.o = Objective(expr=model.x)\n", " model.c = Constraint(expr=model.x >= 1)\n", " model.x.set_value(1.0)\n", " return model\n", "\n", "if __name__ == \"__main__\":\n", "\n", " with SolverFactory(\"ipopt\") as opt:\n", " model = create_model()\n", " results = opt.solve(model, load_solutions=False)\n", " if results.solver.termination_condition != TerminationCondition.optimal:\n", " raise RuntimeError('Solver did not report optimality:\\n%s'\n", " % (results.solver))\n", " model.solutions.load_from(results)\n", " print(\"Objective: %s\" % (model.o()))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The basic work flow that takes place above can be summarized as:\n", " 1. Create an ASL solver plugin that uses the _ipopt_ executable appearing in the shell search PATH.\n", " 2. Construct a Pyomo model.\n", " 3. Solve the Pyomo model.\n", " 1. Output the Pyomo model as an NL file.\n", " 2. Invoke the solver (which produces an SOL file).\n", " 3. Read the SOL file into a Pyomo results object.\n", " 4. Check the solver termination condition stored in the results object.\n", " 5. Load the solution stored in the results object into the Pyomo model.\n", "\n", "The remainder of this example shows how to implement step 3 without the use of the ASL solver plugin.\n", "\n", "### A note about using the **_with_** statement\n", "\n", "In the code provided with this example we make use of Python's **_with_** statement when dealing with objects returned from Pyomo _Factory_ functions such as SolverFactory and ReaderFactory. Pyomo makes use of a Plugin system to instantiate these objects. As a result, they must be deactivated before going out of scope in order to prevent a memory leak. Deactivation of Plugins is managed automatically by the **_with_** statement, but can also be done by calling the _deactivate_ method directly on the Plugin object.\n", "\n", "## Writing the NL File\n", "\n", "The code block below defines the function **_write_\\__nl_** that outputs a Pyomo model as an NL file and saves the pertinent symbol map data to a file using pickle. This symbol map data will allow a solution stored in an SOL file to be loaded into any Pyomo model with matching component names. The last section of this code block shows how this function can be used with a small example model." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " NL File: example.nl\n", "Symbol Map File: example.nl.symbol_map.pickle\n" ] } ], "source": [ "# %load write.py\n", "import pyomo.environ\n", "from pyomo.core import ComponentUID\n", "from pyomo.opt import ProblemFormat\n", "# use fast version of pickle (python 2 or 3)\n", "from six.moves import cPickle as pickle\n", "\n", "def write_nl(model, nl_filename, **kwds):\n", " \"\"\"\n", " Writes a Pyomo model in NL file format and stores\n", " information about the symbol map that allows it to be\n", " recovered at a later time for a Pyomo model with\n", " matching component names.\n", " \"\"\"\n", " symbol_map_filename = nl_filename+\".symbol_map.pickle\"\n", "\n", " # write the model and obtain the symbol_map\n", " _, smap_id = model.write(nl_filename,\n", " format=ProblemFormat.nl,\n", " io_options=kwds)\n", " symbol_map = model.solutions.symbol_map[smap_id]\n", "\n", " # save a persistent form of the symbol_map (using pickle) by\n", " # storing the NL file label with a ComponentUID, which is\n", " # an efficient lookup code for model components (created\n", " # by John Siirola)\n", " tmp_buffer = {} # this makes the process faster\n", " symbol_cuid_pairs = tuple(\n", " (symbol, ComponentUID(var_weakref(), cuid_buffer=tmp_buffer))\n", " for symbol, var_weakref in symbol_map.bySymbol.items())\n", " with open(symbol_map_filename, \"wb\") as f:\n", " pickle.dump(symbol_cuid_pairs, f)\n", "\n", " return symbol_map_filename\n", "\n", "if __name__ == \"__main__\":\n", " from script import create_model\n", "\n", " model = create_model()\n", " nl_filename = \"example.nl\"\n", " symbol_map_filename = write_nl(model, nl_filename)\n", " print(\" NL File: %s\" % (nl_filename))\n", " print(\"Symbol Map File: %s\" % (symbol_map_filename))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The first argument to this function is the Pyomo model. The second argument is the name to use for the NL file. Along with the NL file, another file with the suffix \".symbol_map.pickle\" will be created that contains information that can be used to efficiently rebuild the symbol map for any Pyomo model with component names matching those used to build the NL file. Additional options can be passed to the NL writer as keywords to this function. These include:\n", "* **show_section_timing**: Print timing after writing major sections of the NL file. (default=_False_) \n", "* **skip_trivial_constraints**: Skip writing constraints whose body section is fixed (i.e., no variables). (default=_False_)\n", "* **file_determinism**: Sets the level of effort placed on ensuring the NL file is written deterministically. The value of this keyword will affect the row and column ordering assigned to Pyomo constraints and variables in the NLP matrix, respectively.\n", " * 0: declaration order only \n", " * 1: sort index sets of indexed components after declaration order (default)\n", " * 2: sort component names (overriding declaration order) as well as index sets\n", "* **symbolic_solver_labels**: Generate .row and .col files identifying constraint and variable indices in the NLP matrix. (default=_False_)\n", "* **include_all_variable_bounds**: Include all variables that are on active blocks of the Pyomo model in the bounds section of the NL file. This includes variables that do not appear in any objective or constraint expressions. (default=_False_)\n", "* **output_fixed_variable_bounds**: Allow variables that are fixed to appear in the body of preprocessed expressions. Fixing takes place by using a variable's current value as the upper and lower bound in the bounds section of the NL file. This option is experimental. (default=_False_)\n", "\n", "The **symbolic_solver_labels** option, when set to _True_, outputs files containing similar information to what is output by this function to recover the symbol map. The difference is that this function outputs component lookup codes (the ComponentUID class) that are meant to allow efficient recovery of components on models that make use of index Sets and/or Blocks. The .row and .col files are meant as debugging tools and use human readable names that are not efficient for recovering model components.\n", "\n", "## Invoking the Solver\n", "\n", "The solver can be invoked directly from the command shell or by using Python's built-in utilities for executing shell commands. For most ASL-based solvers, we need to use an additional command-line option such as \"-s\" (before the input file) or \"-AMPL\" (after the input file) in order to tell the AMPL Solver Library we want it to store the solution into an SOL file. The code block below issues a bash command that uses the _ipopt_ executable to solve our example model and generate an SOL file. This command requires that the _ipopt_ executable can be found in the shell search PATH and that the code block from the previous section has been executed." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "******************************************************************************\n", "This program contains Ipopt, a library for large-scale nonlinear optimization.\n", " Ipopt is released as open source code under the Eclipse Public License (EPL).\n", " For more information visit http://projects.coin-or.org/Ipopt\n", "******************************************************************************\n", "\n", "This is Ipopt version 3.12.3, running with linear solver ma27.\n", "\n", "Number of nonzeros in equality constraint Jacobian...: 0\n", "Number of nonzeros in inequality constraint Jacobian.: 1\n", "Number of nonzeros in Lagrangian Hessian.............: 0\n", "\n", "Total number of variables............................: 1\n", " variables with only lower bounds: 0\n", " variables with lower and upper bounds: 0\n", " variables with only upper bounds: 0\n", "Total number of equality constraints.................: 0\n", "Total number of inequality constraints...............: 1\n", " inequality constraints with only lower bounds: 1\n", " inequality constraints with lower and upper bounds: 0\n", " inequality constraints with only upper bounds: 0\n", "\n", "iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls\n", " 0 1.0000000e+00 0.00e+00 0.00e+00 -1.0 0.00e+00 - 0.00e+00 0.00e+00 0\n", " 1 1.0001504e+00 0.00e+00 1.50e-09 -3.8 9.85e-03 - 1.00e+00 1.00e+00h 1\n", " 2 1.0000018e+00 0.00e+00 1.84e-11 -5.7 1.49e-04 - 1.00e+00 1.00e+00f 1\n", " 3 9.9999999e-01 0.00e+00 2.51e-14 -8.6 1.84e-06 - 1.00e+00 1.00e+00f 1\n", "\n", "Number of Iterations....: 3\n", "\n", " (scaled) (unscaled)\n", "Objective...............: 9.9999999250590355e-01 9.9999999250590355e-01\n", "Dual infeasibility......: 2.5091040356528538e-14 2.5091040356528538e-14\n", "Constraint violation....: 0.0000000000000000e+00 0.0000000000000000e+00\n", "Complementarity.........: 2.5059035957398297e-09 2.5059035957398297e-09\n", "Overall NLP error.......: 2.5059035957398297e-09 2.5059035957398297e-09\n", "\n", "\n", "Number of objective function evaluations = 4\n", "Number of objective gradient evaluations = 4\n", "Number of equality constraint evaluations = 0\n", "Number of inequality constraint evaluations = 4\n", "Number of equality constraint Jacobian evaluations = 0\n", "Number of inequality constraint Jacobian evaluations = 4\n", "Number of Lagrangian Hessian evaluations = 3\n", "Total CPU secs in IPOPT (w/o function evaluations) = 0.001\n", "Total CPU secs in NLP function evaluations = 0.000\n", "\n", "EXIT: Optimal Solution Found.\n", " \n", "Ipopt 3.12.3: Optimal Solution Found\n" ] } ], "source": [ "%%bash\n", "ipopt -s example.nl" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loading the SOL File\n", "\n", "The code block below defines the function **_read_\\__sol_** that produces a results object that can be loaded into a Pyomo model from a given SOL file. The function first calls Pyomo's built-in SOL file reader to produce a bare results object. Symbols used by the SOL file reader take the form < type character >< index >, where < type character > is one of 'o' (objective), 'v' (variable), or 'c' (constraint) and < index > is the row / column index for constraints / variables in the NLP matrix and 0 for the objective (e.g., 'o0', 'v3', 'c1'). These symbols are mapped to component identifiers in the symbol map file created by the **_write_\\__nl_** function from above. The results object returned from the **_read_\\__sol_** function can be loaded into a Pyomo model just like that returned from the _solve_ method on a Pyomo solver plugin when the **_load_\\__solutions_** keyword is set to _False_. The last section of this code block shows how this function can be used to load a solution into a _copy_ of the model used in the section on writing the NL file. It assumes the code blocks in the previous two sections have been executed." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Objective: 0.9999999925059035\n" ] } ], "source": [ "# %load read.py\n", "import pyomo.environ\n", "from pyomo.core import SymbolMap\n", "from pyomo.opt import (ReaderFactory,\n", " ResultsFormat)\n", "# use fast version of pickle (python 2 or 3)\n", "from six.moves import cPickle as pickle\n", "\n", "def read_sol(model, sol_filename, symbol_map_filename, suffixes=[\".*\"]):\n", " \"\"\"\n", " Reads the solution from the SOL file and generates a\n", " results object with an appropriate symbol map for\n", " loading it into the given Pyomo model. By default all\n", " suffixes found in the NL file will be extracted. This\n", " can be overridden using the suffixes keyword, which\n", " should be a list of suffix names or regular expressions\n", " (or None).\n", " \"\"\"\n", " if suffixes is None:\n", " suffixes = []\n", "\n", " # parse the SOL file\n", " with ReaderFactory(ResultsFormat.sol) as reader:\n", " results = reader(sol_filename, suffixes=suffixes)\n", "\n", " # regenerate the symbol_map for this model\n", " with open(symbol_map_filename, \"rb\") as f:\n", " symbol_cuid_pairs = pickle.load(f)\n", " symbol_map = SymbolMap()\n", " symbol_map.addSymbols((cuid.find_component(model), symbol)\n", " for symbol, cuid in symbol_cuid_pairs)\n", "\n", " # tag the results object with the symbol_map\n", " results._smap = symbol_map\n", "\n", " return results\n", "\n", "if __name__ == \"__main__\":\n", " from pyomo.opt import TerminationCondition\n", " from script import create_model\n", "\n", " model = create_model()\n", " sol_filename = \"example.sol\"\n", " symbol_map_filename = \"example.nl.symbol_map.pickle\"\n", " results = read_sol(model, sol_filename, symbol_map_filename)\n", " if results.solver.termination_condition != \\\n", " TerminationCondition.optimal:\n", " raise RuntimeError(\"Solver did not terminate with status = optimal\")\n", " model.solutions.load_from(results)\n", " print(\"Objective: %s\" % (model.o()))\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.1" } }, "nbformat": 4, "nbformat_minor": 1 }