Merge https://github.com/angavrilov/dfhack

Author: peterix

COMPILE.rst

@@ -152,10 +152,13 @@ Valid and useful build types include 'Release', 'Debug' and
 ================================
 Using the library as a developer
 ================================
-Currently, the only way to use the library is to write a plugin that can be loaded by it.
+
+Currently, the most direct way to use the library is to write a plugin that can be loaded by it.
 All the plugins can be found in the 'plugins' folder. There's no in-depth documentation
 on how to write one yet, but it should be easy enough to copy one and just follow the pattern.
 
+Other than through plugins, it is possible to use DFHack via remote access interface, or by writing Lua scripts.
+
 The most important parts of DFHack are the Core, Console, Modules and Plugins.
 
 * Core acts as the centerpiece of DFHack - it acts as a filter between DF and SDL and synchronizes the various plugins with DF.
@@ -171,6 +174,24 @@ The main license is zlib/libpng, some bits are MIT licensed, and some are BSD li
 Feel free to add your own extensions and plugins. Contributing back to
 the dfhack repository is welcome and the right thing to do :)
 
+DF data structure definitions
+=============================
+
+DFHack uses information about the game data structures, represented via xml files in the library/xml/ submodule.
+
+Data structure layouts are described in files following the df.*.xml name pattern. This information is transformed by a perl script into C++ headers describing the structures, and associated metadata for the Lua wrapper. These headers and data are then compiled into the DFHack libraries, thus necessitating a compatibility break every time layouts change; in return it significantly boosts the efficiency and capabilities of DFHack code.
+
+Global object addresses are stored in symbols.xml, which is copied to the dfhack release package and loaded as data at runtime.
+
+Remote access interface
+=======================
+
+DFHack supports remote access by exchanging Google protobuf messages via a TCP socket. Both the core and plugins can define remotely accessible methods. The ``dfhack-run`` command uses this interface to invoke ordinary console commands.
+
+Currently the supported set of requests is limited, because the developers don't know what exactly is most useful.
+
+Protocol client implementations exist for Java and C#.
+
 Contributing to DFHack
 ======================
 

Compile.html

@@ -334,10 +334,12 @@ <h1 class="title">Building DFHACK</h1>
 </li>
 <li><a class="reference internal" href="#build-types" id="id12">Build types</a></li>
 <li><a class="reference internal" href="#using-the-library-as-a-developer" id="id13">Using the library as a developer</a><ul>
-<li><a class="reference internal" href="#contributing-to-dfhack" id="id14">Contributing to DFHack</a><ul>
-<li><a class="reference internal" href="#coding-style" id="id15">Coding style</a></li>
-<li><a class="reference internal" href="#how-to-get-new-code-into-dfhack" id="id16">How to get new code into DFHack</a></li>
-<li><a class="reference internal" href="#memory-research" id="id17">Memory research</a></li>
+<li><a class="reference internal" href="#df-data-structure-definitions" id="id14">DF data structure definitions</a></li>
+<li><a class="reference internal" href="#remote-access-interface" id="id15">Remote access interface</a></li>
+<li><a class="reference internal" href="#contributing-to-dfhack" id="id16">Contributing to DFHack</a><ul>
+<li><a class="reference internal" href="#coding-style" id="id17">Coding style</a></li>
+<li><a class="reference internal" href="#how-to-get-new-code-into-dfhack" id="id18">How to get new code into DFHack</a></li>
+<li><a class="reference internal" href="#memory-research" id="id19">Memory research</a></li>
 </ul>
 </li>
 </ul>
@@ -470,9 +472,10 @@ <h1><a class="toc-backref" href="#id12">Build types</a></h1>
 </div>
 <div class="section" id="using-the-library-as-a-developer">
 <h1><a class="toc-backref" href="#id13">Using the library as a developer</a></h1>
-<p>Currently, the only way to use the library is to write a plugin that can be loaded by it.
+<p>Currently, the most direct way to use the library is to write a plugin that can be loaded by it.
 All the plugins can be found in the 'plugins' folder. There's no in-depth documentation
 on how to write one yet, but it should be easy enough to copy one and just follow the pattern.</p>
+<p>Other than through plugins, it is possible to use DFHack via remote access interface, or by writing Lua scripts.</p>
 <p>The most important parts of DFHack are the Core, Console, Modules and Plugins.</p>
 <ul class="simple">
 <li>Core acts as the centerpiece of DFHack - it acts as a filter between DF and SDL and synchronizes the various plugins with DF.</li>
@@ -485,18 +488,30 @@ <h1><a class="toc-backref" href="#id13">Using the library as a developer</a></h1
 The main license is zlib/libpng, some bits are MIT licensed, and some are BSD licensed.</p>
 <p>Feel free to add your own extensions and plugins. Contributing back to
 the dfhack repository is welcome and the right thing to do :)</p>
+<div class="section" id="df-data-structure-definitions">
+<h2><a class="toc-backref" href="#id14">DF data structure definitions</a></h2>
+<p>DFHack uses information about the game data structures, represented via xml files in the library/xml/ submodule.</p>
+<p>Data structure layouts are described in files following the df.*.xml name pattern. This information is transformed by a perl script into C++ headers describing the structures, and associated metadata for the Lua wrapper. These headers and data are then compiled into the DFHack libraries, thus necessitating a compatibility break every time layouts change; in return it significantly boosts the efficiency and capabilities of DFHack code.</p>
+<p>Global object addresses are stored in symbols.xml, which is copied to the dfhack release package and loaded as data at runtime.</p>
+</div>
+<div class="section" id="remote-access-interface">
+<h2><a class="toc-backref" href="#id15">Remote access interface</a></h2>
+<p>DFHack supports remote access by exchanging Google protobuf messages via a TCP socket. Both the core and plugins can define remotely accessible methods. The <tt class="docutils literal"><span class="pre">dfhack-run</span></tt> command uses this interface to invoke ordinary console commands.</p>
+<p>Currently the supported set of requests is limited, because the developers don't know what exactly is most useful.</p>
+<p>Protocol client implementations exist for Java and C#.</p>
+</div>
 <div class="section" id="contributing-to-dfhack">
-<h2><a class="toc-backref" href="#id14">Contributing to DFHack</a></h2>
+<h2><a class="toc-backref" href="#id16">Contributing to DFHack</a></h2>
 <p>Several things should be kept in mind when contributing to DFHack.</p>
 <div class="section" id="coding-style">
-<h3><a class="toc-backref" href="#id15">Coding style</a></h3>
+<h3><a class="toc-backref" href="#id17">Coding style</a></h3>
 <p>DFhack uses ANSI formatting and four spaces as indentation. Line
 endings are UNIX. The files use UTF-8 encoding. Code not following this
 won't make me happy, because I'll have to fix it. There's a good chance
 I'll make <em>you</em> fix it ;)</p>
 </div>
 <div class="section" id="how-to-get-new-code-into-dfhack">
-<h3><a class="toc-backref" href="#id16">How to get new code into DFHack</a></h3>
+<h3><a class="toc-backref" href="#id18">How to get new code into DFHack</a></h3>
 <p>You can send patches or make a clone of the github repo and ask me on
 the IRC channel to pull your code in. I'll review it and see if there
 are any problems. I'll fix them if they are minor.</p>
@@ -506,7 +521,7 @@ <h3><a class="toc-backref" href="#id16">How to get new code into DFHack</a></h3>
 fixing.</p>
 </div>
 <div class="section" id="memory-research">
-<h3><a class="toc-backref" href="#id17">Memory research</a></h3>
+<h3><a class="toc-backref" href="#id19">Memory research</a></h3>
 <p>If you want to do memory research, you'll need some tools and some knowledge.
 In general, you'll need a good memory viewer and optionally something
 to look at machine code without getting crazy :)</p>

LUA_API.rst

@@ -4,9 +4,26 @@ DFHack Lua API
 
 .. contents::
 
-====================
-DF structure wrapper
-====================
+The current version of DFHack has extensive support for
+the Lua scripting language, providing access to:
+
+1. Raw data structures used by the game.
+2. Many C++ functions for high-level access to these
+   structures, and interaction with dfhack itself.
+3. Some functions exported by C++ plugins.
+
+Lua code can be used both for writing scripts, which
+are treated by DFHack command line prompt almost as
+native C++ commands, and invoked by plugins written in c++.
+
+This document describes native API available to Lua in detail.
+For the most part it does not describe utility functions
+implemented by Lua files located in hack/lua/...
+
+
+=========================
+DF data structure wrapper
+=========================
 
 DF structures described by the xml files in library/xml are exported
 to lua code as a tree of objects and functions under the ``df`` global,
@@ -426,13 +443,17 @@ not destroy any objects allocated in this way, so the user
 should be prepared to catch the error and do the necessary
 cleanup.
 
-================
-DFHack utilities
-================
+==========
+DFHack API
+==========
 
 DFHack utility functions are placed in the ``dfhack`` global tree.
 
-Currently it defines the following features:
+Native utilities
+================
+
+Input & Output
+--------------
 
 * ``dfhack.print(args...)``
 
@@ -451,6 +472,7 @@ Currently it defines the following features:
 * ``dfhack.color([color])``
 
   Sets the current output color. If color is *nil* or *-1*, resets to default.
+  Returns the previous color value.
 
 * ``dfhack.is_interactive()``
 
@@ -473,15 +495,27 @@ Currently it defines the following features:
 
   If the interactive console is not accessible, returns *nil, error*.
 
+
+Exception handling
+------------------
+
+* ``dfhack.error(msg[,level[,verbose]])``
+
+  Throws a dfhack exception object with location and stack trace.
+  The verbose parameter controls whether the trace is printed by default.
+
+* ``qerror(msg[,level])``
+
+  Calls ``dfhack.error()`` with ``verbose`` being *false*. Intended to
+  be used for user-caused errors in scripts, where stack traces are not
+  desirable.
+
 * ``dfhack.pcall(f[,args...])``
 
   Invokes f via xpcall, using an error function that attaches
   a stack trace to the error. The same function is used by SafeCall
   in C++, and dfhack.safecall.
 
-  The returned error is a table with separate ``message`` and
-  ``stacktrace`` string fields; it implements ``__tostring``.
-
 * ``safecall(f[,args...])``, ``dfhack.safecall(f[,args...])``
 
   Just like pcall, but also prints the error using printerr before
@@ -491,6 +525,34 @@ Currently it defines the following features:
 
   Compares to coroutine.resume like dfhack.safecall vs pcall.
 
+* ``dfhack.exception``
+
+  Metatable of error objects used by dfhack. The objects have the
+  following properties:
+
+  ``err.where``
+    The location prefix string, or *nil*.
+  ``err.message``
+    The base message string.
+  ``err.stacktrace``
+    The stack trace string, or *nil*.
+  ``err.cause``
+    A different exception object, or *nil*.
+  ``err.thread``
+    The coroutine that has thrown the exception.
+  ``err.verbose``
+    Boolean, or *nil*; specifies if where and stacktrace should be printed.
+  ``tostring(err)``, or ``err:tostring([verbose])``
+    Converts the exception to string.
+
+* ``dfhack.exception.verbose``
+
+  The default value of the ``verbose`` argument of ``err:tostring()``.
+
+
+Locking and finalization
+------------------------
+
 * ``dfhack.with_suspend(f[,args...])``
 
   Calls ``f`` with arguments after grabbing the DF core suspend lock.
@@ -529,7 +591,7 @@ Currently it defines the following features:
 
 
 Persistent configuration storage
-================================
+--------------------------------
 
 This api is intended for storing configuration options in the world itself.
 It probably should be restricted to data that is world-dependent.
@@ -571,7 +633,7 @@ functions can just copy values in memory without doing any actual I/O.
 However, currently every entry has a 180+-byte dead-weight overhead.
 
 Material info lookup
-====================
+--------------------
 
 A material info record has fields:
 
@@ -1148,6 +1210,11 @@ Internal API
 These functions are intended for the use by dfhack developers,
 and are only documented here for completeness:
 
+* ``dfhack.internal.scripts``
+
+  The table used by ``dfhack.run_script()`` to give every script its own
+  global environment, persistent between calls to the script.
+
 * ``dfhack.internal.getAddress(name)``
 
   Returns the global address ``name``, or *nil*.
@@ -1156,14 +1223,38 @@ and are only documented here for completeness:
 
   Sets the global address ``name``. Returns the value of ``getAddress`` before the change.
 
-* ``dfhack.internal.getBase()``
+* ``dfhack.internal.getVTable(name)``
+
+  Returns the pre-extracted vtable address ``name``, or *nil*.
 
-  Returns the base address of the process.
+* ``dfhack.internal.getRebaseDelta()``
+
+  Returns the ASLR rebase offset of the DF executable.
 
 * ``dfhack.internal.getMemRanges()``
 
   Returns a sequence of tables describing virtual memory ranges of the process.
 
+* ``dfhack.internal.memmove(dest,src,count)``
+
+  Wraps the standard memmove function. Accepts both numbers and refs as pointers.
+
+* ``dfhack.internal.memcmp(ptr1,ptr2,count)``
+
+  Wraps the standard memcmp function.
+
+* ``dfhack.internal.memscan(haystack,count,step,needle,nsize)``
+
+  Searches for ``needle`` of ``nsize`` bytes in ``haystack``,
+  using ``count`` steps of ``step`` bytes.
+  Returns: *step_idx, sum_idx, found_ptr*, or *nil* if not found.
+
+* ``dfhack.internal.diffscan(old_data, new_data, start_idx, end_idx, eltsize[, oldval, newval, delta])``
+
+  Searches for differences between buffers at ptr1 and ptr2, as integers of size eltsize.
+  The oldval, newval or delta arguments may be used to specify additional constraints.
+  Returns: *found_index*, or *nil* if end reached.
+
 
 Core interpreter context
 ========================
@@ -1231,6 +1322,42 @@ Features:
   Invokes all listeners contained in the event in an arbitrary
   order using ``dfhack.safecall``.
 
+
+=======
+Modules
+=======
+
+DFHack sets up the lua interpreter so that the built-in ``require``
+function can be used to load shared lua code from hack/lua/.
+The ``dfhack`` namespace reference itself may be obtained via
+``require('dfhack')``, although it is initially created as a
+global by C++ bootstrap code.
+
+The following functions are provided:
+
+* ``mkmodule(name)``
+
+  Creates an environment table for the module. Intended to be used as::
+
+    local _ENV = mkmodule('foo')
+    ...
+    return _ENV
+
+  If called the second time, returns the same table; thus providing reload support.
+
+* ``reload(name)``
+
+  Reloads a previously ``require``-d module *"name"* from the file.
+  Intended as a help for module development.
+
+* ``dfhack.BASE_G``
+
+  This variable contains the root global environment table, which is
+  used as a base for all module and script environments. Its contents
+  should be kept limited to the standard Lua library and API described
+  in this document.
+
+
 =======
 Plugins
 =======
@@ -1292,3 +1419,40 @@ sort
 
 Does not export any native functions as of now. Instead, it
 calls lua code to perform the actual ordering of list items.
+
+
+=======
+Scripts
+=======
+
+Any files with the .lua extension placed into hack/scripts/*
+are automatically used by the DFHack core as commands. The
+matching command name consists of the name of the file sans
+the extension.
+
+**NOTE:** Scripts placed in subdirectories still can be accessed, but
+do not clutter the ``ls`` command list; thus it is preferred
+for obscure developer-oriented scripts and scripts used by tools.
+When calling such scripts, always use '/' as the separator for
+directories, e.g. ``devel/lua-example``.
+
+Scripts are re-read from disk every time they are used
+(this may be changed later to check the file change time); however
+the global variable values persist in memory between calls.
+Every script gets its own separate environment for global
+variables.
+
+Arguments are passed in to the scripts via the **...** built-in
+quasi-variable; when the script is called by the DFHack core,
+they are all guaranteed to be non-nil strings.
+
+DFHack core invokes the scripts in the *core context* (see above);
+however it is possible to call them from any lua code (including
+from other scripts) in any context, via the same function the core uses:
+
+* ``dfhack.run_script(name[,args...])``
+
+  Run a lua script in hack/scripts/, as if it was started from dfhack command-line.
+  The ``name`` argument should be the name stem, as would be used on the command line.
+
+Note that this function lets errors propagate to the caller.