Merge pull request #205 from Washi1337/development

4.7.0

AsmResolver.sln

@@ -63,6 +63,10 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "TheAnswer", "test\TestBinar
 EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "AsmResolver.Benchmarks", "test\AsmResolver.Benchmarks\AsmResolver.Benchmarks.csproj", "{CB766958-8D07-4E2D-8A38-C21617B1C45D}"
 EndProject
+Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "tools", "tools", "{0D04B5DF-8D16-488F-8767-E49803D3AF46}"
+EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "AsmResolver.PE.Exports.OrdinalMapper", "tools\AsmResolver.PE.Exports.OrdinalMapper\AsmResolver.PE.Exports.OrdinalMapper.csproj", "{3E4B6896-269F-4129-9746-965338779C31}"
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Any CPU = Debug|Any CPU
@@ -343,6 +347,18 @@ Global
 		{CB766958-8D07-4E2D-8A38-C21617B1C45D}.Release|x64.Build.0 = Release|Any CPU
 		{CB766958-8D07-4E2D-8A38-C21617B1C45D}.Release|x86.ActiveCfg = Release|Any CPU
 		{CB766958-8D07-4E2D-8A38-C21617B1C45D}.Release|x86.Build.0 = Release|Any CPU
+		{3E4B6896-269F-4129-9746-965338779C31}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{3E4B6896-269F-4129-9746-965338779C31}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{3E4B6896-269F-4129-9746-965338779C31}.Debug|x64.ActiveCfg = Debug|Any CPU
+		{3E4B6896-269F-4129-9746-965338779C31}.Debug|x64.Build.0 = Debug|Any CPU
+		{3E4B6896-269F-4129-9746-965338779C31}.Debug|x86.ActiveCfg = Debug|Any CPU
+		{3E4B6896-269F-4129-9746-965338779C31}.Debug|x86.Build.0 = Debug|Any CPU
+		{3E4B6896-269F-4129-9746-965338779C31}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{3E4B6896-269F-4129-9746-965338779C31}.Release|Any CPU.Build.0 = Release|Any CPU
+		{3E4B6896-269F-4129-9746-965338779C31}.Release|x64.ActiveCfg = Release|Any CPU
+		{3E4B6896-269F-4129-9746-965338779C31}.Release|x64.Build.0 = Release|Any CPU
+		{3E4B6896-269F-4129-9746-965338779C31}.Release|x86.ActiveCfg = Release|Any CPU
+		{3E4B6896-269F-4129-9746-965338779C31}.Release|x86.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
@@ -375,6 +391,7 @@ Global
 		{A2D97B54-884C-46CA-BB42-9F87D5AA0A87} = {786C1732-8C96-45DD-97BB-639C9AA7F45B}
 		{BC3C945E-0AB5-4039-B6D9-43A265CC703A} = {B3AF102B-ABE1-41B2-AE48-C40702F45AB0}
 		{CB766958-8D07-4E2D-8A38-C21617B1C45D} = {786C1732-8C96-45DD-97BB-639C9AA7F45B}
+		{3E4B6896-269F-4129-9746-965338779C31} = {0D04B5DF-8D16-488F-8767-E49803D3AF46}
 	EndGlobalSection
 	GlobalSection(ExtensibilityGlobals) = postSolution
 		SolutionGuid = {3302AC79-6D23-4E7D-8C5F-C0C7261044D0}

Directory.Build.props

@@ -7,7 +7,7 @@
         <RepositoryUrl>https://github.com/Washi1337/AsmResolver</RepositoryUrl>
         <RepositoryType>git</RepositoryType>
         <LangVersion>9</LangVersion>
-        <Version>4.6.0</Version>
+        <Version>4.7.0</Version>
     </PropertyGroup>
 
 </Project>

appveyor.yml

@@ -4,7 +4,7 @@
       - master
 
   image: Visual Studio 2019
-  version: 4.6.0-master-build.{build}
+  version: 4.7.0-master-build.{build}
   configuration: Release
 
   skip_commits:
@@ -33,7 +33,7 @@
       - development
 
   image: Visual Studio 2019
-  version: 4.6.0-dev-build.{build}
+  version: 4.7.0-dev-build.{build}
   configuration: Release
 
   skip_commits:

docs/dotnet/advanced-pe-image-building.rst

@@ -99,6 +99,27 @@ If everything is supposed to be preserved as much as possible, then instead of s
     Preserving RIDs within metadata tables might require AsmResolver to make use of the Edit-And-Continue metadata tables (such as the pointer tables). The resulting tables stream could therefore be renamed from ``#~`` to ``#-``, and the file size might increase.
 
 
+String folding in #Strings stream
+---------------------------------
+
+Named metadata members (such as types, methods and fields) are assigned a name by referencing a string in the ``#Strings`` stream by its starting offset. When a metadata member has a name that is a suffix of another member's name, then it is possible to only store the longer name in the ``#Strings`` stream, and let the member with the shorter name use an offset within the middle of this longer name. For example, consider two members with the names ``ABCDEFG`` and ``DEFG``. If ``ABCDEFG`` is stored at offset ``1``, then the name ``DEFG`` is implicitly defined at offset ``1 + 3 = 4``, and can thus be referenced without appending ``DEFG`` to the stream a second time.
+
+By default, the PE image builder will fold strings in the ``#Strings`` stream as described in the above. However, for some input binaries, this might make the building process take a significant amount of time. Therefore, to disable this folding of strings, specify the ``NoStringsStreamOptimization`` flag in your ``DotNetDirectoryFactory``:
+
+.. code-block:: csharp
+
+    factory.MetadataBuilderFlags |= MetadataBuilderFlags.NoStringsStreamOptimization;
+
+
+.. warning::
+    Some obfuscated binaries might include lots of members that have very long but similar names. For these types of binaries, disabling this optimization can result in a significantly larger output file size.
+
+
+.. note::
+
+    When ``PreserveStringIndices`` is set and string folding is enabled (``NoStringsStreamOptimization`` is unset), the PE image builder will not fold strings from the original ``#Strings`` stream into each other. However, it will still try to reuse these original strings as much as possible.
+
+
 Preserving maximum stack depth
 ------------------------------
 

docs/dotnet/basics.rst

@@ -26,7 +26,7 @@ The above will create a module that references mscorlib.dll 4.0.0.0 (.NET Framew
 Opening a .NET module
 ---------------------
 
-Opening a .NET module can be done through one of the `FromXXX` methods from the ``ModuleDefinition`` class:
+Opening a .NET module can be done through one of the ``FromXXX`` methods from the ``ModuleDefinition`` class:
 
 .. code-block:: csharp
 
@@ -73,7 +73,7 @@ On Windows, if a module is loaded and mapped in memory (e.g. as a dependency def
 Writing a .NET module
 ---------------------
 
-Writing a .NET module can be done through one of the `Write` method overloads.
+Writing a .NET module can be done through one of the ``Write`` method overloads.
 
 .. code-block:: csharp
 
@@ -127,7 +127,7 @@ Opening (multi-module) .NET assemblies can be done in a very similar fashion as
     var assembly = AssemblyDefinition.FromImage(peImage);
 
     
-If you want to read large files (+100MB), consider using memory mapped I/O instead:
+Similar to reading module definitions, if you want to read large files (+100MB), consider using memory mapped I/O instead:
 
 .. code-block:: csharp
 
@@ -138,7 +138,7 @@ If you want to read large files (+100MB), consider using memory mapped I/O inste
 Writing a .NET assembly
 -----------------------
 
-Writing a .NET assembly can be done through one of the `Write` method overloads.
+Writing a .NET assembly can be done through one of the ``Write`` method overloads.
 
 .. code-block:: csharp
 

docs/dotnet/importing.rst

@@ -1,9 +1,11 @@
+.. _dotnet-reference-importing:
+
 Reference Importing
 ===================
 
 .NET modules use entries in the TypeRef or MemberRef tables to reference types or members from external assemblies. Importing references into the current module therefore form a key role when creating new- or modifying existing .NET modules. When a member is not imported into the current module, a ``MemberNotImportedException`` will be thrown when you are trying to create a PE image or write the module to the disk.
 
-AsmResolver provides the ``ReferenceImporter`` class that does most of the heavy lifting for you.
+AsmResolver provides the ``ReferenceImporter`` class that does most of the heavy lifting.
 
 All samples in this document assume there is an instance of ``ReferenceImporter`` created using the following code:
 
@@ -92,8 +94,9 @@ Types and members can also be imported by passing on an instance of various ``Sy
 There is limited support for importing compound types. Types that can be imported through reflection include:
 
 - Pointer types.
-- By reference types.
-- Array types: If an array contains only one dimension, a ``SzArrayTypeSignature`` is returned. Otherwise a ``ArrayTypeSignature`` is created.
+- By-reference types.
+- Array types:
+    - If an array contains only one dimension, a ``SzArrayTypeSignature`` is returned. Otherwise a ``ArrayTypeSignature`` is created.
 - Generic parameters.
 - Generic type instantiations.
 
@@ -105,11 +108,13 @@ Instantiations of generic methods are supported.
 Common Caveats using the Importer 
 ---------------------------------
 
-**Caching and reuse of instances:**
+Caching and reuse of instances
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The default implementation of ``ReferenceImporter`` does not maintain a cache. Each call to any of the import methods will result in a new instance of the imported member.  The exception to this rule is when the member passed onto the importer is defined in the module the importer is targeting itself, or was already a reference imported by an importer into the target module. In both of these cases, the same instance of this member definition or reference will be returned instead.
 
-**Importing cross-framework versions**
+Importing cross-framework versions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The ``ReferenceImporter`` does not support importing across different versions of the target framework. Members are being imported as-is, and are not automatically adjusted to conform with other versions of a library. 
 

docs/dotnet/managed-method-bodies.rst

@@ -40,6 +40,7 @@ Instructions that are assembled into the method body are automatically disassemb
     var instructions = body.Instructions;
 
 The ``CilInstruction`` class defines three basic properties:
+
 - ``Offset``: The offset of the instruction, relative to the start of the code stream.
 - ``OpCode``: The operation the instruction performs.
 - ``Operand``: The operand of the instruction.
@@ -196,6 +197,9 @@ Below an example on how to use the ``ReferenceImporter`` to emit a call to ``Con
     body.Instructions.Add(new CilInstruction(CilOpCodes.Call, writeLine));
 
 
+More information on the capabilities and limitations of the ``ReferenceImporter`` can be found in :ref:`dotnet-reference-importing`.
+
+
 Expanding and optimising macros
 -------------------------------
 

docs/dotnet/member-tree.rst

@@ -4,7 +4,7 @@ The Member Tree
 Assemblies and modules
 ----------------------
 
-The root of every .NET assembly is represented by the ``AssemblyDefinition`` class. This class exposes basic information such as name, version and public key token, but also a collection of all modules that are defined in the assembly. Modules are represented by the `ModuleDefinition` class.
+The root of every .NET assembly is represented by the ``AssemblyDefinition`` class. This class exposes basic information such as name, version and public key token, but also a collection of all modules that are defined in the assembly. Modules are represented by the ``ModuleDefinition`` class.
 
 Below an example that enumerates all modules defined in an assembly.
 
@@ -14,7 +14,7 @@ Below an example that enumerates all modules defined in an assembly.
     foreach (var module in assembly.Modules)
         Console.WriteLine(module.Name);
 
-Most .NET assemblies only have one module. This main module is also known as the manifest module, and can be accessed directly through the `AssemblyDefinition.ManifestModule` property.
+Most .NET assemblies only have one module. This main module is also known as the manifest module, and can be accessed directly through the ``AssemblyDefinition.ManifestModule`` property.
 
 
 Obtaining types in a module

docs/dotnet/unmanaged-method-bodies.rst

@@ -10,19 +10,22 @@ AsmResolver supports creating new method bodies that are implemented this way. T
     using AsmResolver.DotNet.Code.Native;
 
 
+Prerequisites
+-------------
+
+Before you can start adding native method bodies to a .NET module, a few prerequisites have to be met. Failing to do so will make the CLR not run your mixed mode application, and might throw runtime or image format exceptions, even if everything else conforms to the right format. This section will go over these requirements briefly.
+
 Allowing native code in modules
--------------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Before you can start adding native method bodies to a .NET module, it is required to change a couple of flags in the headers of the executable. In particular, to make the CLR run a mixed mode application, the ``ILOnly`` flag needs to be unset:
+To make the CLR treat the output file as a mixed mode application, the ``ILOnly`` flag needs to be unset:
 
 .. code-block:: csharp 
 
     ModuleDefinition module = ...
     module.Attributes &= ~DotNetDirectoryFlags.ILOnly;
 
-Failing to do so will make the CLR not run your mixed mode application, and might throw runtime or image format exceptions, even if everything else conforms to the right format.
-
-Furthermore, make sure the right architecture is specified. For example, for an x86 64-bit binary, use the folloing:
+Furthermore, make sure the right architecture is specified. For example, for an x86 64-bit binary, use the following:
 
 .. code-block:: csharp
 
@@ -39,6 +42,19 @@ For 32-bit x86 binaries, use the following:
     module.Attributes |= DotNetDirectoryFlags.Bit32Required;
 
 
+Flags for native methods
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+As per ECMA-335 specification, a method definition can only represent a native function via Platform Invoke (P/Invoke). While P/Invoke is usually used for importing functions from external libraries (such as `kernel32.dll`), it is also needed for implementing native methods that are defined within the current .NET module itself. Therefore, to be able to assign a valid native body to a method, the right flags need to be set in both the ``Attributes`` and ``ImplAttributes`` property of a ``MethodDefinition``:
+
+.. code-block:: csharp
+
+    MethodDefinition method = ...
+
+    method.Attributes |= MethodAttributes.PInvokeImpl;
+    method.ImpleAttributes |= MethodImplAttributes.Native | MethodImplAttributes.Unmanaged | MethodImplAttributes.PreserveSig;
+
+
 The NativeMethodBody class
 --------------------------
 
@@ -50,7 +66,7 @@ Each ``NativeMethodBody`` is assigned to exactly one ``MethodDefinition``. Upon
 
     MethodDefinition method = ...
 
-    NativeMethodBody body = new NativeMethodBody(method);
+    var body = new NativeMethodBody(method);
     method.NativeMethodBody = body;
 
 

docs/faq.rst

@@ -7,7 +7,7 @@ Why are there so many libraries / packages instead of just one?
 
 Not everyone will need everything from the code base of AsmResolver. For example, someone that is only interested in reading the PE headers of an input file does not require any of the functionality of the ``AsmResolver.DotNet`` package.
 
-This is why AsmResolver´s design philosophy is not to be one monolithic library, but rather be a toolsuite of libraries. By splitting up in multiple smaller libraries, the user can carefully select the packages that they need and leave out what they do not need. This can easily shave off 100s of kilobytes from the total size of code that is shipped.
+This is why AsmResolver's design philosophy is not to be one monolithic library, but rather be a toolsuite of libraries. By splitting up in multiple smaller libraries, the user can carefully select the packages that they need and leave out what they do not need. This can easily shave off 100s of kilobytes from the total size of code that is shipped.
 
 
 Why does AsmResolver throw so many errors on reading/writing?
@@ -17,7 +17,7 @@ AsmResolver does verification of the input file, and if it finds anything that i
 
 AsmResolver often can ignore and recover from kinds of errors, but this is not enabled by default. To enable these, please refer to :ref:`pe-custom-error-handling` (for .NET images: :ref:`dotnet-advanced-module-reading` and :ref:`dotnet-image-builder-diagnostics`). Be careful with ignoring errors though. Especially for disabling writer verification can cause the output to not work anymore unless you know what you are doing.
 
-If it still breaks and believe it is a bug, report it on the `issues board <https://github.com/Washi1337/AsmResolver/issues>`_.
+If it still breaks and you believe it is a bug, please report it on the `issues board <https://github.com/Washi1337/AsmResolver/issues>`_.
 
 
 Why does the executable not work anymore after modifying it with AsmResolver?
@@ -28,12 +28,13 @@ A couple of things can be happening here:
 - AsmResolver´s PE builder has a bug.
 - You are changing something in the executable you are not supposed to change.
 - You are changing something that results in the executable not function anymore.
+- The target binary is actively trying to prevent you from applying any modifications (this happens a lot with obfuscated binaries).
 
 With great power comes great responsibility. Changing the wrong things in the input executable file can result in the output stop working.
 
-For .NET applications, make sure your application conforms with specification (`ECMA-335 <https://www.ecma-international.org/publications/files/ECMA-ST/ECMA-335.pdf>`_). To help you find problems, try reopening the executable in AsmResolver and look for errors reported by the reader. Alternatively, try verifying the output file using ``peverify`` for .NET Framework applications (usually located in ``C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX X.X Tools.``) or ``dotnet ILVerify`` for .NET Core / .NET 5+ applications, and look for any irregularities reported by these tool.
+For .NET applications, make sure your application conforms with specification (`ECMA-335 <https://www.ecma-international.org/publications/files/ECMA-ST/ECMA-335.pdf>`_). To help you with finding problems in your final output, try reopening the executable in AsmResolver and look for errors reported by the reader. Alternatively, using tools such as ``peverify`` for .NET Framework applications (usually located in ``C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX X.X Tools.``) or ``dotnet ILVerify`` for .NET Core / .NET 5+ applications, can also help narrowing down what might have gone wrong.
 
-If you believe it is a bug in AsmResolver, report it on the `issues board <https://github.com/Washi1337/AsmResolver/issues>`_.
+If you believe it is a bug in AsmResolver, please report it on the `issues board <https://github.com/Washi1337/AsmResolver/issues>`_.
 
 
 In Mono.Cecil / dnlib my code works, but it does not work in AsmResolver, why? 
@@ -44,9 +45,9 @@ Essentially, two things can be happening here:
 - AsmResolver´s code could have a bug.
 - You are misusing AsmResolver´s API.
 
-Remember, while perhaps the public API of ``AsmResolver.DotNet`` might look similar on face value to other libraries (such as Mono.Cecil or dnlib), AsmResolver itself follows a very different design philosophy than these libraries. As such, a lot of classes in those libraries will not map one-to-one to AsmResolver classes directly. Check the documentation to make sure you are not misusing the API.
+It is important to remember that, while the public API of ``AsmResolver.DotNet`` looks similar on face value to other libraries (such as Mono.Cecil or dnlib), AsmResolver itself follows a very different design philosophy than these libraries. As such, a lot of classes in those libraries will not map one-to-one to AsmResolver classes directly. Check the documentation to make sure you are not misusing the API.
 
-If you believe it is a bug, report it on the `issues board <https://github.com/Washi1337/AsmResolver/issues>`_.
+If you believe it is a bug, please report it on the `issues board <https://github.com/Washi1337/AsmResolver/issues>`_.
 
 
 Does AsmResolver have a concept similar to writer events in dnlib? 

docs/peimage/basics.rst

@@ -65,7 +65,8 @@ Building an image back to a PE file can be done by using one of the classes that
 
 .. note::
 
-    Currently AsmResolver only provides a builder for .NET images.
+    Currently AsmResolver only provides a full fletched builder for .NET images.
+
 
 Building a .NET image can be done through the ``AsmResolver.PE.DotNet.Builder.ManagedPEFileBuilder`` class: