Port Test Scripts to Python 3 (#154)

* Migrate test scripts to Python 3

* Add copyright notice to modified files

* Update README and developement documentation

* Documentation and requirements.txt updates

* Update FakeNet-NG version

* Update setup.py

CHANGELOG.txt

@@ -1,3 +1,10 @@
+Version 1.4.13
+--------------
+* Port test scripts to Python 3
+* Replace pyping package with icmplib as the former is not compatible with Python 3
+* Add additional logging to test.py to output the number of tests passed/failed
+* Add copyright notice to README.md, splash screen, and test.py
+
 Version 1.4.12
 --------------
 * Increase default timeout for HTTPListener to 10 seconds

README.md

@@ -36,15 +36,18 @@ application's specific functionality and prototyping PoCs.
 Installation
 ============
 
-You can install FakeNet-NG in a few different ways.
+You can install FakeNet-NG in a few different ways. Note that the following
+installation processes will retrieve third-party open-source libraries used
+by FakeNet-NG to your system. These libraries will be dynamically loaded at
+runtime, and some of these libraries may be LGPL licensed.
 
 Stand-alone executable
 ----------------------
 
 It is easiest to simply download the compiled version which can be obtained from
 the releases page:
 
-    https://github.com/fireeye/flare-fakenet-ng/releases
+    https://github.com/mandiant/flare-fakenet-ng/releases
 
 Execute FakeNet-NG by running 'fakenet.exe'.
 
@@ -66,13 +69,17 @@ Installation on Linux requires the following dependencies:
  * libnetfilterqueue development files (e.g. libnetfilter-queue-dev for
    Ubuntu).
 
+Install these dependencies using the following command:
+
+    sudo apt-get install build-essential python-dev libnetfilter-queue-dev
+
 Install FakeNet-NG as a Python module using pip:
 
-    pip install https://github.com/fireeye/flare-fakenet-ng/zipball/master
+    pip install https://github.com/mandiant/flare-fakenet-ng/zipball/master
 
 Or by obtaining the latest source code and installing it manually:
 
-    git clone https://github.com/fireeye/flare-fakenet-ng/
+    git clone https://github.com/mandiant/flare-fakenet-ng/
 
 Change directory to the downloaded flare-fakenet-ng and run:
 
@@ -104,7 +111,7 @@ install dependencies as follows:
 
 3) Download the FakeNet-NG source code:
 
-    git clone https://github.com/fireeye/flare-fakenet-ng
+    git clone https://github.com/mandiant/flare-fakenet-ng
 
 Execute FakeNet-NG by running it with a Python interpreter in a privileged
 shell:
@@ -129,6 +136,7 @@ parameter to get simple help:
                              Version  1.0
       _____________________________________________________________
                        Developed by FLARE Team
+        Copyright (C) 2016-2022 Mandiant, Inc. All rights reserved.
       _____________________________________________________________
     Usage: fakenet.py [options]:
 
@@ -183,6 +191,7 @@ and an HTTP connection:
                              Version  1.0
       _____________________________________________________________
                        Developed by FLARE Team
+        Copyright (C) 2016-2022 Mandiant, Inc. All rights reserved.
       _____________________________________________________________
 
     07/06/16 10:20:52 PM [           FakeNet] Loaded configuration file: configs/default.ini
@@ -847,5 +856,5 @@ Contact
 =======
 
 For bugs, crashes, or other comments please contact
-FakeNet@fireeye.com.
+FakeNet@mandiant.com.
 

docs/developing.md

@@ -122,9 +122,19 @@ version numbers are to be incorporated in artifacts such as directory names.
 Branching, Pull Requests, and Merging
 -------------------------------------
 
-FireEye only:
-* Create branches directly under the `fireeye/` GitHub repository, not under a
+Mandiant only:
+* Create branches directly under the `mandiant/` GitHub repository, not under a
   private fork.
+* Create a new local branch:
+```
+git clone https://github.com/mandiant/flare-fakenet-ng.git
+git checkout -b <your_branch_name>
+```
+* Make necessary changes, commit and push your local branch to remote:
+```
+git push origin <your_branch_name>
+```
+* Create a pull request
 
 All contributors:
 * Pull request comment should bear bulleted list for inclusion in change log.
@@ -256,17 +266,17 @@ fakenet1.4.3\
     		+-- ssl_detector.py
 ```
 
-FireEye only:
+Mandiant only:
 * Create a zip file of the release distribution named `fakenet<ver>.zip`, where
   `<ver>` is the dot-decimal version number. For example: `fakenet1.4.3.zip`.
 * The top-level directory in the zip file should be `fakenet<ver>` e.g.
   `fakenet1.4.3`.
 * Test the distribution before adding it to the appropriate release tag.
 
-Tagging a Release (FireEye only)
+Tagging a Release (Mandiant only)
 --------------------------------
 
-1. Visit <https://github.com/fireeye/flare-fakenet-ng/releases>
+1. Visit <https://github.com/mandiant/flare-fakenet-ng/releases>
 2. Click `Draft a new release`
 3. Tag version: `v<ver>`, e.g. `v1.4.3`
 4. Release title: `FakeNet-NG <ver>`, e.g. `FakeNet-NG 1.4.3`

docs/testing.md

@@ -78,9 +78,14 @@ the FakeNet machine during testing. For details, see the section titled "Using
 ### Test Script Dependencies
 The test script requires dependencies over and above what FakeNet calls for. As
 of this writing, they include:
-* `pyping`
 * `irc`
 * `requests`
+* `icmplib`
+
+Install these dependencies using `requirements.txt` in the `test` folder:
+```
+pip install -r requirements.txt
+```
 
 ### Test Script Usage
 FakeNet supports Internet simulation for the local machine (i.e. in

fakenet/fakenet.py

@@ -1,3 +1,5 @@
+# Copyright (C) 2016-2022 Mandiant, Inc. All rights reserved.
+
 #!/usr/bin/env python
 #
 # FakeNet-NG is a next generation dynamic network analysis tool for malware
@@ -331,9 +333,10 @@ def main():
  | | / ____ \| . \| |____| |\  | |____   | |      | |\  | |__| |
  |_|/_/    \_\_|\_\______|_| \_|______|  |_|      |_| \_|\_____|
 
-                        Version 1.4.12
+                        Version 1.4.13
   _____________________________________________________________
                    Developed by FLARE Team
+    Copyright (C) 2016-2022 Mandiant, Inc. All rights reserved.
   _____________________________________________________________
                                                """
 

setup.py

@@ -23,12 +23,12 @@
 
 setup(
     name='FakeNet NG',
-    version='1.4.12',
+    version='1.4.13',
     description="",
     long_description="",
-    author="FireEye FLARE Team with credit to Peter Kacherginsky as the original developer",
-    author_email='FakeNet@fireeye.com',
-    url='https://www.github.com/fireeye/flare-fakenet-ng',
+    author="Mandiant FLARE Team with credit to Peter Kacherginsky as the original developer",
+    author_email='FakeNet@mandiant.com',
+    url='https://www.github.com/mandiant/flare-fakenet-ng',
     packages=[
         'fakenet',
     ],

test/requirements.txt

@@ -0,0 +1,4 @@
+netifaces
+icmplib
+irc
+requests

test/test.py

@@ -1,3 +1,5 @@
+# Copyright (C) 2016-2022 Mandiant, Inc. All rights reserved.
+
 import os
 import re
 import sys
@@ -6,7 +8,6 @@
 import ctypes
 import signal
 import socket
-import pyping
 import ftplib
 import poplib
 import shutil
@@ -20,8 +21,9 @@
 import netifaces
 import subprocess
 import irc.client
-import ConfigParser
+import configparser
 from collections import OrderedDict
+from icmplib import ping
 
 logger = logging.getLogger('FakeNetTests')
 logging.basicConfig(format='%(message)s', level=logging.INFO)
@@ -53,7 +55,7 @@ def ign_sigint():
                                shell=shl,
                                close_fds = cfds,
                                preexec_fn = preexec).pid
-    except Exception, e:
+    except Exception as e:
         logger.info('Error: Failed to execute command: %s', execute_cmd)
         logger.info('       %s', e)
         return None
@@ -357,7 +359,7 @@ def _filterMatchingTests(self, tests, matchspec):
             # Iterating over tests first, match specifications second to
             # preserve the order of the selected tests. Less efficient to
             # compile every regex several times, but less confusing.
-            for testname, test in tests.items():
+            for testname, test in list(tests.items()):
 
                 # First determine if it is to be excluded, in which case,
                 # remove it and do not evaluate further match specifications.
@@ -431,7 +433,7 @@ def _testGeneric(self, label, config, tests, matchspec=[]):
             while True:
                 logger.info('Type \'ok\' to continue, or \'exit\' to stop')
                 try:
-                    ok = raw_input()
+                    ok = input()
                 except EOFError:
                     ok = 'exit'
 
@@ -444,8 +446,11 @@ def _testGeneric(self, label, config, tests, matchspec=[]):
         logger.info('Testing')
         logger.info('-' * 79)
 
+        passedTests = 0
+        failedTests = 0
+
         # Do each test
-        for desc, (callback, args, expected) in tests.iteritems():
+        for desc, (callback, args, expected) in tests.items():
             logger.debug('Testing: %s' % (desc))
             passed = self._tryTest(desc, callback, args, expected)
 
@@ -454,12 +459,18 @@ def _testGeneric(self, label, config, tests, matchspec=[]):
                 logger.debug('Retrying: %s' % (desc))
                 passed = self._tryTest(desc, callback, args, expected)
 
+            if passed:
+                passedTests += 1
+            else:
+                failedTests += 1
+
             self._printStatus(desc, passed)
 
             time.sleep(0.5)
 
         logger.info('-' * 79)
         logger.info('Tests complete')
+        logger.info('%s/%s tests passed,  %s/%s tests failed' % (passedTests, len(tests), failedTests, len(tests)))
         logger.info('-' * 79)
 
         if self.settings.singlehost:
@@ -490,7 +501,7 @@ def _test_sk(self, proto, host, port, teststring=None, expected=None,
             s.connect((host, port))
 
             if teststring is None:
-                teststring = 'Testing FakeNet-NG'
+                teststring = b'Testing FakeNet-NG'
 
             if expected is None:
                 # RawListener is an echo server unless otherwise configured
@@ -519,8 +530,8 @@ def _test_sk(self, proto, host, port, teststring=None, expected=None,
         return retval
 
     def _test_icmp(self, host):
-        r = pyping.ping(host, count=1)
-        return (r.ret_code == 0)
+        r = ping(host, count=1)
+        return r.is_alive
 
     def _test_ns(self, hostname, expected):
        return (expected == socket.gethostbyname(hostname))
@@ -547,12 +558,12 @@ def _test_pop(self, hostname, port=None, timeout=5):
         lines = msg[1]
         octets = msg[2]
 
-        if not response.startswith('+OK'):
+        if not response.startswith(b'+OK'):
             msg = 'POP3 response does not start with "+OK"'
             logger.error(msg)
             return False
 
-        if not 'Alice' in ''.join(lines):
+        if not b'Alice' in b''.join(lines):
             msg = 'POP3 message did not contain expected string'
             raise FakeNetTestException(msg)
             return False
@@ -604,7 +615,7 @@ def _test_http(self, hostname, port=None, scheme=None, uri=None,
 
         return retval
 
-    def _test_ftp(self, hostname, port=None):
+    def _test_ftp(self, hostname, port=0):
         """Note that the FakeNet-NG Proxy listener won't know what to do with
         this client if you point it at some random port, because the client
         listens silently for the server 220 welcome message which doesn't give
@@ -722,11 +733,11 @@ def testGeneral(self, matchspec=[]):
             t['TCP localhost @ bound'] = (self._test_sk, (tcp, localhost, 1337), True)
             t['TCP localhost @ unbound'] = (self._test_sk, (tcp, localhost, 9999), False)
 
-        t['TCP custom test static Base64'] = (self._test_sk, (tcp, ext_ip, 1000, 'whatever', '\x0fL\x0aR\x0e'), True)
-        t['TCP custom test static string'] = (self._test_sk, (tcp, ext_ip, 1001, 'whatever', 'static string TCP response'), True)
-        t['TCP custom test static file'] = (self._test_sk, (tcp, ext_ip, 1002, 'whatever', 'sample TCP raw file response'), True)
-        whatever = 'whatever'  # Ensures matching test/expected for TCP dynamic
-        t['TCP custom test dynamic'] = (self._test_sk, (tcp, ext_ip, 1003, whatever, ''.join([chr(ord(c)+1) for c in whatever])), True)
+        t['TCP custom test static Base64'] = (self._test_sk, (tcp, ext_ip, 1000, b'whatever', b'\x0fL\x0aR\x0e'), True)
+        t['TCP custom test static string'] = (self._test_sk, (tcp, ext_ip, 1001, b'whatever', b'static string TCP response'), True)
+        t['TCP custom test static file'] = (self._test_sk, (tcp, ext_ip, 1002, b'whatever', b'sample TCP raw file response'), True)
+        whatever = b'whatever'  # Ensures matching test/expected for TCP dynamic
+        t['TCP custom test dynamic'] = (self._test_sk, (tcp, ext_ip, 1003, whatever, b''.join([chr(c+1).encode("utf-8") for c in whatever])), True)
 
         t['UDP external IP @ bound'] = (self._test_sk, (udp, ext_ip, 1337), True)
         t['UDP external IP @ unbound'] = (self._test_sk, (udp, ext_ip, 9999), True)
@@ -738,9 +749,9 @@ def testGeneral(self, matchspec=[]):
             t['UDP localhost @ bound'] = (self._test_sk, (udp, localhost, 1337), True)
             t['UDP localhost @ unbound'] = (self._test_sk, (udp, localhost, 9999), False)
 
-        t['UDP custom test static Base64'] = (self._test_sk, (udp, ext_ip, 1000, 'whatever', '\x0fL\x0aR\x0e'), True)
-        whatever = 'whatever2'  # Ensures matching test/expected for UDP dynamic
-        t['UDP custom test dynamic'] = (self._test_sk, (udp, ext_ip, 1003, whatever, ''.join([chr(ord(c)+1) for c in whatever])), True)
+        t['UDP custom test static Base64'] = (self._test_sk, (udp, ext_ip, 1000, b'whatever', b'\x0fL\x0aR\x0e'), True)
+        whatever = b'whatever2'  # Ensures matching test/expected for UDP dynamic
+        t['UDP custom test dynamic'] = (self._test_sk, (udp, ext_ip, 1003, whatever, b''.join([chr(c+1).encode("utf-8") for c in whatever])), True)
 
         t['ICMP external IP'] = (self._test_icmp, (ext_ip,), True)
         t['ICMP arbitrary host'] = (self._test_icmp, (arbitrary,), True)
@@ -804,7 +815,7 @@ class FakeNetConfig:
     """Convenience class to read/modify/rewrite a configuration template."""
 
     def __init__(self, path, singlehostmode=True, proxied=True, redirectall=True):
-        self.rawconfig = ConfigParser.RawConfigParser()
+        self.rawconfig = configparser.RawConfigParser()
         self.rawconfig.read(path)
 
         if singlehostmode: