Version 0.29.0

.github/PULL_REQUEST_TEMPLATE.md

@@ -5,6 +5,7 @@
 - [ ] If necessary, have you bumped the version number? We will usually do this for you.
 - [ ] Have you included py.test tests with your pull request. (Not yet necessary)
 - [ ] Ensured your code is as close to PEP 8 compliant as possible?
+- [ ] Ensured your pull request is to the next-release branch?
 
 If you haven't completed the above items, please wait to create a PR until you have done so. We will try to review and reply to PRs as quickly as possible.
 

CHANGELOG.md

@@ -1,3 +1,61 @@
+**v0.29.0**
+* [[TeamMsgExtractor #207](https://github.com/TeamMsgExtractor/msg-extractor/issues/207)] Made it so that unspecified dates are handled properly. For clarification, an unspecified date is a custom value in MSG files for dates that means that the date is unspecified. It is distinctly different from a property not existing, which will still return None. For unspecified dates, `datetime.datetime.max` is returned. While perhaps not the best solution, it will have to do for now.
+* Fixed an issue where `utils.parseType` was returning a string for the date when it makes more sense to return an actual datetime instance.
+* [[TeamMsgExtractor #165](https://github.com/TeamMsgExtractor/msg-extractor/issues/165)] [[TeamMsgExtractor #191](https://github.com/TeamMsgExtractor/msg-extractor/issues/191)] Completely redesigned all existing save functions. You can now properly save to custom locations under custom file names. This change may break existing code for several reasons. First, all arguments have been changed to keyword arguments. Second, a few keyword arguments have been renamed to better fit the naming conventions.
+* [[TeamMsgExtractor #200](https://github.com/TeamMsgExtractor/msg-extractor/issues/200)] Changed imports to use relative imports instead of hard imports where applicable.
+* Updated the save functions to no longer rely on the current working directory to save things. The module now does what it can to use hard pathing so that if you spontaneously change working directory it will not cause problems. This should also allow for saving to be threaded, if I am correct.
+* [[TeamMsgExtractor #197](https://github.com/TeamMsgExtractor/msg-extractor/issues/197)] Added new property `Message.defaultFolderName`. This property returns the default name to be used for a Message if none of the options change the name.
+* [[TeamMsgExtractor #201](https://github.com/TeamMsgExtractor/msg-extractor/issues/201)] Fixed an issue where if the class type was all caps it would not be recognized. According to the documentation the comparisons should have been case insensitive, but I must have misread it at some point.
+* [[TeamMsgExtractor #202](https://github.com/TeamMsgExtractor/msg-extractor/issues/202)] Module will now handle path lengths in a semi-intelligent way to determine how best to save the MSG files. Default path length max is 255.
+* [[TeamMsgExtractor #203](https://github.com/TeamMsgExtractor/msg-extractor/issues/203)] Fixed an issue where having multiple "." characters in your file name would cause the directories to be incorrectly named when using the `useFileName` (now `useMsgFilename`) argument in the save function.
+* [[TeamMsgExtractor #204](https://github.com/TeamMsgExtractor/msg-extractor/issues/204)] Fixed an issue where the failsafe name used by attachments wasn't being encoded before hand causing encoding errors.
+* MSG files with a type of simply `IPM` will now be returned as `MSGFile` by `openMsg`, as this specifies that no format has been specified.
+* [[TeamMsgExtractor #214](https://github.com/TeamMsgExtractor/msg-extractor/issues/214)] Attachments that error because the MSG class type wasn't recognized or isn't supported will now correctly be `UnsupportedAttachment` instead of `BrokenAttachment`.
+* Improved internal code in many functions to make them faster and more efficient.
+* `openMsg` will now tell you if a class type is simply unsupported rather than unrecognized. If it is found in the list, the function will raise `UnsupportedMSGTypeError`.
+* Added caching to `MSGFile.listDir`. I found that if you have larger files this single function might be taking up over half of the processing time because of how many times it is used in the module.
+* Fully implemented raw saving.
+* Extended the `Contact` class to have more properties.
+* Added new function `MSGFile._ensureSetTyped` which acts like the other ensure set functions but doesn't require you to know the type. Prefer to use other ensure set function when you know exactly what type it will be.
+* Changed `Message.saveRaw` to `MSGFile.saveRaw`.
+* Changed `MSGFile.saveRaw` to take a path and save the contents to a zip file.
+* Corrected the help doc to reflect the current repository (was still on mattgwwalker).
+* Fixed a bug that would cause an exception on trying to access the RTF body on a file that didn't have one. This is now correctly returning `None`.
+* The `raw` keyword of `Message.save` now actually works.
+* Added property `Attachment.randomFilename` which allows you to get the randomly generated name for attachments that don't have a usable one otherwise.
+* Added function `Attachment.regenerateRandomName` for creating a new random name if necessary.
+* Added function `Attachment.getFilename`. This function is used to get the name an attachment will be saved with given the specified arguments. Arguments are identical to `Attachment.save`.
+* Changed pull requests to reflect new style.
+* Added additional properties for defined MSG file fields.
+* Added zip file support for the `Attachment.save` and `Message.save`. Simply pass a path for the `zip` keyword argument and it will create a new `ZipFile` instance and save all of it's data inside there. Alternatively, you can pass an instance of a class that is either a `ZipFile` or `ZipFile`-like and it will simply use that. When this argument is defined, the `customPath` argument refers to the path inside the zip file.
+* Added the `html` and `rtf` keywords to `Message.save`. These will attempt to save the body in the html or rtf format, respectively. If the program cannot save in those formats, it will raise an exception unless the `allowFallback` keyword argument is `True`.
+* Changed `utils.hasLen` to use `hasattr` instead of the try-except method it was using.
+* Added new option `recipientSeparator` to `MessageBase` allowing you to specify a custom recipient separator (default is ";" to match Microsoft Outlook).
+* Changed the `openMsg` function in `Attachment` to not be strict. This allows you to actually open the MSG file even if we don't recognize the type of embedded MSG that is being used.
+* Attempted to normalize encoding names throughout the module so that a certain encoding will only show up using one name and not multiple.
+* Finally figured out what CRC32 algorithm is used in named properties after directly asking in a Microsoft forum (see the thread [here](https://docs.microsoft.com/en-us/answers/questions/574894/ms-oxmsg-specifies-the-use-of-crc-32-checksums-wit.html)). Fortunately the is already defined in the `compressed-rtf` module so we can take advantage of that.
+* Reworked `MessageBase._genRecipient` to improve it (because what on earth was that code it was using before?). Variables in the function are now more descriptive. Added comments in several places.
+* Many renames to better fit naming convention:
+    * `dev.setup_dev_logger` to `dev.setupDevLogger`.
+    * `MSGFile.fix_path` to `MSGFile.fixPath`.
+    * `MessageBase.save_attachments` to `MessageBase.saveAttachments`.
+    * `*.Exists` to `exists`.
+    * `*.ExistsTypedProperty` to `*.existsTypedProperty`.
+    * `prop.create_prop` to `prop.createProp`.
+    * `Properties.attachment_count` to `Properties.attachmentCount`.
+    * `Properties.next_attachment_id` to `Properties.nextAttachmentId`.
+    * `Properties.next_recipient_id` to `Properties.nextRecipientId`.
+    * `Properties.recipient_count` to `Properties.recipientCount`.
+    * `utils.get_command_args` to `utils.getCommandArgs`.
+    * `utils.get_full_class_name` to `utils.getFullClassName`.
+    * `utils.get_input` to `utils.getInput`.
+    * `utils.has_len` to `utils.hasLen`.
+    * `utils.setup_logging` to `utils.setupLogging`.
+    * `constants.int_to_data_type` to `constants.intToDataType`.
+    * `constants.int_to_intelligence` to `constants.intToIntelligence`.
+    * `constants.int_to_recipient_type` to `constants.intToRecipientType`.
+    * Misc internal function variables.
+
 **v0.28.7**
 * Added hex versions of the `MULTIPLE_X_BYTES` constants.
 * Added `1048` to `constants.MULTIPLE_16_BYTES`

README.rst

@@ -180,7 +180,7 @@ Credits
 
 `Matthew Walker`_ - Original developer and owner
 
-`Destiny Peterson (The Elemental of Destruction)`_ - Principle programmer, manager, and msg file "expert"
+`Destiny Peterson (The Elemental of Destruction)`_ - Co-owner, principle programmer, knows more about msg files than anyone probably should
 
 `JP Bourget`_ - Senior programmer, readability and organization expert, secondary manager
 
@@ -197,8 +197,8 @@ And thank you to everyone who has opened an issue and helped us track down those
 .. |License: GPL v3| image:: https://img.shields.io/badge/License-GPLv3-blue.svg
    :target: LICENSE.txt
 
-.. |PyPI3| image:: https://img.shields.io/badge/pypi-0.28.7-blue.svg
-   :target: https://pypi.org/project/extract-msg/0.28.7/
+.. |PyPI3| image:: https://img.shields.io/badge/pypi-0.29.0-blue.svg
+   :target: https://pypi.org/project/extract-msg/0.29.0/
 
 .. |PyPI1| image:: https://img.shields.io/badge/python-2.7+-brightgreen.svg
    :target: https://www.python.org/downloads/release/python-2715/

extract_msg/__init__.py

@@ -27,20 +27,20 @@
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 __author__ = 'Destiny Peterson & Matthew Walker'
-__date__ = '2021-03-02'
-__version__ = '0.28.7'
+__date__ = '2022-01-13'
+__version__ = '0.29.0'
 
 import logging
 
-from extract_msg import constants
-from extract_msg.appointment import Appointment
-from extract_msg.attachment import Attachment
-from extract_msg.contact import Contact
-from extract_msg.exceptions import UnrecognizedMSGTypeError
-from extract_msg.message import Message
-from extract_msg.message_base import MessageBase
-from extract_msg.msg import MSGFile
-from extract_msg.prop import create_prop
-from extract_msg.properties import Properties
-from extract_msg.recipient import Recipient
-from extract_msg.utils import openMsg, properHex
+from . import constants
+from .appointment import Appointment
+from .attachment import Attachment
+from .contact import Contact
+from .exceptions import UnrecognizedMSGTypeError
+from .message import Message
+from .message_base import MessageBase
+from .msg import MSGFile
+from .prop import createProp
+from .properties import Properties
+from .recipient import Recipient
+from .utils import openMsg, properHex

extract_msg/__main__.py

@@ -10,7 +10,7 @@ def main():
     # Setup logging to stdout, indicate running from cli
     CLI_LOGGING = 'extract_msg_cli'
 
-    args = utils.get_command_args(sys.argv[1:])
+    args = utils.getCommandArgs(sys.argv[1:])
     level = logging.INFO if args.verbose else logging.WARNING
     currentdir = os.getcwdu() # Store this just in case the paths that have been given are relative
     if args.out_path:
@@ -29,17 +29,17 @@ def main():
 
         from extract_msg import validation
 
-        val_results = {x[0]: validation.validate(x[0]) for x in args.msgs}
+        valResults = {x[0]: validation.validate(x[0]) for x in args.msgs}
         filename = 'validation {}.json'.format(int(time.time()))
         print('Validation Results:')
-        pprint.pprint(val_results)
+        pprint.pprint(valResults)
         print('These results have been saved to {}'.format(filename))
         with open(filename, 'w') as fil:
-            fil.write(json.dumps(val_results))
-        utils.get_input('Press enter to exit...')
+            fil.write(json.dumps(valResults))
+        utils.getInput('Press enter to exit...')
     else:
         if not args.dump_stdout:
-            utils.setup_logging(args.config_path, level, args.log, args.file_logging)
+            utils.setupLogging(args.config_path, level, args.log, args.file_logging)
         for x in args.msgs:
             try:
                 with Message(x[0]) as msg:
@@ -48,7 +48,7 @@ def main():
                         print(msg.body)
                     else:
                         os.chdir(out)
-                        msg.save(toJson = args.json, useFileName = args.use_filename, ContentId = args.cid)#, html = args.html, rtf = args.html, args.allowFallback)
+                        msg.save(json = args.json, useMsgFilename = args.use_filename, contentId = args.cid, html = args.html, rtf = args.html, allowFallback = args.allowFallback)
             except Exception as e:
                 print("Error with file '" + x[0] + "': " +
                       traceback.format_exc())

extract_msg/appointment.py

@@ -1,14 +1,15 @@
-from extract_msg import constants
-from extract_msg.attachment import Attachment
-from extract_msg.message_base import MessageBase
+from . import constants
+from .attachment import Attachment
+from .message_base import MessageBase
+
 
 class Appointment(MessageBase):
     """
     Parser for Microsoft Outlook Appointment files.
     """
 
-    def __init__(self, path, prefix = '', attachmentClass = Attachment, filename = None, delayAttachments = False, overrideEncoding = None, attachmentErrorBehavior = constants.ATTACHMENT_ERROR_THROW):
-        MessageBase.__init__(self, path, prefix, attachmentClass, filename, delayAttachments, overrideEncoding, attachmentErrorBehavior)
+    def __init__(self, path, prefix = '', attachmentClass = Attachment, filename = None, delayAttachments = False, overrideEncoding = None, attachmentErrorBehavior = constants.ATTACHMENT_ERROR_THROW, recipientSeparator = ';'):
+        MessageBase.__init__(self, path, prefix, attachmentClass, filename, delayAttachments, overrideEncoding, attachmentErrorBehavior, recipientSeparator)
 
     @property
     def appointmentClassType(self):