Unable to walk NIF, string too long

[psi29a]

@niftools/pyffi-reviewer

Issue Overview

 Error with ../../../Private/UIX/UIX FILES/Data Files/Meshes/TOE/RedMoonGa01.NIF: ('string too long (0xB43A827B at 0x0000BFE8)',)
 Error with ../../../Private/UIX/UIX FILES/Data Files/Meshes/TOEspy/BlueMoonGa01.NIF: ('string too long (0xB43A827B at 0x0000BFE9)',)
 Error with ../../../Private/UIX/UIX FILES/Data Files/Meshes/TOEspy/Card_Explosion.NIF: ('string too long (0xBF800000 at 0x0000B3DD)',)
 Error with ../../../Private/UIX/UIX FILES/Data Files/Meshes/TOEspy/XCard_Explosion.nif: ('string too long (0xBF800000 at 0x0000B2EE)',)

There are at least 4 files that can't be walked due to a string being too long. I'm not sure if this is a problem the NIF or with pyffi. The only thing in common is that they were created by the Bethesda plugin in 3DSMax.

Version Information

Pyffi Version Info

2.2.2

Platform information

Ubuntu (16.10) Linux

Context

I was running nifwalk over all NIFs I planned on releasing to make sure I had all the resources.

Steps to Reproduce

from pyffi.formats.nif import NifFormat
NifFormat.walkData(nif_path)

raise ValueError

Expected Result

an NIF object that I can iterate over

Actual Result

Error message.

[psi29a]

Here is the NIF to test, is licensed under CC0 so we can do with as we please.
https://dl.dropboxusercontent.com/u/396161/RedMoonGa01.NIF

[psi29a]

I put a print statement on the read function
In the class SizedString(BasicBase, EditableLineEdit), in the read function I printed the size of the string:

(venv) bcurtis@WhiteQueen:~/Workspace/Private/UIX-R$ python error.py
../../../Private/UIX/UIX FILES/Data Files/Meshes/TOE/RedMoonGa01.NIF, DEBUG 6
DEBUG 6
DEBUG 17
DEBUG 10
DEBUG 15
DEBUG 18
DEBUG 14
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 10
DEBUG 15
DEBUG 18
DEBUG 14
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 10
DEBUG 15
DEBUG 18
DEBUG 14
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 10
DEBUG 19
DEBUG 15
DEBUG 15
DEBUG 18
DEBUG 14
DEBUG 10
DEBUG 15
DEBUG 18
DEBUG 14
DEBUG 10
DEBUG 15
DEBUG 18
DEBUG 14
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 6
DEBUG 17
DEBUG 10
DEBUG 15
DEBUG 14
DEBUG 16
DEBUG 17
DEBUG 17
DEBUG 19
DEBUG 26
DEBUG 18
DEBUG 18
DEBUG 19
DEBUG 16
DEBUG 16
DEBUG 15
DEBUG 15
DEBUG 15
DEBUG 15
DEBUG 15
DEBUG 15
DEBUG 15
DEBUG 15
DEBUG 15
DEBUG 23
DEBUG 16
DEBUG 17
DEBUG 17
DEBUG 21
DEBUG 26
DEBUG 14
DEBUG 3023733371
string too long (0xB43A827B at 0x0000BFE8)

3023733371 is a rather large string, larger than the filesize I believe.

So either there is a problem with the NIF, or there is structure reading problem.

[psi29a]

To test:

from pyffi.formats.nif import NifFormat
from sys import stdout

UIX_PATH = "../../../Private/UIX/UIX FILES/Data Files/Meshes/TOE/RedMoonGa01.NIF"


for stream, data in NifFormat.walkData(UIX_PATH):
    try:
        if True:
            print(stream.name, sep='', end=', ', file=stdout, flush=True)
        data.read(stream)
    except ValueError as ex:
        print(" Error with {0}: {1}".format(stream.name, str(ex.args)), sep='', end='\n',
              file=stdout, flush=True)
    except Exception as ex:
        print(ex)
        raise

[neomonkeus]

Usually you get the error when there is a structural problem in the nif.xml.
I would try an load it in NifSkope to see if it can. Probably should do it with both the current version and the version that pyffi is using.

[psi29a]

Older nifskope loaded the NIF (and I can manipulate it) but gave a warning, the latest 2.x refuses to load the NIF at all and gives the warning I showed above.

[psi29a]

NIFskope 1.1.3:

next block does not start with a NiString"
"failed to load nif from '/home/bcurtis/Private/UIX/UIX FILES/Data Files/Meshes/TOE/RedMoonGa01.NIF'"

but I can at least manipulate the NIF.

2.0.0-dev just refuses to load and give the warning message above.

[psi29a]

So I used NIFSlope 1.1.3 to save the contents of RedMoonGa01.NIF to a file and it is 49.1KiB while the oriignal file is 303.4KiB

I'm wondering if the original file had textures embedded in it, this would account for the filesizes differences if NIFSkope and suppoting libs do not support inernal/embedded textures.

[neomonkeus]

I would ask @jonwd7 for input. The latest nif.xml that is used by NifSkope has changed the way it defines binary data. Even still if its complaining in the 1.1.3, I presuming the corresponding version also doesn't support this, just that NifSkope previously didn't block on this failure. I think we probably need to move this to being a nifxml issue?

[psi29a]

I have no idea really... so you're saying that nif.xml defines how pyffi (and friends) work?

[neomonkeus]

Pyffi reads a format description, in this case an xml file, generates a runtime model, providing supporting functionality to read/write/manipulation/helper functionality for the in-memory instance of the populated model.