Development environment change

[SkyLeach]

This pull request is in keeping with the overall virtual environment/multiple sources requirements of heavy development.

When working with panda3d and modules developers may need or wish to try out features within a python virtual environment.

They may recompile panda from source (as I do) with new flags and then wish to have those dynamic libraries available within a virtual environment rather than a system library.

This change merely adds allowance for building (not installing) against a panda source/build location.

In my test code, I have the envvar LOCAL_PANDA_BUILD=~/src/panda3d/built

Tested with panda_LUI on Mac OSX. In this source base, the variable must come in as -DLOCAL_PANDA_BUILD:STRING=<value of envvar>

[tobspr]

Just to get it right, basically your PR adds support for building e.g. LUI against a Panda3D version which was locally built and not installed on your system?

[SkyLeach]

@tobspr yes exactly.

However, you'll notice that there are more changes. This is because I didn't realize that some of my files got blown away before I forked and pushed them. I did my original editing and testing from within panda_LUI. the scripts/interrogate.py and scripts/common.py changes are missing from this PR.

I'm working on redoing them.

The discussion of why this is really important is under the panda3d issues list. There is also a PR pending there ( that is indirectly related to this ) and has to do with python virtualenv development that may depend on modules that conflict with one another and thus should never be installed into the system libraries.

A short tl;dr on it is basically that macos gets really messed up if you replace the system python. Most python developers will therefore use homebrew to install open source tools, python included, into a non-system location (usually /usr/local with the framework in /user/local/Frameworks/).

The default panda builds and installs place the headers and includes into either the system /Library or user ~/Library paths. This is ok as far as it goes, being a stable build.

When working with modules, especially custom modules, however one must keep various projects in mind. An unstable .dylib or .so in a system path can break everything system-wide.

Thus most python developers use a virtualenv where any given project is actually linking panda from ~/.virtualenvs/<project_name>/lib/python/site-packages/panda3d .

This location does not, however, contain non-wheel dynamic libraries or includes/headers.

Thus using import panda3d.core and panda3d.core.__file__ with abspath is actually linking against the wrong panda and very difficult to work around without creating a very unstable development system. Using those is fine for non-developer linking, but not for developer linking.

[SkyLeach]

merged commits from master and tested.

[tobspr]

I am a bit unhappy with this PR since it has so many changes which are unrelated (For example the logging). This makes it hard to track what it actually does ..
Also, I am not sure if your changes are actually required on windows, since you can perfectly have your local panda installation, and it will grab it depending on the ppython location. I am not really experienced with virtualenv and how it will work in this case, though.

[SkyLeach]

Yes, the changes are also helpful on windows. Specifically in these situations:

1. Panda3D is installed only via wheel: the libraries will not be found.
2. Panda3D is installed in a nonstandard location: the libraries and headers will not be found.
3. Panda3D is installed in the standard location, but the available panda3d does not have features desired by the module developer. Overwriting the default install with a potentially unstable panda may be disastrous to other workflows.
4. Testing new versions of panda against end-user modules that can fatally crash python when loading the .dll

Wheel installations (virtual environments) are extensively used by python developers due to the need to develop against specific module versions. Each virtual environment depends on a very explicit install of modules (usually) and this is doubly true for people developing third-party modules with c/c++ extensions.

As to the logging, check it out now. I added a couple of custom handlers that should make things appear identical to before, but retaining the functionality of modular logging and redirection (a best-practices for all python modules).

The log formatter isn't required, it's just makes python developers very unhappy when support modules call the print function directly. The logger supports INFO (default) VERBOSE (custom) ERROR and WARN loglevels as well as log redirection by level to files.

[SkyLeach]

Examples:

No express loglevel

| => LOCAL_PANDA_BUILD=~/src/panda3d/built python build.py
Creating directory/Users/magregor/src/panda_LUI/osx_amd64_panda1.10.0_py36
Executing command: cmake /Users/magregor/src/panda_LUI -DCMAKE_BUILD_TYPE=RelWithDebInfo -DPYTHON_EXECUTABLE:STRING=/Users/magregor/.virtualenvs/SIM/bin/python -DPROJECT_NAME:STRING=lui -DINTERROGATE_LIB:STRING=p3interrogatedb -DCMAKE_CL_64:STRING=1 -DLOCAL_PANDA_BUILD:STRING=/Users/magregor/src/panda3d/built -DTHIRDPARTY_WIN_DIR= -DHAVE_LIB_EIGEN=TRUE -DHAVE_LIB_FREETYPE=TRUE -DIGATE_VERBOSE=0 -DOPTIMIZE=3

-- Configuring done
-- Generating done
-- Build files have been written to: /Users/magregor/src/panda_LUI/osx_amd64_panda1.10.0_py36
Executing command: cmake --build . --config RelWithDebInfo -- -j7

Scanning dependencies of target lui
[  3%] Building CXX object CMakeFiles/lui.dir/source/interrogate_module.cpp.o
[  7%] Building CXX object CMakeFiles/lui.dir/source/interrogate_wrapper.cpp.o
[ 11%] Linking CXX shared library lui.so
[100%] Built target lui
Success!

Clean build, debugging enabled:

| => LOCAL_PANDA_BUILD=~/src/panda3d/built python build.py --debug
Debug enabled.
Verbose logging enabled.
Creating directory/Users/magregor/src/panda_LUI/osx_amd64_panda1.10.0_py36
Executing command: cmake /Users/magregor/src/panda_LUI -DCMAKE_BUILD_TYPE=RelWithDebInfo -DPYTHON_EXECUTABLE:STRING=/Users/magregor/.virtualenvs/SIM/bin/python -DPROJECT_NAME:STRING=lui -DINTERROGATE_LIB:STRING=p3interrogatedb -DCMAKE_CL_64:STRING=1 -DLOCAL_PANDA_BUILD:STRING=/Users/magregor/src/panda3d/built -DTHIRDPARTY_WIN_DIR= -DHAVE_LIB_EIGEN=TRUE -DHAVE_LIB_FREETYPE=TRUE -DIGATE_VERBOSE=0 -DOPTIMIZE=3

-- The C compiler identification is AppleClang 9.0.0.9000039
-- The CXX compiler identification is AppleClang 9.0.0.9000039
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonLibs: /usr/local/Frameworks/Python.framework/Versions/3.6/lib/libpython3.6m.dylib (found version "3.6.4")
-- Found Freetype: /Users/magregor/src/panda3d/built/lib/libfreetype.dylib (found version "2.9.0")
-- Found Panda: ;/Users/magregor/src/panda3d/built/lib/libp3framework.dylib;/Users/magregor/src/panda3d/built/lib/libpanda.dylib;/Users/magregor/src/panda3d/built/lib/libpandaexpress.dylib;/Users/magregor/src/panda3d/built/lib/libp3dtool.dylib;/Users/magregor/src/panda3d/built/lib/libp3dtoolconfig.dylib;/Users/magregor/src/panda3d/built/lib/libp3direct.dylib;/Users/magregor/src/panda3d/built/lib/libp3interrogatedb.dylib
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_CL_64


-- Build files have been written to: /Users/magregor/src/panda_LUI/osx_amd64_panda1.10.0_py36
Executing command: cmake --build . --config RelWithDebInfo -- -j7

Scanning dependencies of target lui
[  3%] Building CXX object CMakeFiles/lui.dir/source/interrogate_module.cpp.o
[  7%] Building CXX object CMakeFiles/lui.dir/source/interrogate_wrapper.cpp.o
[ 11%] Building CXX object CMakeFiles/lui.dir/source/config_lui.cxx.o
[ 14%] Building CXX object CMakeFiles/lui.dir/source/luiAtlas.cxx.o
[ 18%] Building CXX object CMakeFiles/lui.dir/source/luiAtlasPacker.cxx.o
[ 22%] Building CXX object CMakeFiles/lui.dir/source/luiAtlasDescriptor.cxx.o
[ 25%] Building CXX object CMakeFiles/lui.dir/source/luiAtlasPool.cxx.o
[ 29%] Building CXX object CMakeFiles/lui.dir/source/luiBaseElement.cxx.o
[ 33%] Building CXX object CMakeFiles/lui.dir/source/luiBaseLayout.cxx.o
/Users/magregor/src/panda_LUI/source/luiBaseElement.cxx:146:5: warning: expression result unused [-Wunused-value]
    _debug_name;
    ^~~~~~~~~~~
[ 37%] Building CXX object CMakeFiles/lui.dir/source/luiBounds.cxx.o
[ 40%] Building CXX object CMakeFiles/lui.dir/source/luiChunkDescriptor.cxx.o
[ 44%] Building CXX object CMakeFiles/lui.dir/source/luiColorable.cxx.o
1 warning generated.
[ 48%] Building CXX object CMakeFiles/lui.dir/source/luiEventData.cxx.o
[ 51%] Building CXX object CMakeFiles/lui.dir/source/luiExpression.cxx.o
[ 55%] Building CXX object CMakeFiles/lui.dir/source/luiFontPool.cxx.o
[ 59%] Building CXX object CMakeFiles/lui.dir/source/luiHorizontalLayout.cxx.o
[ 62%] Building CXX object CMakeFiles/lui.dir/source/luiInputHandler.cxx.o
[ 66%] Building CXX object CMakeFiles/lui.dir/source/luiObject.cxx.o
[ 70%] Building CXX object CMakeFiles/lui.dir/source/luiRect.cxx.o
[ 74%] Building CXX object CMakeFiles/lui.dir/source/luiRegion.cxx.o
[ 77%] Building CXX object CMakeFiles/lui.dir/source/luiRoot.cxx.o
[ 81%] Building CXX object CMakeFiles/lui.dir/source/luiSprite.cxx.o
[ 85%] Building CXX object CMakeFiles/lui.dir/source/luiText.cxx.o
[ 88%] Building CXX object CMakeFiles/lui.dir/source/luiVertexChunk.cxx.o
[ 92%] Building CXX object CMakeFiles/lui.dir/source/luiVertexPool.cxx.o
[ 96%] Building CXX object CMakeFiles/lui.dir/source/luiVerticalLayout.cxx.o
[100%] Linking CXX shared library lui.so
[100%] Built target lui
Success!

Notice that the logging format is set to output the module name along with messages. It can also be configured to output timestamps and other information.

[tobspr]

Panda3D is installed only via wheel: the libraries will not be found.

That is right, but wheel builds are not generally supported since they do not include the panda3d headers. So building against a wheel build is not possible, until wheel builds which also include the headers are available.

Panda3D is installed in a nonstandard location: the libraries and headers will not be found.

The libraries and headers will be found on windows, since you run it via the python provided by the panda3d sdk, which is installed at the same location. Or do I misunderstand that?

Panda3D is installed in the standard location, but the available panda3d does not have features desired by the module developer. Overwriting the default install with a potentially unstable panda may be disastrous to other workflows.

I am not sure what you mean here. If panda does not have the required features (e.g. freetype, eigen) then you also can't link against it.

Testing new versions of panda against end-user modules that can fatally crash python when loading the .dll

I am also not sure what you mean here. Could you explain?

I am not against the logging, I am just saying that it messes up the PR so that its hard to track the actual changes.