Re-work 'testing' directory structure

[adrifoster]

The current 'testing' direction is getting a bit out of hand. I'd like to create some sub-directories so that things are easier to find and use. My proposal is below:

testing/
├── run_functional.py   (formerly run_functional_tests.py) 
├── run_unit.py   (formerly run_unit_tests.py)
├── generate_test.py   (formerly generate_empty_test.py)
├── CMakeLists.txt 
├── environment.yml (formerly testing.yml)
├── README.md (formerly README.testing.md)
├── config/
│   ├── functional.cfg (formerly functional_test.cfg)
│   └── unit.cfg (formerly unit_tests.cfg)
├── tests/                        
│   ├── functional/
│   │   ├── ... testing dirs ...
│   ├── unit/
│   │   ├── ... testing dirs ...
│   ├── fortran_shr/
│   │   ├── CMakeLists.txt
│   │   └── *.F90
│   ├── data/            
│   │    └── *.csv, *.nc...
│
├── framework/             
│  ├── __init__.py 
│  ├── functional_class.py
│  ├── functional_class_with_drivers.py
│  ├── builder.py (formerly build_fortran_tests.py)
│  ├── generator.py (formerly test_generator_class.py)
│  ├── loader.py (formerly load_functional_tests.py)
│  ├── utils/
│  │   ├── __init__.py
│  │   ├── general.py (formerly utils.py)
│  │   ├── plotting.py (formerly utils_plotting.py)
│  │   ├── paths.py (formerly path_utils.py)
│  │   └── protocols.py
│
├── templates/
│   ├── cmake_ftest_template.txt
│   ├── cmake_utest_template.txt
│   ├── fortran_test_template.txt
│   ├── pfunit_template.txt
│   └── test_class_template.txt
├── docs/
│   ├── cime_setup.md

   

Explanation of each directory:

---

The top-level directory contains the executable scripts so they are easy to find and run, hide all supporting classes and helpers inside a framework directory

---

config/ - contains config files for the tests

---

tests/ - contains only code/files that represent actual tests

• functional tests (each in own directory)
• unit tests (each in own directory)
• shared Fortran modules used by tests
• data

---

framework/ - machinery to compile/run tests

framework/utils contains utilities for these

---

templates/ - templates for writing new tests

---

docs/ - documents

---

I think this will be nice because:

1. we still can just do testing/run_functional.py like always
2. no "pip install" required, all internal, we can just use a relative path import from the top-level scripts
3. much cleaner and clearer - we only see 6 files at the top, not ~30

I would love to get folks' feedback on this structure! @ekluzek @rgknox @glemieux

[ekluzek]

In general this looks good to me. Names are simplified and take advantage of the parent directory being "testing" names don't need to repeat that. You also change some files to standard names like environment.yml. I like the idea of removing pip from this.

In the current structure, there is the testing directory and there's also the functional_unit_testing at the same level. I take it that this only changes things under "testing" so doesn't change the functional_unit_testing directory at the same level right? And I'm not sure how that directory tree relates to this testing one?

I have some specific question/comments that I'll give in the next comment.

[ekluzek]

OK, questions/comments on this level:

├── tests/                        
│   ├── functional/
│   │   ├── ... testing dirs ...
│   ├── unit/
│   │   ├── ... testing dirs ...
│   ├── fortran_shr/
│   │   ├── CMakeLists.txt
│   │   └── *.F90
│   ├── data/            
│   │    └── *.csv, *.nc...

fortran_shr, probably has utility code that's shared between functional and unit right? This could be utility code, but also mocks or stubs. I think "fortran" doesn't help to tell you what goes there. I think you are using "fortran" to maybe distinguish it from python code that might be shared right? But, most of the python stuff will be in the framework directory level so that distinction isn't important. And it seems like there could be python code that you'd want to share between unit and functional that could go there. If you'd want it to be separate you could start a subdirectory for python code -- but I don't think you need one. So how about just "share" instead of "fortran_shr"? Or to be more explicit something like "functional_unit_shared"?

For the "data" directory, I think this is just going to apply for the functional testing, so maybe it should go underneath there? But, having it at that level also makes sense.

[ekluzek]

The other name I wonder about is:

├── framework/

I think this might be the right name, as I'm trying to think of alternatives and am not coming up with anything. This directory is really the python modules that enable the top level scripts. So in CTSM world it's similar to our "python/ctsm/<specific_type>" directory structure. And it means the python modules will be referenced with in import statements as framework.<specific_type>. I personally like how CIME and ctsm have either the CIME. or ctsm. prefix. I do see the argument that because this is embedded under FATES that it goes without saying. But, personally I'd like to see a prefix like FATES.test_framework or FATES.test.framework maybe? What do you think about that @adrifoster?

[adrifoster]

In general this looks good to me. Names are simplified and take advantage of the parent directory being "testing" names don't need to repeat that. You also change some files to standard names like environment.yml. I like the idea of removing pip from this.

In the current structure, there is the testing directory and there's also the functional_unit_testing at the same level. I take it that this only changes things under "testing" so doesn't change the functional_unit_testing directory at the same level right? And I'm not sure how that directory tree relates to this testing one?

I have some specific question/comments that I'll give in the next comment.

Yes in the future @rgknox and I will be moving the functional_unit_testing tests into this testing structure (so tests would either go somewhere in tests/functional or tests/unit depending. So that directory will go away eventually.

[adrifoster]

The other name I wonder about is:

├── framework/

I think this might be the right name, as I'm trying to think of alternatives and am not coming up with anything. This directory is really the python modules that enable the top level scripts. So in CTSM world it's similar to our "python/ctsm/<specific_type>" directory structure. And it means the python modules will be referenced with in import statements as framework.<specific_type>. I personally like how CIME and ctsm have either the CIME. or ctsm. prefix. I do see the argument that because this is embedded under FATES that it goes without saying. But, personally I'd like to see a prefix like FATES.test_framework or FATES.test.framework maybe? What do you think about that @adrifoster?

right now the testing python directory is divorced from the tooling python directory. So I'm not sure adding an extra FATES directory here would be helpful. And I think discussing combining these different directories is outside the scope of this issue/PR.