Enable multiple layers of overlap for Restricted Additive Schwarz

[andrthu]

Make it possible to add multiple layers of overlap between subdomains after partitioning the grid in parallel, and introducing a separation between overlap and ghost cells (overlap and copy in index set naming). The main changes:

1. Fix the addOverlapLayer function to allow for multiple layers of overlap.

2. Introduce the overlap attribute in the parallel index set to differentiate overlap and ghost/copy cells.

3. Add GhostEntity as a potential Entity partitioning type. This means that a cell can be one of three partitioning types: Interior, Overlap and Ghost.

4. Cells perforated by a well will always be of type Ghost.

The purpose of multiple layers of overlap is to enable the Restricted Additive Schwarz parallel preconditioner. With (1)-(4) only small changes are needed in opm-simulators to use Restricted Additive Schwarz.

[blattms]

Can you please give us a bit more explanation? What is the difference between a GhostEntity and an
OverlapEntity? Why is it needed (in the grid and elsewhere)?

Your PR will change the way that AMG is working in parallel and I suspect it might break it. For AMG with CpGrid there should only be owner and copy indices unless I am mistaking.

It might also lead to wrong partition types in the grid in the DUNE sense. Basically you would need to mark facets between overlap and ghosts as "Front".

Thanks for the bugfix in the code and the documentation. They certainly would have deserved their own commit.

[andrthu]

The reason for introducing the GhostEntity is that for RAS to work, we need to be able separate useful(OverlapEntity) and useless(GhostEntity) overlap cells. A cell is an OverlapEntity if its associated index set attribute is "overlap", and it is GhostEntity if its associated index set attribute is "copy".

When the default 1 layer of overlap is added between subdomains, all overlap cells will have partition type GhostEntity, and all the non-owner index set attributes would be "copy". If two layers of overlap is added, overlap cells that have a face intersection with an interior cell will be of type OverlapEntity(ignoring wells), while overlap cells that don't will be of type GhostEntity.

When the linear system matrix is assembled, rows associated with OverlapEntity cells will be correct, while GhostEntity rows are not. The OverlapEntity rows can be used in the preconditioning of the local system(RAS), while the GhostEntity rows can not.

You are correct about GhostEntity breaking the parallel dune-grid interface. The best thing might be to drop the GhostEntity and just use the index set to create a separation between useful and useless overlap cells.

[blattms]

I am repeating here for reference what I already said privately.

The best thing might be to drop the GhostEntity and just use the index set to create a separation between useful and useless overlap cells.

That does not work either (at least not with a lot of changes to CpGrid which will imply a lot of debugging and testing the dune-grid interface of CpGrid). The indexsets together with the tag are used to indicate the partition types in the grid interface and they are also used to set up the communication interfaces in CpGrid::communicate. Breaking these will impact usability of CpGrid for downstream users of the grid as a DUNE grid and I doubt that this is what we want.

I also think that we made a lot of good experiences in the past by keeping the concepts of linear algebra and grid (at least a bit) separate. Therefore I would not like to further mix these up and instead make it right.

Once can use the dune-grid interface to distinguish what you call ghost and overlap cells during the assembly or when modifying the matrix. For an element in the overlap you loop over its intersections if there is one where Intersection::neighbor() and Intersection::border() both return false then it is a processor boundary but not a domain boundary and hence is what you are describing as a ghost-entity. This should be enough to do your manipulations.

For your RAS preconditioner to work you will need to have or implement a matching LinearOperator (I guess this is the one missing and of course it needs to handle wells like the current one), ScalarProduct and Preconditioner.

[hnil]

For simple users the separation of the grid consept of interior ghost ... and the linear algebra part owner, overlap and copy is difficult to understand. For the use of the RAS preconditioner the overlap is seperated in usefull overlap i.e. overlap and copy which is non usefull only needed to get right linear operator without extra operations.