Skip to content

Commit

Permalink
Merge branch 'master' of github.com:nest/nest-simulator into fix-arti…
Browse files Browse the repository at this point in the history
…fact-upload
heplesser committed Jun 4, 2024
2 parents ead9845 + d49d1b5 commit f5d5095
Showing 51 changed files with 2,102 additions and 3,448 deletions.
4 changes: 3 additions & 1 deletion .github/workflows/nestbuildmatrix.yml
Original file line number Diff line number Diff line change
@@ -707,6 +707,8 @@ jobs:
echo "backend : svg" > $HOME/.matplotlib/matplotlibrc
- name: "Run NEST testsuite"
env:
DO_TESTS_SKIP_TEST_REQUIRING_MANY_CORES: ${{ contains(matrix.use, 'mpi') && contains(matrix.use, 'openmp') }}
run: |
pwd
cd "$NEST_VPATH"
@@ -757,7 +759,7 @@ jobs:
- name: "Set up Python 3.x"
uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5.0.0
with:
python-version: 3.9
python-version: 3.12

- name: "Install MacOS system dependencies"
run: |
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@
[![Documentation](https://img.shields.io/readthedocs/nest-simulator?logo=readthedocs&logo=Read%20the%20Docs&label=Documentation)](https://nest-simulator.org/documentation)
[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/2218/badge)](https://bestpractices.coreinfrastructure.org/projects/2218)
[![License](http://img.shields.io/:license-GPLv2+-green.svg)](http://www.gnu.org/licenses/gpl-2.0.html)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.8069926.svg)](https://doi.org/10.5281/zenodo.8069926)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.10834751.svg)](https://doi.org/10.5281/zenodo.10834751)

[![Latest release](https://img.shields.io/github/release/nest/nest-simulator.svg?color=brightgreen&label=latest%20release)](https://github.com/nest/nest-simulator/releases)
[![GitHub contributors](https://img.shields.io/github/contributors/nest/nest-simulator?logo=github)](https://github.com/nest/nest-simulator)
2 changes: 1 addition & 1 deletion doc/htmldoc/hpc/parallel_computing.rst
Original file line number Diff line number Diff line change
@@ -161,7 +161,7 @@ Spikes between neurons and devices
Synaptic plasticity models
~~~~~~~~~~~~~~~~~~~~~~~~~~

For synapse models supporting plasticity, synapse dynamics in the
For synapse models supporting :hxt_ref:`plasticity`, synapse dynamics in the
``Connection`` object are always handled by the virtual process of the
`target node`.

2 changes: 1 addition & 1 deletion doc/htmldoc/tutorials/music_tutorial/music_tutorial_1.rst
Original file line number Diff line number Diff line change
@@ -56,7 +56,7 @@ A NEST network consists of three types of elements: neurons, devices,
and connections between them.

Neurons are the basic building blocks, and in NEST they are generally
spiking point neuron models. Devices are supporting units that for
spiking :hxt_ref:`point neuron` models. Devices are supporting units that for
instance generate inputs to neurons or record data from them. The
Poisson spike generator, the spike recorder recording device, and the
MUSIC input and output proxies are all devices. Neurons and devices are
Original file line number Diff line number Diff line change
@@ -64,10 +64,10 @@ STDP synapses

For the majority of synapses, all of their parameters are accessible via
:py:func:`.GetDefaults` and :py:func:`.SetDefaults`. Synapse models implementing
spike-timing dependent plasticity are an exception to this, as their
:hxt_ref:`spike-timing dependent plasticity` are an exception to this, as their
dynamics are driven by the postsynaptic :hxt_ref:`spike train` as well as the
pre-synaptic one. As a consequence, the time constant of the depressing
window of :hxt_ref:`STDP` is a parameter of the postsynaptic neuron. It can be set
window of STDP is a parameter of the postsynaptic neuron. It can be set
as follows:

::
13 changes: 8 additions & 5 deletions lib/sli/nest-init.sli
Original file line number Diff line number Diff line change
@@ -488,9 +488,14 @@ def

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% add conversion from NodeCollection
/cva [/nodecollectiontype]
/cva_g load
% add new functions to trie if it exists, else create new
/cva dup lookup not
{
trie
} if
[/connectiontype] /cva_C load addtotrie
[/nodecollectiontype] { /all cva_g_l } addtotrie
[/nodecollectiontype /literaltype] /cva_g_l load addtotrie
def

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -1426,8 +1431,6 @@ def
[/integertype /integertype] /TimeCommunicationAlltoallv_i_i load addtotrie
def

/cva [/connectiontype] /cva_C load def

/abort
{
statusdict /exitcodes get /userabort get
1 change: 1 addition & 0 deletions libnestutil/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -27,6 +27,7 @@ set( nestutil_sources
lockptr.h
logging_event.h logging_event.cpp
logging.h
nest_types.h
numerics.h numerics.cpp
regula_falsi.h
sort.h
File renamed without changes.
144 changes: 144 additions & 0 deletions libnestutil/numerics.cpp
Original file line number Diff line number Diff line change
@@ -20,6 +20,11 @@
*
*/

#include <cstdlib>
#include <iostream>
#include <numeric>

#include "nest_types.h"
#include "numerics.h"

#ifndef HAVE_M_E
@@ -126,3 +131,142 @@ is_integer( double n )
// factor 4 allows for two bits of rounding error
return frac_part < 4 * n * std::numeric_limits< double >::epsilon();
}

long
mod_inverse( long a, long m )
{
/*
The implementation here is based on the extended Euclidean algorithm which
solves
a x + m y = gcd( a, m ) = 1 mod m
for x and y. Note that the m y term is zero mod m, so the equation is equivalent
to
a x = 1 mod m
Since we only need x, we can ignore y and use just half of the algorithm.
We can assume without loss of generality that a < m, because if a = a' + j m with
0 < a' < m, we have
a x mod m = (a' + j m) x mod m = a' x + j x m mod m = a' x.
This implies that m ≥ 2.
For details on the algorithm, see D. E. Knuth, The Art of Computer Programming,
ch 4.5.2, Algorithm X (vol 2), and ch 1.2.1, Algorithm E (vol 1).
*/

assert( 0 < a );
assert( 2 <= m );

const long a_orig = a;
const long m_orig = m;

// If a ≥ m, the algorithm needs two extra rounds to transform this to
// a' < m, so we take care of this in a single step here.
a = a % m;

// Use half of extended Euclidean algorithm required to compute inverse
long s_0 = 1;
long s_1 = 0;

while ( a > 0 )
{
// get quotient and remainder in one go
const auto res = div( m, a );
m = a;
a = res.rem;

// line ordering matters here
const long s_0_new = -res.quot * s_0 + s_1;
s_1 = s_0;
s_0 = s_0_new;
}

// ensure positive result
s_1 = ( s_1 + m_orig ) % m_orig;

assert( m == 1 ); // gcd() == 1 required
assert( ( a_orig * s_1 ) % m_orig == 1 ); // self-test

return s_1;
}

size_t
first_index( long period, long phase0, long step, long phase )
{
assert( period > 0 );
assert( step > 0 );
assert( 0 <= phase0 and phase0 < period );
assert( 0 <= phase and phase < period );

/*
The implementation here is based on
https://math.stackexchange.com/questions/25390/how-to-find-the-inverse-modulo-m
We first need to solve
phase0 + k step = phase mod period
<=> k step = ( phase - phase0 ) = d_phase mod period
This has a solution iff d = gcd(step, period) divides d_phase.
Then, if d = 1, the solution is unique and given by
k' = mod_inv(step) * d_phase mod period
If d > 1, we need to divide the equation by it and solve
(step / d) k_0 = d_phase / d mod (period / d)
The set of solutions is then given by
k_j = k_0 + j * period / d for j = 0, 1, ..., d-1
and we need the smallest of these. Now we are interested in
an index given by k * step with a period of lcm(step, period)
(the outer_period below, marked by | in the illustration in
the doxygen comment), we immediately run over
k_j * step = k_0 * step + j * step * period / d mod lcm(step, period)
below and take the smallest. But since step * period / d = lcm(step, period),
the term in j above vanishes and k_0 * step mod lcm(step, period) is actually
the solution.
We do all calculations in signed long since we may encounter negative
values during the algorithm. The result will be non-negative and returned
as size_t. This is important because the "not found" case is signaled
by invalid_index, which is size_t.
*/

// This check is not only a convenience: If step == k * period, we only match if
// phase == phase0 and the algorithm below will fail if we did not return here
// immediately, because we get d == period -> period_d = 1 and modular inverse
// for modulus 1 makes no sense.
if ( phase == phase0 )
{
return 0;
}

const long d_phase = ( phase - phase0 + period ) % period;
const long d = std::gcd( step, period );

if ( d_phase % d != 0 )
{
return nest::invalid_index; // no solution exists
}

// Scale by GCD, since modular inverse requires gcd==1
const long period_d = period / d;
const long step_d = step / d;
const long d_phase_d = d_phase / d;

// Compute k_0 and multiply by step, see explanation in introductory comment
const long idx = ( d_phase_d * mod_inverse( step_d, period_d ) * step ) % std::lcm( period, step );

return static_cast< size_t >( idx );
}
48 changes: 48 additions & 0 deletions libnestutil/numerics.h
Original file line number Diff line number Diff line change
@@ -131,4 +131,52 @@ double dtruncate( double );
*/
bool is_integer( double );

/**
* Returns inverse of integer a modulo m
*
* For integer a > 0, m ≥ 2, find x so that ( a * x ) mod m = 1.
*/
long mod_inverse( long a, long m );

/**
* Return first matching index for entry in container with double periodicity.
*
* Consider
* - a container of arbitrary length containing elements (e.g. GIDs) which map to certain values,
* e.g., VP(GID), with a given *period*, e.g., the number of virtual processes
* - that the phase (e.g., the VP number) of the first entry in the container is *phase0*
* - that we slice the container with a given *step*, causing a double periodicity
* - that we want to know the index of the first element in the container with a given *phase*,
* e.g., the first element on a given VP
*
* If such an index x exists, it is given by
*
* x = phase0 + k' mod lcm(period, step)
* k' = min_k ( phase0 + k step = phase mod period )
*
* As an example, consider
*
* idx 0 1 2 3 4 5 6 7 8 9 10 11 | 12 13 14 15 16 17 18
* gid 1 2 3 4 5 6 7 8 9 10 11 12 | 13 14 15 16 17 18 19
* vp 1 2 3 0 1 2 3 0 1 2 3 0 | 1 2 3 0 1 2 3
* * * * * | * * *
* 1 0 3 2 |
*
* Here, idx is a linear index into the container, gid neuron ids which map to the given vp numbers.
* The container is sliced with step=3, i.e., starting with the first element we take every third, as
* marked by the asterisks. The question is then at which index we find the first entry belonging to
* vp 0, 1, 2, or 3. The numbers in the bottom row show for clarity on which VP we find the respective
* asterisks. The | symbol marks where the pattern repeats itself.
*
* phase0 in this example is 1, i.e., the VP of the first element in the container.
*
* @note This function returns that index. It is the responsibility of the caller to check if the returned index
* is within the bounds of the actual container—the algorithm assumes an infinite container.
*
* @note The function returns *invalid_index* if no solution exists.
*
* See comments in the function definition for implementation details.
*/
size_t first_index( long period, long phase0, long step, long phase );

#endif
2 changes: 1 addition & 1 deletion models/iaf_psc_delta.cpp
Original file line number Diff line number Diff line change
@@ -154,7 +154,7 @@ nest::iaf_psc_delta::Parameters_::set( const DictionaryDatum& d, Node* node )
}
if ( c_m_ <= 0 )
{
throw BadProperty( "Capacitance must be >0." );
throw BadProperty( "Capacitance must be > 0." );
}
if ( t_ref_ < 0 )
{
1 change: 0 additions & 1 deletion nestkernel/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -44,7 +44,6 @@ set ( nestkernel_sources
histentry.h histentry.cpp
model.h model.cpp
model_manager.h model_manager_impl.h model_manager.cpp
nest_types.h
nest_datums.h nest_datums.cpp
nest_names.cpp nest_names.h
nestmodule.h nestmodule.cpp
13 changes: 7 additions & 6 deletions nestkernel/conn_builder.cpp
Original file line number Diff line number Diff line change
@@ -623,7 +623,7 @@ nest::OneToOneBuilder::connect_()
Node* target = n->get_node();

const size_t tnode_id = n->get_node_id();
const long lid = targets_->get_lid( tnode_id );
const long lid = targets_->get_nc_index( tnode_id );
if ( lid < 0 ) // Is local node in target list?
{
continue;
@@ -823,7 +823,7 @@ nest::AllToAllBuilder::connect_()
const size_t tnode_id = n->get_node_id();

// Is the local node in the targets list?
if ( targets_->get_lid( tnode_id ) < 0 )
if ( targets_->get_nc_index( tnode_id ) < 0 )
{
continue;
}
@@ -1106,7 +1106,7 @@ nest::FixedInDegreeBuilder::connect_()
const size_t tnode_id = n->get_node_id();

// Is the local node in the targets list?
if ( targets_->get_lid( tnode_id ) < 0 )
if ( targets_->get_nc_index( tnode_id ) < 0 )
{
continue;
}
@@ -1538,7 +1538,7 @@ nest::BernoulliBuilder::connect_()
const size_t tnode_id = n->get_node_id();

// Is the local node in the targets list?
if ( targets_->get_lid( tnode_id ) < 0 )
if ( targets_->get_nc_index( tnode_id ) < 0 )
{
continue;
}
@@ -1654,7 +1654,7 @@ nest::PoissonBuilder::connect_()
const size_t tnode_id = n->get_node_id();

// Is the local node in the targets list?
if ( targets_->get_lid( tnode_id ) < 0 )
if ( targets_->get_nc_index( tnode_id ) < 0 )
{
continue;
}
@@ -1847,7 +1847,8 @@ nest::TripartiteBernoulliWithPoolBuilder::connect_()
}
else
{
std::copy_n( third_->begin() + get_first_pool_index_( target.lid ), pool_size_, std::back_inserter( pool ) );
std::copy_n(
third_->begin() + get_first_pool_index_( target.nc_index ), pool_size_, std::back_inserter( pool ) );
}

// step 3, iterate through indegree to make connections for this target
2 changes: 0 additions & 2 deletions nestkernel/connection_creator.h
Original file line number Diff line number Diff line change
@@ -87,8 +87,6 @@ class ConnectionCreator
* - "mask": Mask definition (dictionary or masktype).
* - "kernel": Kernel definition (dictionary, parametertype, or double).
* - "synapse_model": The synapse model to use.
* - "targets": Which targets (model or lid) to select (dictionary).
* - "sources": Which targets (model or lid) to select (dictionary).
* - "weight": Synaptic weight (dictionary, parametertype, or double).
* - "delay": Synaptic delays (dictionary, parametertype, or double).
* - other parameters are interpreted as synapse parameters, and may
16 changes: 8 additions & 8 deletions nestkernel/connection_creator_impl.h
Original file line number Diff line number Diff line change
@@ -264,7 +264,7 @@ ConnectionCreator::pairwise_bernoulli_on_source_( Layer< D >& source,

if ( not tgt->is_proxy() )
{
const Position< D > target_pos = target.get_position( ( *tgt_it ).lid );
const Position< D > target_pos = target.get_position( ( *tgt_it ).nc_index );

if ( mask_.get() )
{
@@ -339,7 +339,7 @@ ConnectionCreator::pairwise_bernoulli_on_target_( Layer< D >& source,
const int thread_id = kernel().vp_manager.get_thread_id();
try
{
NodeCollection::const_iterator target_begin = target_nc->local_begin();
NodeCollection::const_iterator target_begin = target_nc->thread_local_begin();
NodeCollection::const_iterator target_end = target_nc->end();

for ( NodeCollection::const_iterator tgt_it = target_begin; tgt_it < target_end; ++tgt_it )
@@ -348,7 +348,7 @@ ConnectionCreator::pairwise_bernoulli_on_target_( Layer< D >& source,

assert( not tgt->is_proxy() );

const Position< D > target_pos = target.get_position( ( *tgt_it ).lid );
const Position< D > target_pos = target.get_position( ( *tgt_it ).nc_index );

if ( mask_.get() )
{
@@ -423,7 +423,7 @@ ConnectionCreator::pairwise_poisson_( Layer< D >& source,

if ( not tgt->is_proxy() )
{
const Position< D > target_pos = target.get_position( ( *tgt_it ).lid );
const Position< D > target_pos = target.get_position( ( *tgt_it ).nc_index );

if ( mask_.get() )
{
@@ -477,7 +477,7 @@ ConnectionCreator::fixed_indegree_( Layer< D >& source,
throw IllegalConnection( "Spatial Connect with fixed_indegree to devices is not possible." );
}

NodeCollection::const_iterator target_begin = target_nc->MPI_local_begin();
NodeCollection::const_iterator target_begin = target_nc->rank_local_begin();
NodeCollection::const_iterator target_end = target_nc->end();

// protect against connecting to devices without proxies
@@ -504,7 +504,7 @@ ConnectionCreator::fixed_indegree_( Layer< D >& source,

size_t target_thread = tgt->get_thread();
RngPtr rng = get_vp_specific_rng( target_thread );
Position< D > target_pos = target.get_position( ( *tgt_it ).lid );
Position< D > target_pos = target.get_position( ( *tgt_it ).nc_index );

// We create a source pos vector here that can be updated with the
// source position. This is done to avoid creating and destroying
@@ -637,7 +637,7 @@ ConnectionCreator::fixed_indegree_( Layer< D >& source,
Node* const tgt = kernel().node_manager.get_node_or_proxy( target_id );
size_t target_thread = tgt->get_thread();
RngPtr rng = get_vp_specific_rng( target_thread );
Position< D > target_pos = target.get_position( ( *tgt_it ).lid );
Position< D > target_pos = target.get_position( ( *tgt_it ).nc_index );

unsigned long target_number_connections = std::round( number_of_connections_->value( rng, tgt ) );

@@ -769,7 +769,7 @@ ConnectionCreator::fixed_outdegree_( Layer< D >& source,
throw IllegalConnection( "Spatial Connect with fixed_outdegree to devices is not possible." );
}

NodeCollection::const_iterator target_begin = target_nc->MPI_local_begin();
NodeCollection::const_iterator target_begin = target_nc->rank_local_begin();
NodeCollection::const_iterator target_end = target_nc->end();

for ( NodeCollection::const_iterator tgt_it = target_begin; tgt_it < target_end; ++tgt_it )
55 changes: 43 additions & 12 deletions nestkernel/free_layer.h
Original file line number Diff line number Diff line change
@@ -48,9 +48,9 @@ template < int D >
class FreeLayer : public Layer< D >
{
public:
Position< D > get_position( size_t sind ) const;
void set_status( const DictionaryDatum& );
void get_status( DictionaryDatum& ) const;
Position< D > get_position( size_t sind ) const override;
void set_status( const DictionaryDatum& ) override;
void get_status( DictionaryDatum&, NodeCollection const* ) const override;

protected:
/**
@@ -61,14 +61,14 @@ class FreeLayer : public Layer< D >
template < class Ins >
void communicate_positions_( Ins iter, NodeCollectionPTR node_collection );

void insert_global_positions_ntree_( Ntree< D, size_t >& tree, NodeCollectionPTR node_collection );
void insert_global_positions_ntree_( Ntree< D, size_t >& tree, NodeCollectionPTR node_collection ) override;
void insert_global_positions_vector_( std::vector< std::pair< Position< D >, size_t > >& vec,
NodeCollectionPTR node_collection );
NodeCollectionPTR node_collection ) override;

/**
* Calculate the index in the position vector on this MPI process based on the local ID.
*
* @param lid local ID of the node
* @param lid global index of node within layer
* @return index in the local position vector
*/
size_t lid_to_position_id_( size_t lid ) const;
@@ -255,15 +255,46 @@ FreeLayer< D >::set_status( const DictionaryDatum& d )

template < int D >
void
FreeLayer< D >::get_status( DictionaryDatum& d ) const
FreeLayer< D >::get_status( DictionaryDatum& d, NodeCollection const* nc ) const
{
Layer< D >::get_status( d );
Layer< D >::get_status( d, nc );

TokenArray points;
for ( typename std::vector< Position< D > >::const_iterator it = positions_.begin(); it != positions_.end(); ++it )

if ( not nc )
{
// This is needed by NodeCollectionMetadata::operator==() which does not have access to the node collection
for ( const auto& pos : positions_ )
{
points.push_back( pos.getToken() );
}
}
else
{
points.push_back( it->getToken() );
// Selecting the right positions
// - Coordinates for all nodes in the underlying primitive node collection
// which belong to this rank are stored in positions_
// - nc has information on which nodes actually belong to it, especially
// important for sliced collections with step > 1.
// - Use the rank-local iterator over the node collection to pick the right
// nodes, then step in lockstep through the positions_ array.
auto nc_it = nc->rank_local_begin();
const auto nc_end = nc->end();
if ( nc_it < nc_end )
{
// Node index in node collection is global to NEST, so we need to scale down
// to get right indices into positions_, which has only rank-local data.
const size_t n_procs = kernel().mpi_manager.get_num_processes();
size_t pos_idx = ( *nc_it ).nc_index / n_procs;
size_t step = nc_it.get_step_size() / n_procs;

for ( ; nc_it < nc->end(); pos_idx += step, ++nc_it )
{
points.push_back( positions_.at( pos_idx ).getToken() );
}
}
}

def2< TokenArray, ArrayDatum >( d, names::positions, points );
}

@@ -288,7 +319,7 @@ FreeLayer< D >::communicate_positions_( Ins iter, NodeCollectionPTR node_collect
// know that all nodes in the NodeCollection have proxies. Likewise, if it returns false we know that
// no nodes have proxies.
NodeCollection::const_iterator nc_begin =
node_collection->has_proxies() ? node_collection->MPI_local_begin() : node_collection->begin();
node_collection->has_proxies() ? node_collection->rank_local_begin() : node_collection->begin();
NodeCollection::const_iterator nc_end = node_collection->end();

// Reserve capacity in the vector based on number of local nodes. If the NodeCollection is sliced,
@@ -299,7 +330,7 @@ FreeLayer< D >::communicate_positions_( Ins iter, NodeCollectionPTR node_collect
// Push node ID into array to communicate
local_node_id_pos.push_back( ( *nc_it ).node_id );
// Push coordinates one by one
const auto pos = get_position( ( *nc_it ).lid );
const auto pos = get_position( ( *nc_it ).nc_index );
for ( int j = 0; j < D; ++j )
{
local_node_id_pos.push_back( pos[ j ] );
18 changes: 9 additions & 9 deletions nestkernel/grid_layer.h
Original file line number Diff line number Diff line change
@@ -122,12 +122,12 @@ class GridLayer : public Layer< D >
* @param sind index of node
* @returns position of node.
*/
Position< D > get_position( size_t sind ) const;
Position< D > get_position( size_t sind ) const override;

/**
* Get position of node. Also allowed for non-local nodes.
*
* @param lid local index of node
* @param lid global index of node within layer
* @returns position of node.
*/
Position< D > lid_to_position( size_t lid ) const;
@@ -148,17 +148,17 @@ class GridLayer : public Layer< D >

Position< D, size_t > get_dims() const;

void set_status( const DictionaryDatum& d );
void get_status( DictionaryDatum& d ) const;
void set_status( const DictionaryDatum& d ) override;
void get_status( DictionaryDatum& d, NodeCollection const* ) const override;

protected:
Position< D, size_t > dims_; ///< number of nodes in each direction.

template < class Ins >
void insert_global_positions_( Ins iter, NodeCollectionPTR node_collection );
void insert_global_positions_ntree_( Ntree< D, size_t >& tree, NodeCollectionPTR node_collection );
void insert_global_positions_ntree_( Ntree< D, size_t >& tree, NodeCollectionPTR node_collection ) override;
void insert_global_positions_vector_( std::vector< std::pair< Position< D >, size_t > >& vec,
NodeCollectionPTR node_collection );
NodeCollectionPTR node_collection ) override;
};

template < int D >
@@ -206,9 +206,9 @@ GridLayer< D >::set_status( const DictionaryDatum& d )

template < int D >
void
GridLayer< D >::get_status( DictionaryDatum& d ) const
GridLayer< D >::get_status( DictionaryDatum& d, NodeCollection const* nc ) const
{
Layer< D >::get_status( d );
Layer< D >::get_status( d, nc );

( *d )[ names::shape ] = std::vector< size_t >( dims_.get_vector() );
}
@@ -286,7 +286,7 @@ GridLayer< D >::insert_global_positions_( Ins iter, NodeCollectionPTR node_colle
for ( auto gi = node_collection->begin(); gi < node_collection->end(); ++gi )
{
const auto triple = *gi;
*iter++ = std::pair< Position< D >, size_t >( lid_to_position( triple.lid ), triple.node_id );
*iter++ = std::pair< Position< D >, size_t >( lid_to_position( triple.nc_index ), triple.node_id );
}
}

17 changes: 7 additions & 10 deletions nestkernel/layer.h
Original file line number Diff line number Diff line change
@@ -69,17 +69,19 @@ class AbstractLayer

/**
* Export properties of the layer by setting
* entries in the status dictionary.
* entries in the status dictionary, respects slicing of given NodeCollection
* @param d Dictionary.
*
* @note If nullptr is passed for NodeCollection*, full metadata irrespective of any slicing is returned.
*/
virtual void get_status( DictionaryDatum& ) const = 0;
virtual void get_status( DictionaryDatum&, NodeCollection const* ) const = 0;

virtual unsigned int get_num_dimensions() const = 0;

/**
* Get position of node. Only possible for local nodes.
*
* @param lid index of node within layer
* @param lid global index of node within layer
* @returns position of node as std::vector
*/
virtual std::vector< double > get_position_vector( const size_t lid ) const = 0;
@@ -234,13 +236,8 @@ class Layer : public AbstractLayer
*/
void set_status( const DictionaryDatum& ) override;

/**
* Export properties of the layer by setting
* entries in the status dictionary.
*
* @param d Dictionary.
*/
void get_status( DictionaryDatum& ) const override;
//! Retrieve status, slice according to node collection if given
void get_status( DictionaryDatum&, NodeCollection const* ) const override;

unsigned int
get_num_dimensions() const override
15 changes: 11 additions & 4 deletions nestkernel/layer_impl.h
Original file line number Diff line number Diff line change
@@ -92,7 +92,7 @@ Layer< D >::set_status( const DictionaryDatum& d )

template < int D >
void
Layer< D >::get_status( DictionaryDatum& d ) const
Layer< D >::get_status( DictionaryDatum& d, NodeCollection const* nc ) const
{
( *d )[ names::extent ] = std::vector< double >( extent_.get_vector() );
( *d )[ names::center ] = std::vector< double >( ( lower_left_ + extent_ / 2 ).get_vector() );
@@ -105,6 +105,13 @@ Layer< D >::get_status( DictionaryDatum& d ) const
{
( *d )[ names::edge_wrap ] = true;
}

if ( nc )
{
// This is for backward compatibility with some tests and scripts
// TODO: Rename parameter
( *d )[ names::network_size ] = nc->size();
}
}

template < int D >
@@ -286,12 +293,12 @@ template < int D >
void
Layer< D >::dump_nodes( std::ostream& out ) const
{
for ( NodeCollection::const_iterator it = this->node_collection_->MPI_local_begin();
for ( NodeCollection::const_iterator it = this->node_collection_->rank_local_begin();
it < this->node_collection_->end();
++it )
{
out << ( *it ).node_id << ' ';
get_position( ( *it ).lid ).print( out );
get_position( ( *it ).nc_index ).print( out );
out << std::endl;
}
}
@@ -345,7 +352,7 @@ Layer< D >::dump_connections( std::ostream& out,
Layer< D >* tgt_layer = dynamic_cast< Layer< D >* >( target_layer.get() );

out << ' ';
const long tnode_lid = tgt_layer->node_collection_->get_lid( target_node_id );
const long tnode_lid = tgt_layer->node_collection_->get_nc_index( target_node_id );
assert( tnode_lid >= 0 );
tgt_layer->compute_displacement( source_pos, tnode_lid ).print( out );
out << '\n';
27 changes: 0 additions & 27 deletions nestkernel/nest.cpp
Original file line number Diff line number Diff line change
@@ -416,31 +416,4 @@ node_collection_array_index( const Datum* datum, const bool* array, unsigned lon
return new NodeCollectionDatum( NodeCollection::create( node_ids ) );
}

void
slice_positions_if_sliced_nc( DictionaryDatum& dict, const NodeCollectionDatum& nc )
{
// If metadata contains node positions and the NodeCollection is sliced, get only positions of the sliced nodes.
if ( dict->known( names::positions ) )
{
const auto positions = getValue< TokenArray >( dict, names::positions );
if ( nc->size() != positions.size() )
{
TokenArray sliced_points;
// Iterate only local nodes
NodeCollection::const_iterator nc_begin = nc->has_proxies() ? nc->MPI_local_begin() : nc->begin();
NodeCollection::const_iterator nc_end = nc->end();
for ( auto node = nc_begin; node < nc_end; ++node )
{
// Because the local ID also includes non-local nodes, it must be adapted to represent
// the index for the local node position.
const auto index =
static_cast< size_t >( std::floor( ( *node ).lid / kernel().mpi_manager.get_num_processes() ) );
sliced_points.push_back( positions[ index ] );
}
def2< TokenArray, ArrayDatum >( dict, names::positions, sliced_points );
}
}
}


} // namespace nest
9 changes: 0 additions & 9 deletions nestkernel/nest.h
Original file line number Diff line number Diff line change
@@ -191,15 +191,6 @@ std::vector< double > apply( const ParameterDatum& param, const DictionaryDatum&
Datum* node_collection_array_index( const Datum* datum, const long* array, unsigned long n );
Datum* node_collection_array_index( const Datum* datum, const bool* array, unsigned long n );

/**
* @brief Get only positions of the sliced nodes if metadata contains node positions and the NodeCollection is sliced.
*
* Puts an array of positions sliced the same way as a sliced NodeCollection into dict.
* Positions have to be sliced on introspection because metadata of a sliced NodeCollection
* for internal consistency and efficiency points to the metadata of the original
* NodeCollection.
*/
void slice_positions_if_sliced_nc( DictionaryDatum& dict, const NodeCollectionDatum& nc );
}


28 changes: 11 additions & 17 deletions nestkernel/nestmodule.cpp
Original file line number Diff line number Diff line change
@@ -510,17 +510,8 @@ NestModule::GetMetadata_gFunction::execute( SLIInterpreter* i ) const
"InvalidNodeCollection: note that ResetKernel invalidates all previously created NodeCollections." );
}

NodeCollectionMetadataPTR meta = nc->get_metadata();
DictionaryDatum dict = DictionaryDatum( new Dictionary );

// return empty dict if NC does not have metadata
if ( meta.get() )
{
meta->get_status( dict );
slice_positions_if_sliced_nc( dict, nc );

( *dict )[ names::network_size ] = nc->size();
}
nc->get_metadata_status( dict );

i->OStack.pop();
i->OStack.push( dict );
@@ -1053,13 +1044,16 @@ NestModule::Cvnodecollection_ivFunction::execute( SLIInterpreter* i ) const
}

void
NestModule::Cva_gFunction::execute( SLIInterpreter* i ) const
NestModule::Cva_g_lFunction::execute( SLIInterpreter* i ) const
{
i->assert_stack_load( 1 );
NodeCollectionDatum nodecollection = getValue< NodeCollectionDatum >( i->OStack.pick( 0 ) );
ArrayDatum node_ids = nodecollection->to_array();
i->assert_stack_load( 2 );

i->OStack.pop();
const std::string selection = getValue< std::string >( i->OStack.pick( 0 ) );
NodeCollectionDatum nodecollection = getValue< NodeCollectionDatum >( i->OStack.pick( 1 ) );

ArrayDatum node_ids = nodecollection->to_array( selection );

i->OStack.pop( 2 );
i->OStack.push( node_ids );
i->EStack.pop();
}
@@ -1120,7 +1114,7 @@ NestModule::Find_g_iFunction::execute( SLIInterpreter* i ) const
NodeCollectionDatum nodecollection = getValue< NodeCollectionDatum >( i->OStack.pick( 1 ) );
const long node_id = getValue< long >( i->OStack.pick( 0 ) );

const auto res = nodecollection->get_lid( node_id );
const auto res = nodecollection->get_nc_index( node_id );
i->OStack.pop( 2 );
i->OStack.push( res );
i->EStack.pop();
@@ -2160,7 +2154,7 @@ NestModule::init( SLIInterpreter* i )
i->createcommand( "cvnodecollection_i_i", &cvnodecollection_i_ifunction );
i->createcommand( "cvnodecollection_ia", &cvnodecollection_iafunction );
i->createcommand( "cvnodecollection_iv", &cvnodecollection_ivfunction );
i->createcommand( "cva_g", &cva_gfunction );
i->createcommand( "cva_g_l", &cva_g_lfunction );
i->createcommand( "size_g", &size_gfunction );
i->createcommand( "ValidQ_g", &validq_gfunction );
i->createcommand( "join_g_g", &join_g_gfunction );
33 changes: 2 additions & 31 deletions nestkernel/nestmodule.h
Original file line number Diff line number Diff line change
@@ -847,10 +847,10 @@ class NestModule : public SLIModule
void execute( SLIInterpreter* ) const override;
} cvnodecollection_ivfunction;

class Cva_gFunction : public SLIFunction
class Cva_g_lFunction : public SLIFunction
{
void execute( SLIInterpreter* ) const override;
} cva_gfunction;
} cva_g_lfunction;

class Size_gFunction : public SLIFunction
{
@@ -1468,35 +1468,6 @@ class NestModule : public SLIModule
* linear, exponential and other.
*
*
* Parameter name: source
*
* Type: dictionary
*
* Parameter description:
*
* The source dictionary enables us to give further detail on
* how the nodes in the source layer used in the connection function
* should be processed.
*
* Parameters:
* model* literal
* lid^ integer
*
* *modeltype (i.e. /iaf_psc_alpha) of nodes that should be connected to
* in the layer. All nodes are used if this variable isn't set.
* ^Nesting depth of nodes that should be connected to. All layers are used
* if this variable isn't set.
*
*
* Parameter name: target
*
* Type: dictionary
*
* Parameter description:
*
* See description for source dictionary.
*
*
* Parameter name: number_of_connections
*
* Type: integer
883 changes: 602 additions & 281 deletions nestkernel/node_collection.cpp

Large diffs are not rendered by default.

738 changes: 628 additions & 110 deletions nestkernel/node_collection.h

Large diffs are not rendered by default.

11 changes: 7 additions & 4 deletions nestkernel/spatial.h
Original file line number Diff line number Diff line change
@@ -60,9 +60,9 @@ class LayerMetadata : public NodeCollectionMetadata
void set_status( const DictionaryDatum&, bool ) override {};

void
get_status( DictionaryDatum& d ) const override
get_status( DictionaryDatum& d, NodeCollection const* nc ) const override
{
layer_->get_status( d );
layer_->get_status( d, nc );
}

//! Returns pointer to object with layer representation
@@ -102,8 +102,11 @@ class LayerMetadata : public NodeCollectionMetadata
// Compare status dictionaries of this layer and rhs layer
DictionaryDatum dict( new Dictionary() );
DictionaryDatum rhs_dict( new Dictionary() );
get_status( dict );
rhs_layer_metadata->get_status( rhs_dict );

// Since we do not have access to the node collection here, we
// compare based on all metadata, irrespective of any slicing
get_status( dict, /* nc */ nullptr );
rhs_layer_metadata->get_status( rhs_dict, /* nc */ nullptr );
return *dict == *rhs_dict;
}

1,441 changes: 0 additions & 1,441 deletions pynest/examples/brunel-py-ex-12502-0.gdf

This file was deleted.

1,422 changes: 0 additions & 1,422 deletions pynest/examples/brunel-py-in-12503-0.gdf

This file was deleted.

35 changes: 35 additions & 0 deletions pynest/nest/lib/hl_api_types.py
Original file line number Diff line number Diff line change
@@ -39,6 +39,7 @@
is_literal,
restructure_data,
)
from .hl_api_parallel_computing import Rank
from .hl_api_simulation import GetKernelStatus

try:
@@ -514,6 +515,40 @@ def tolist(self):

return list(self.get("global_id")) if len(self) > 1 else [self.get("global_id")]

def _to_array(self, selection="all"):
"""
Debugging helper to extract GIDs from node collections.
`selection` can be `"all"`, `"rank"` or `"thread"` and extracts either all
nodes or those on the rank or thread on which it is executed. For `"thread"`,
separate lists are returned for all local threads independently.
"""

res = sli_func("cva_g_l", self, selection)

if selection == "all":
return {"All": res}
elif selection == "rank":
return {f"Rank {Rank()}": res}
elif selection == "thread":
t_res = {}
thr = None
ix = 0
while ix < len(res):
while ix < len(res) and res[ix] != 0:
t_res[thr].append(res[ix])
ix += 1
assert ix == len(res) or ix + 3 <= len(res)
if ix < len(res):
assert res[ix] == 0 and res[ix + 2] == 0
thr = res[ix + 1]
assert thr not in t_res
t_res[thr] = []
ix += 3
return t_res
else:
return res

def index(self, node_id):
"""
Find the index of a node ID in the `NodeCollection`.
6 changes: 3 additions & 3 deletions pynest/nest/raster_plot.py
Original file line number Diff line number Diff line change
@@ -55,17 +55,16 @@ def extract_events(data, time=None, sel=None):
"""
val = []

t_min, t_max = 0, float("inf")
if time:
t_max = time[-1]
if len(time) > 1:
t_min = time[0]
else:
t_min = 0

for v in data:
t = v[1]
node_id = v[0]
if time and (t < t_min or t >= t_max):
if not (t_min <= t < t_max):
continue
if not sel or node_id in sel:
val.append(v)
@@ -131,6 +130,7 @@ def from_file_pandas(fname, **kwargs):
"""Use pandas."""
data = None
for f in fname:
# pylint: disable=possibly-used-before-assignment
dataFrame = pandas.read_table(f, header=2, skipinitialspace=True)
newdata = dataFrame.values

19 changes: 19 additions & 0 deletions sli/tokenarray.h
Original file line number Diff line number Diff line change
@@ -262,13 +262,32 @@ class TokenArray
}

// Insertion, deletion

/**
* Insert element at end.
*
* @note Calling with literal value can lead to undefined behavior. The following seems safe:
*
* TokenArray ta;
* const size_t zero = 0;
* ta.push_back( zero );
*/
void
push_back( const Token& t )
{
clone();
data->push_back( t );
}

/**
* Insert element at end.
*
* @note Calling with literal value can lead to undefined behavior. The following seems safe:
*
* TokenArray ta;
* const size_t zero = 0;
* ta.push_back( zero );
*/
void
push_back( Datum* d )
{
10 changes: 9 additions & 1 deletion testsuite/do_tests.sh
Original file line number Diff line number Diff line change
@@ -520,8 +520,16 @@ if test "${PYTHON}"; then
for numproc in $(cd ${PYNEST_TEST_DIR}/mpi/; ls -d */ | tr -d '/'); do
XUNIT_FILE="${REPORTDIR}/${XUNIT_NAME}_mpi_${numproc}.xml"
PYTEST_ARGS="--verbose --timeout $TIME_LIMIT --junit-xml=${XUNIT_FILE} ${PYNEST_TEST_DIR}/mpi/${numproc}"

if "${DO_TESTS_SKIP_TEST_REQUIRING_MANY_CORES:-false}"; then
PYTEST_ARGS="${PYTEST_ARGS} -m 'not requires_many_cores'"
fi

set +e
$(sli -c "${numproc} (${PYTHON} -m pytest) (${PYTEST_ARGS}) mpirun =only") 2>&1 | tee -a "${TEST_LOGFILE}"

# We need to use eval here because $() splits run_command in weird ways
run_command="$(sli -c "${numproc} (${PYTHON} -m pytest) (${PYTEST_ARGS}) mpirun =only")"
eval "${run_command}" 2>&1 | tee -a "${TEST_LOGFILE}"
set -e
done
fi
7 changes: 1 addition & 6 deletions testsuite/mpitests/test_spatial_distributed_positions.sli
Original file line number Diff line number Diff line change
@@ -30,18 +30,13 @@ skip_if_not_threaded
ResetKernel
<< /total_num_virtual_procs 4 >> SetKernelStatus

<< /uniform << /min 0.0 /max 1.0 >> >> CreateParameter /pos_param Set
pos_param pos_param dimension2d /pos Set

/layer_spec
<< /positions pos
/n 4
<< /positions [ 1 4 ] { 0 2 arraystore } Table
/edge_wrap false
/elements /iaf_psc_alpha
>> def

/layer layer_spec CreateLayer def

layer GetMetadata /meta Set

{
12 changes: 12 additions & 0 deletions testsuite/pytests/conftest.py
Original file line number Diff line number Diff line change
@@ -47,6 +47,18 @@ def test_gsl():
import testutil # noqa


def pytest_configure(config):
"""
Add NEST-specific markers.
See https://docs.pytest.org/en/8.0.x/how-to/mark.html.
"""
config.addinivalue_line(
"markers",
"requires_many_cores: mark tests as needing many cores (deselect with '-m \"not requires_many_cores\"')",
)


@pytest.fixture(scope="module", autouse=True)
def safety_reset():
"""
3 changes: 3 additions & 0 deletions testsuite/pytests/connect_test_base.py
Original file line number Diff line number Diff line change
@@ -451,6 +451,9 @@ def get_degrees(fan, pop1, pop2):
degrees = np.sum(M, axis=1)
elif fan == "out":
degrees = np.sum(M, axis=0)
else:
raise ValueError(f"fan must be 'in' or 'out', got '{fan}'.")

return degrees


1 change: 1 addition & 0 deletions testsuite/pytests/mpi/2/test_issue_3108.py
1 change: 1 addition & 0 deletions testsuite/pytests/mpi/3/test_issue_3108.py
1 change: 1 addition & 0 deletions testsuite/pytests/mpi/4/test_issue_3108.py
2 changes: 1 addition & 1 deletion testsuite/pytests/sli2py_mpi/README.md
Original file line number Diff line number Diff line change
@@ -2,4 +2,4 @@

Test in this directory run NEST with different numbers of MPI ranks and compare results.

See documentation in mpi_test_wrappe.py for details.
See documentation in mpi_test_wrapper.py for details.
7 changes: 5 additions & 2 deletions testsuite/pytests/sli2py_mpi/mpi_test_wrapper.py
Original file line number Diff line number Diff line change
@@ -139,8 +139,11 @@ def _params_as_str(self, *args, **kwargs):
return ", ".join(
part
for part in (
", ".join(f"{arg}" for arg in args),
", ".join(f"{key}={value}" for key, value in kwargs.items()),
", ".join(f"{arg if not inspect.isfunction(arg) else arg.__name__}" for arg in args),
", ".join(
f"{key}={value if not inspect.isfunction(value) else value.__name__}"
for key, value in kwargs.items()
),
)
if part
)
7 changes: 7 additions & 0 deletions testsuite/pytests/test_clopath_synapse.py
Original file line number Diff line number Diff line change
@@ -127,6 +127,9 @@ def test_SynapseDepressionFacilitation(self):
"tau_u_bar_plus": 114.0,
"delay_u_bars": 5.0,
}
else:
raise ValueError(f"Unsupported neuron model '{nrn_model}'")

syn_weights = []
# Loop over pairs of spike trains
for s_t_pre, s_t_post in zip(spike_times_pre, spike_times_post):
@@ -146,6 +149,8 @@ def test_SynapseDepressionFacilitation(self):
conn_weight = 80.0
elif nrn_model == "hh_psc_alpha_clopath":
conn_weight = 2000.0
else:
raise ValueError(f"Unsupported neuron model '{nrn_model}'")

spike_gen_post = nest.Create("spike_generator", 1, {"spike_times": s_t_post})

@@ -176,6 +181,8 @@ def test_SynapseDepressionFacilitation(self):
correct_weights = [57.82638722, 72.16730112, 149.43359357, 103.30408341, 124.03640668, 157.02882555]
elif nrn_model == "hh_psc_alpha_clopath":
correct_weights = [70.14343863, 99.49206222, 178.1028757, 119.63314118, 167.37750688, 178.83111685]
else:
raise ValueError(f"Unsupported neuron model '{nrn_model}'")

self.assertTrue(np.allclose(syn_weights, correct_weights, rtol=1e-7))

3 changes: 3 additions & 0 deletions testsuite/pytests/test_connect_tripartite_bernoulli.py
Original file line number Diff line number Diff line change
@@ -177,6 +177,9 @@ def get_degrees(fan, pop1, pop2):
degrees = np.sum(M, axis=1)
elif fan == "out":
degrees = np.sum(M, axis=0)
else:
raise ValueError(f"fan must be 'in' or 'out', got '{fan}'.")

return degrees


14 changes: 14 additions & 0 deletions testsuite/pytests/test_get_connections.py
Original file line number Diff line number Diff line change
@@ -84,6 +84,20 @@ def test_get_connections_with_sliced_node_collection():
assert actual_sources == expected_sources


def test_get_connections_with_sliced_node_collection_2():
"""Test that ``GetConnections`` works with sliced ``NodeCollection``."""

nodes = nest.Create("iaf_psc_alpha", 11)
nest.Connect(nodes, nodes)

# ([ 2 3 4 ] + [ 8 9 10 11 ])[::3] -> [2 8 11]
conns = nest.GetConnections((nodes[1:4] + nodes[7:])[::3])
actual_sources = conns.get("source")

expected_sources = [2] * 11 + [8] * 11 + [11] * 11
assert actual_sources == expected_sources


def test_get_connections_bad_source_raises():
"""Test that ``GetConnections`` raises an error when called with 0."""

48 changes: 48 additions & 0 deletions testsuite/pytests/test_issue_3106.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# -*- coding: utf-8 -*-
#
# test_issue_3106.py
#
# This file is part of NEST.
#
# Copyright (C) 2004 The NEST Initiative
#
# NEST is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 2 of the License, or
# (at your option) any later version.
#
# NEST is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with NEST. If not, see <http://www.gnu.org/licenses/>.


import nest
import pytest


@pytest.mark.skipif_missing_threads
def test_connect_with_threads_slice_and_mpi():
"""
Test that connection with sliced layer is possible on multiple threads.
"""

num_neurons = 10
nest.local_num_threads = 4

layer = nest.Create(
model="parrot_neuron",
n=num_neurons,
positions=nest.spatial.free(pos=nest.random.uniform(min=-1, max=1), num_dimensions=2, edge_wrap=False),
)

tgts = layer[::3]

# Distance-dependent weight forces use of layer-connect
nest.Connect(layer, tgts, {"rule": "pairwise_bernoulli", "p": 1}, {"weight": nest.spatial.distance})
# nest.Connect(layer, tgts, {"rule": "fixed_indegree", "indegree": 5})

assert nest.num_connections == len(layer) * len(tgts)
298 changes: 298 additions & 0 deletions testsuite/pytests/test_issue_3108.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,298 @@
# -*- coding: utf-8 -*-
#
# test_issue_3108.py
#
# This file is part of NEST.
#
# Copyright (C) 2004 The NEST Initiative
#
# NEST is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 2 of the License, or
# (at your option) any later version.
#
# NEST is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with NEST. If not, see <http://www.gnu.org/licenses/>.


import itertools

import nest
import pytest

"""
Test in this file were developed for regressions under three MPI processes.
They should be run with 1, 3 and 4 MPI processes to ensure all passes under various settings.
The spatial tests test that NodeCollection::rank_local_begin() works.
The connect tests test that NodeCollection::thread_local_begin() works.
"""

# Needs up to 16 cores when run on 4 MPI ranks.
# Experiences severe slowdown on Github runners under Linux with MPI and OpenMP
pytestmark = pytest.mark.requires_many_cores

if nest.ll_api.sli_func("is_threaded"):
num_threads = [1, 2, 3, 4]
else:
num_threads = [1]


@pytest.mark.parametrize("n_threads", num_threads)
@pytest.mark.parametrize(
"transform",
[
lambda nc: nc[::3],
lambda nc: nc[1:],
lambda nc: nc[:5] + nc[8:],
lambda nc: (nc[:5] + nc[8:])[-1:],
lambda nc: (nc[:5] + nc[8:])[::2],
lambda nc: (nc[:5] + nc[8:])[::3],
lambda nc: (nc[:5] + nc[9:])[::2],
lambda nc: (nc[:5] + nc[9:])[::3],
lambda nc: (nc[:5] + nc[8:])[7:],
lambda nc: (nc[:5] + nc[8:])[7::3],
],
)
def test_slice_node_collections(n_threads, transform):
nest.ResetKernel()
nest.local_num_threads = n_threads

n_orig = nest.Create("parrot_neuron", 128)
n_orig_gids = n_orig._to_array()["All"]

n_sliced = transform(n_orig)
n_pyslice_gids = transform(n_orig_gids)

n_pyslice_gids_on_rank = sorted(
n.global_id for n in nest.NodeCollection(n_pyslice_gids) if n.vp % nest.NumProcesses() == nest.Rank()
)

assert n_sliced._to_array()["All"] == n_pyslice_gids

n_sliced_gids_by_rank = n_sliced._to_array("rank")
assert len(n_sliced_gids_by_rank) == 1
assert sorted(next(iter(n_sliced_gids_by_rank.values()))) == n_pyslice_gids_on_rank

n_sliced_gids_by_thread = n_sliced._to_array("thread")
assert sorted(itertools.chain(*n_sliced_gids_by_thread.values())) == n_pyslice_gids_on_rank

for thread, tgids in n_sliced_gids_by_thread.items():
for node in nest.NodeCollection(tgids):
assert node.thread == thread


@pytest.mark.parametrize("n_threads", num_threads)
@pytest.mark.parametrize("stride", [1, 2, 3, 7, 12])
def test_slice_single_element_parts(n_threads, stride):
"""Same test as above, but on NC with single-element parts to check stepping over parts"""

nest.ResetKernel()
nest.local_num_threads = n_threads

# Use different models to get single-element node collections
mod_names = [f"pn{n:02d}" for n in range(91)]
for mod in mod_names:
nest.CopyModel("parrot_neuron", mod)

n_orig = sum(nest.Create(mod) for mod in mod_names)
n_orig_gids = n_orig._to_array()["All"]

n_sliced = n_orig[::stride]
n_pyslice_gids = n_orig_gids[::stride]

n_pyslice_gids_on_rank = sorted(
(n.global_id for n in nest.NodeCollection(n_pyslice_gids) if n.vp % nest.NumProcesses() == nest.Rank())
)

assert n_sliced._to_array()["All"] == n_pyslice_gids

n_sliced_gids_by_rank = n_sliced._to_array("rank")
assert len(n_sliced_gids_by_rank) == 1
assert sorted(next(iter(n_sliced_gids_by_rank.values()))) == n_pyslice_gids_on_rank

n_sliced_gids_by_thread = n_sliced._to_array("thread")
assert sorted(itertools.chain(*n_sliced_gids_by_thread.values())) == n_pyslice_gids_on_rank

for thread, tgids in n_sliced_gids_by_thread.items():
for node in nest.NodeCollection(tgids):
assert node.thread == thread


@pytest.mark.parametrize("n_threads", num_threads)
@pytest.mark.parametrize("stride", [1, 2, 3, 7, 12])
def test_multi_parts_slicing(n_threads, stride):
"""Same test as above, but on NC from parts of different size with different gaps"""

nest.ResetKernel()
nest.local_num_threads = n_threads

n_orig = nest.Create("parrot_neuron", 97)
n_orig_gids = n_orig._to_array()["All"]

n_sliced = n_orig[:5] + n_orig[6:7] + n_orig[8:18] + n_orig[23:34] + n_orig[41:]
n_pyslice_gids = n_orig_gids[:5] + n_orig_gids[6:7] + n_orig_gids[8:18] + n_orig_gids[23:34] + n_orig_gids[41:]

n_pyslice_gids_on_rank = sorted(
(n.global_id for n in nest.NodeCollection(n_pyslice_gids) if n.vp % nest.NumProcesses() == nest.Rank())
)

assert n_sliced._to_array()["All"] == n_pyslice_gids

n_sliced_gids_by_rank = n_sliced._to_array("rank")
assert len(n_sliced_gids_by_rank) == 1
assert sorted(next(iter(n_sliced_gids_by_rank.values()))) == n_pyslice_gids_on_rank

n_sliced_gids_by_thread = n_sliced._to_array("thread")
assert sorted(itertools.chain(*n_sliced_gids_by_thread.values())) == n_pyslice_gids_on_rank

for thread, tgids in n_sliced_gids_by_thread.items():
for node in nest.NodeCollection(tgids):
assert node.thread == thread


@pytest.mark.parametrize("n_threads", num_threads)
@pytest.mark.parametrize("start, step", ([[0, 1], [0, 3]] + [[2, n] for n in range(1, 9)]))
def test_get_positions_with_mpi(n_threads, start, step):
"""
Test that correct positions can be obtained from sliced node collections.
Two cases above for starting without offset, the remaining with a small offset.
With the range of step values, combined with 3 and 4 MPI processes, we ensure
that we have cases where the step is half of or a multiple of the number of
processes.
"""

num_neurons = 128

nest.ResetKernel()
nest.local_num_threads = n_threads

# Need floats because NEST returns positions as floats
node_pos = [(float(x), 0.0) for x in range(num_neurons)]

layer = nest.Create(
model="parrot_neuron",
n=num_neurons,
positions=nest.spatial.free(pos=node_pos, edge_wrap=False),
)

pos = layer[start::step].spatial["positions"]
node_ranks = [n.vp % nest.NumProcesses() for n in layer]
assert len(node_ranks) == num_neurons

# pos is a tuple of tuples, so we need to create a tuple for comparison
expected_pos = tuple(
npos for npos, nrk in zip(node_pos[start::step], node_ranks[start::step]) if nrk == nest.Rank()
)

assert pos == expected_pos


@pytest.mark.parametrize("n_threads", num_threads)
@pytest.mark.parametrize("pick", range(0, 7))
def test_get_spatial_for_single_element_and_mpi(n_threads, pick):
"""
Test that spatial information can be collected from a single layer element.
This was an original minimal reproducer for #3108.
"""

num_neurons = 7

nest.ResetKernel()
nest.local_num_threads = n_threads

node_pos = [(float(x), 0.0) for x in range(num_neurons)]

layer = nest.Create(
model="parrot_neuron",
n=num_neurons,
positions=nest.spatial.free(pos=node_pos, edge_wrap=False),
)

# We want to retrieve this on all ranks to see that it does not break NEST
sp = layer[pick].spatial["positions"]

pick_rank = layer[pick].vp % nest.NumProcesses()
if pick_rank == nest.Rank():
assert sp[0] == node_pos[pick]
else:
assert len(sp) == 0


@pytest.mark.parametrize("n_threads", num_threads)
@pytest.mark.parametrize("pick", range(0, 5))
def test_connect_with_single_element_slice_and_mpi(n_threads, pick):
"""
Test that connection with single-element sliced layer is possible on multiple mpi processes.
This was an original minimal reproducer for #3108.
"""

num_neurons = 5

nest.ResetKernel()
nest.local_num_threads = n_threads

layer = nest.Create(
model="parrot_neuron",
n=num_neurons,
positions=nest.spatial.free(pos=nest.random.uniform(min=-1, max=1), num_dimensions=2, edge_wrap=False),
)

# space-dependent syn_spec passed only to force use of ConnectLayers
nest.Connect(layer[pick], layer, {"rule": "pairwise_bernoulli", "p": 1.0}, {"weight": nest.spatial.distance})

local_nodes = tuple(n.global_id for n in layer if n.vp % nest.NumProcesses() == nest.Rank())

c = nest.GetConnections()
src = tuple(c.sources())
tgt = tuple(c.targets())
assert src == (layer[pick].global_id,) * len(local_nodes)
assert sorted(tgt) == sorted(local_nodes)


@pytest.mark.parametrize("n_threads", num_threads)
@pytest.mark.parametrize("sstep", [2, 3, 4, 6])
@pytest.mark.parametrize("tstep", [2, 3, 4, 6])
def test_connect_slice_to_slice_and_mpi(n_threads, sstep, tstep):
"""
Test that connection with stepped source and target layers is possible on multiple mpi processes.
This was an original minimal reproducer for #3108.
"""

num_neurons = 128

nest.ResetKernel()
nest.local_num_threads = n_threads

layer = nest.Create(
model="parrot_neuron",
n=num_neurons,
positions=nest.spatial.free(pos=nest.random.uniform(min=-1, max=1), num_dimensions=2, edge_wrap=False),
)

# space-dependent syn_spec passed only to force use of ConnectLayers
nest.Connect(
layer[::sstep], layer[2::tstep], {"rule": "pairwise_bernoulli", "p": 1.0}, {"weight": nest.spatial.distance}
)

local_nodes = tuple(n.global_id for n in layer if n.vp % nest.NumProcesses() == nest.Rank())
local_targets = set(n.global_id for n in layer[2::tstep] if n.vp % nest.NumProcesses() == nest.Rank())

c = nest.GetConnections()
src = tuple(c.sources())
tgt = tuple(c.targets())

assert len(c) == len(layer[::sstep]) * len(local_targets)
assert set(tgt) == local_targets # all local neurons in layer[2::tstep] must be targets
if local_targets:
assert set(src) == set(layer[::sstep].global_id) # all neurons in layer[::sstep] neurons must be sources
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# -*- coding: utf-8 -*-
#
# test_nodeParametrization.py
# test_node_parametrization.py
#
# This file is part of NEST.
#
38 changes: 38 additions & 0 deletions testsuite/pytests/test_regression_issue-3213.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# -*- coding: utf-8 -*-
#
# test_regression_issue-3213.py
#
# This file is part of NEST.
#
# Copyright (C) 2004 The NEST Initiative
#
# NEST is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 2 of the License, or
# (at your option) any later version.
#
# NEST is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with NEST. If not, see <http://www.gnu.org/licenses/>.

import nest
import pytest

"""
Test that GetConnections works if NodeCollection with gaps is provided as source arg.
"""


def test_get_conns_works():
"""Main concern is that GetConnections() passes, expected number of connections based on all-to-all."""

num_n = 12
n = nest.Create("parrot_neuron", num_n)
nest.Connect(n, n)
pick = [3, 7, 9, 11]
conns = nest.GetConnections(source=n[pick])
assert len(conns) == num_n * len(pick)
14 changes: 8 additions & 6 deletions testsuite/pytests/test_spatial/test_connect_layers.py
Original file line number Diff line number Diff line change
@@ -154,7 +154,7 @@ def _assert_connect_layers_multapses(self, multapses):
else:
self.assertEqual(num_nonunique_conns, 0)

def _assert_connect_sliced(self, pre, post):
def _assert_connect_sliced(self, pre, post, kind):
"""Helper function which asserts that connecting with ConnectLayers on the SLI level
gives the expected number of connections."""
# Using distance based probability with zero weight to
@@ -165,7 +165,9 @@ def _assert_connect_sliced(self, pre, post):

nest.Connect(pre, post, conn_spec)
conns = nest.GetConnections()
result = "{} ({}), pre length={}, post length={}".format(len(conns), expected_conns, len(pre), len(post))
result = "{} ({}), pre length={}, post length={} (kind {})".format(
len(conns), expected_conns, len(pre), len(post), kind
)
print(result)
self.assertEqual(len(conns), expected_conns, "pre length={}, post length={}".format(len(pre), len(post)))

@@ -422,12 +424,12 @@ def test_connect_sliced_grid_layer(self):
layers = self._reset_and_create_sliced(positions)
layer = layers["layer"]
sliced_pre = layers[sliced]
self._assert_connect_sliced(sliced_pre, layer)
self._assert_connect_sliced(sliced_pre, layer, f"{sliced} pre")
for sliced in ["single", "range", "step"]:
layers = self._reset_and_create_sliced(positions)
layer = layers["layer"]
sliced_post = layers[sliced]
self._assert_connect_sliced(layer, sliced_post)
self._assert_connect_sliced(layer, sliced_post, f"{sliced} post")

def test_connect_sliced_free_layer(self):
"""Connecting with sliced free layer"""
@@ -436,12 +438,12 @@ def test_connect_sliced_free_layer(self):
layers = self._reset_and_create_sliced(positions)
layer = layers["layer"]
sliced_pre = layers[sliced]
self._assert_connect_sliced(sliced_pre, layer)
self._assert_connect_sliced(sliced_pre, layer, f"{sliced} pre")
for sliced in ["single", "range", "step"]:
layers = self._reset_and_create_sliced(positions)
layer = layers["layer"]
sliced_post = layers[sliced]
self._assert_connect_sliced(layer, sliced_post)
self._assert_connect_sliced(layer, sliced_post, f"{sliced} post")

def test_connect_synapse_label(self):
indegree = 10
22 changes: 10 additions & 12 deletions testsuite/pytests/test_stdp_nn_synapses.py
Original file line number Diff line number Diff line change
@@ -187,18 +187,16 @@ def reproduce_weight_drift(self, _pre_spikes, _post_spikes, _initial_weight):
t = _pre_spikes[pre_spikes_forced_to_grid.index(time_in_simulation_steps)]

# Evaluating the depression rule.
if self.synapse_parameters["synapse_model"] == "stdp_nn_restr_synapse":
current_nearest_neighbour_pair_is_suitable = t_previous_post > t_previous_pre
# If '<', t_previous_post has already been paired
# with t_previous_pre, thus due to the restricted
# pairing scheme we do not account it.
if (
self.synapse_parameters["synapse_model"] == "stdp_nn_symm_synapse"
or self.synapse_parameters["synapse_model"] == "stdp_nn_pre_centered_synapse"
):
# The current pre-spike is simply paired with the
# nearest post-spike.
current_nearest_neighbour_pair_is_suitable = True
# For the first two rules below, simply pair current pre-spike with nearest post-spike.
# For "nn_restr" and `post < pre`, the previous post has already been paired, thus due
# to the restricted pairing scheme, we do not account it.
current_nearest_neighbour_pair_is_suitable = self.synapse_parameters["synapse_model"] in [
"stdp_nn_symm_synapse",
"stdp_nn_pre_centered_synapse",
] or (
self.synapse_parameters["synapse_model"] == "stdp_nn_restr_synapse"
and t_previous_post > t_previous_pre
)

if current_nearest_neighbour_pair_is_suitable and t_previous_post != -1:
# Otherwise, if == -1, there have been
22 changes: 21 additions & 1 deletion testsuite/pytests/test_tripartite_connect.py
Original file line number Diff line number Diff line change
@@ -122,7 +122,7 @@ def test_block_pool_wide():
assert len(nest.GetConnections(third, post)) == n_primary


def test_bipartitet_raises():
def test_bipartite_raises():
n_pre, n_post, n_third = 4, 2, 8
pre = nest.Create("parrot_neuron", n_pre)
post = nest.Create("parrot_neuron", n_post)
@@ -132,6 +132,26 @@ def test_bipartitet_raises():
nest.TripartiteConnect(pre, post, third, {"rule": "one_to_one"})


@pytest.mark.skipif_missing_threads
def test_sliced_third():
"""Test that connection works on multiple threads when using complex node collection as third factor."""

nest.local_num_threads = 4
nrn = nest.Create("parrot_neuron", 20)
third = (nrn[:3] + nrn[5:])[::3]

nest.TripartiteConnect(
nrn, nrn, third, {"rule": "tripartite_bernoulli_with_pool", "pool_type": "random", "pool_size": 2}
)

t_in = nest.GetConnections(target=third)
t_out = nest.GetConnections(source=third)

assert len(t_in) == len(t_out)
assert set(t_in.targets()) <= set(third.global_id)
assert set(t_out.sources()) <= set(third.global_id)


def test_connect_complex_synspecs():
n_pre, n_post, n_third = 4, 2, 3
pre = nest.Create("parrot_neuron", n_pre)
27 changes: 18 additions & 9 deletions testsuite/summarize_tests.py
Original file line number Diff line number Diff line change
@@ -71,10 +71,15 @@ def parse_result_file(fname):

for pfile in sorted(glob.glob(os.path.join(test_outdir, "*.xml"))):
ph_name = os.path.splitext(os.path.split(pfile)[1])[0].replace("_", " ")
ph_res = parse_result_file(pfile)
results[ph_name] = ph_res
for k, v in ph_res.items():
totals[k] += v
try:
ph_res = parse_result_file(pfile)
results[ph_name] = ph_res
for k, v in ph_res.items():
totals[k] += v
except Exception as err:
msg = f"ERROR: {pfile} not parsable with error {err}"
results[ph_name] = {"Tests": 0, "Skipped": 0, "Failures": 0, "Errors": 0, "Time": 0, "Failed tests": [msg]}
totals["Failed tests"].append(msg)

cols = ["Tests", "Skipped", "Failures", "Errors", "Time"]

@@ -97,10 +102,13 @@ def parse_result_file(fname):
print(tline)
for pn, pr in results.items():
print(f"{pn:<{first_col_w}s}", end="")
for c in cols:
fmt = ".1f" if c == "Time" else "d"
print(f"{pr[c]:{col_w}{fmt}}", end="")
print()
if pr["Tests"] == 0 and pr["Failed tests"]:
print(f"{'--- XML PARSING FAILURE ---':^{len(cols) * col_w}}")
else:
for c in cols:
fmt = ".1f" if c == "Time" else "d"
print(f"{pr[c]:{col_w}{fmt}}", end="")
print()

print(tline)
print(f"{'Total':<{first_col_w}s}", end="")
@@ -111,7 +119,8 @@ def parse_result_file(fname):
print(tline)
print()

if totals["Failures"] + totals["Errors"] > 0:
# Second condition handles xml parsing failures
if totals["Failures"] + totals["Errors"] > 0 or totals["Failed tests"]:
print("THE NEST TESTSUITE DISCOVERED PROBLEMS")
print(" The following tests failed")
for t in totals["Failed tests"]:

0 comments on commit f5d5095

Please sign in to comment.