Copying array by ref from dictionary

[mias1wot]

When I allocate an array and copy it to another array by ref - everything works just fine. But if I add an array to a dictionary and try to copy it by reference, the result array will have incorrect boundaries and trying to do any action with the new variable will make the Excel crash.

Sub Works()
    Dim a() As Variant
    ReDim a(2, 2)
    a(0, 0) = 1
    
    Dim b As Variant
    b = GetArrayByRef(a)
    
    b(1, 1) = 4
End Sub

Sub ExcelCrashes()
    Dim a() As Variant
    ReDim a(2, 2)
    a(0, 0) = 1
    
    Dim dict As New Dictionary
    dict.Add 0, a
    
    
    Dim b As Variant
    b = GetArrayByRef(dict(0)) ' The memory is not allocated properly here.
    
    b(1, 1) = 4 ' Excel crash.
End Sub

[cristianbuse]

Hi @mias1wot

Where can I find the code for ArrayOperations.CopyArrayByRef. I know quite a lot about the topic because I've actually spent tens of hours to figure out a safe way to do this. The only safe way involves a side-allocation of a new SAFEARRAY with correct flags that points into the data. However, there must be a mechanism in place to reset that SAFEARRAY in case the original data is deallocated. Anyway, I can have a look to see what your version does.

[mias1wot]

Sorry, this was just a call to your GetArrayByRef function, I've edited this line. I needed to copy an array from dictionary (key => array) multiple times during code execution so I wanted to make it more efficient by using copy by reference instead of a value based copy. However, when I coded it, it didn't work because as you can see from ExcelCrashes method, GetArrayByRef returns incorrect array with huge boundaries (the example shows 0 to 2, yet it returns something with different huge boundaries) and working with this array makes Excel instantly crash.

[cristianbuse]

Ah ok, makes sense now. Well, the GetArrayByRef in this repo is flawed - it can never be safe unless there is management for when the array gets deallocated.

I have not updated this repo for a while. My big focus in the last year was VBA-FastDictionary.

In your example, were you using Scripting.Dictionary or another implementation?

[cristianbuse]

On another note, it makes sense that b = GetArrayByRef(dict(0)) causes a crash. The dict(0) creates a copy of the array which gets deallocated at soon as GetArrayByRef returns which leaves b pointing to invalid memory.

Can you provide more details on what you are trying to achieve? When I wrote GetArrayByRef I surely did not have the understanding I have now and I might suggest a different approach based on your goals.

[mias1wot]

I'm using standard Scripting.Dictionary.

I see your point. The dict(0) already creates an array copy which is given to GetArrayByRef and, if I understand correctly, the copy is destroyed when the method leaves, right?

I tried to fast-refactor code to clean duplications but it led to multiple array copy which I wanted to avoid by copying by reference instead of value. Now I've already refactored it normally for using different approach. Yet I didn't understand why there is a proper array on the GetArrayByRef exit, but the array is flawed after the assignment b = GetArrayByRef(dict(0)). If the dictionary creates a temporary array copy and gives the method a reference to it and then the array copy gets deallocated after the method is finished, then everything is clear. Thank you! Though it's non-intuitive for me that the methods wants an array reference and dictionary creates a temporary array copy to give a reference to it instead of the original array.

[cristianbuse]

if I understand correctly, the copy is destroyed when the method leaves, right?

Yes. Arrays are always copied except for when they are passed ByRef to a method.

Like I wrote above, I've come a long way since I wrote this, and the only safe way to pass an array without copying is to have a special wrapper class. There are 2 approaches as far as I see it:

1. The special wrapper class has a GetArrayByRef method which returns a side-allocated SAFEARRAY structure. Then the class tracks every single one of these structures so that if the class instance terminates, it cleans all the pvData pointers in those structures. Hope it makes sense
2. The special wrapper class does not track anything, it simply takes ownership of the array and then it can return it. Similar to what Rust does to avoid unnecessary copies. To illustrate this, add an ArrayWrapper class with the following code:

Option Explicit

Const pArrayOffset As Long = 8
Private m_arr As Variant
Private m_vt As VbVarType
Private m_arrPtr As LongPtr
Private ma As MEMORY_ACCESSOR

Public Sub CopyFrom(ByRef arr As Variant)
    If Not IsArray(arr) Then Err.Raise 5, "CopyFrom", "Array required"
    If Not ma.isSet Then InitMemoryAccessor ma
    m_arr = arr
End Sub

Public Sub ClaimFrom(ByRef arr As Variant)
    If Not ma.isSet Then InitMemoryAccessor ma
    '
    ma.sa.pvData = VarPtr(arr)
    ma.sa.rgsabound0.cElements = 1
    '
    Dim vt As VbVarType: vt = ma.ac.dInt(0)
    If (vt And vbArray) = 0 Then
        ma.sa.rgsabound0.cElements = 0
        ma.sa.pvData = NULL_PTR
        Err.Raise 5, "ClaimFrom", "Array required"
    End If
    '
    ma.sa.pvData = ma.sa.pvData + pArrayOffset
    If vt And VT_BYREF Then
        m_vt = vt Xor VT_BYREF
        ma.sa.pvData = ma.ac.dPtr(0)
        m_arrPtr = ma.ac.dPtr(0)
        If m_arrPtr = NULL_PTR Then
            m_arr = arr 'Uninitialized array
        Else
            'Clear Source Pointer
            ma.ac.dPtr(0) = NULL_PTR
            '
            'Transfer to Target
            m_arr = m_arrPtr
            ma.sa.pvData = VarPtr(m_arr)
            ma.ac.dInt(0) = m_vt
        End If
    Else
        m_vt = vt
        '
        'Transfer to Target
        m_arrPtr = ma.ac.dPtr(0)
        m_arr = m_arrPtr
        ma.sa.pvData = VarPtr(m_arr)
        ma.ac.dInt(0) = vt
        '
        'Clear Source
        ma.sa.pvData = VarPtr(arr)
        ma.sa.rgsabound0.cElements = 2
        ma.ac.dCur(0) = 0
        ma.ac.dCur(1) = 0
    End If
    ma.sa.rgsabound0.cElements = 0
    ma.sa.pvData = NULL_PTR
End Sub

Public Sub ReclaimTo(ByRef arr As Variant)
    If Not ma.isSet Then InitMemoryAccessor ma
    '
    ma.sa.pvData = VarPtr(arr)
    ma.sa.rgsabound0.cElements = 1
    '
    Dim vt As VbVarType: vt = ma.ac.dInt(0)
    '
    If vt And VT_BYREF Then
        If (vt And vbArray) = 0 Then
            ma.sa.rgsabound0.cElements = 0
            ma.sa.pvData = NULL_PTR
            Err.Raise 5, "ReclaimTo", "Target Array required"
        ElseIf m_vt = vbEmpty Then
            'Do nothing
        ElseIf (vt Xor VT_BYREF) <> m_vt Then
            ma.sa.rgsabound0.cElements = 0
            ma.sa.pvData = NULL_PTR
            Err.Raise 5, "ReclaimTo", "Invalid Array Data Type"
        End If
        '
        'Swap so we can clean later
        Dim ptr As LongPtr
        '
        ma.sa.pvData = VarPtr(arr) + pArrayOffset
        ma.sa.pvData = ma.ac.dPtr(0)
        ptr = ma.ac.dPtr(0)
        ma.ac.dPtr(0) = m_arrPtr
        '
        ma.sa.pvData = VarPtr(m_arr)
        ma.sa.rgsabound0.cElements = 2
        ma.ac.dCur(0) = 0
        ma.ac.dPtr(1) = ptr
        m_vt = vt Xor VT_BYREF
        ma.ac.dInt(0) = m_vt
        '
        m_arr = Empty 'Clean, if needed
    Else
        ma.sa.pvData = VarPtr(m_arr) + pArrayOffset
        '
        'Transfer to Target
        arr = ma.ac.dPtr(0)
        ma.sa.pvData = VarPtr(arr)
        ma.ac.dInt(0) = m_vt
        '
        'Clear Source
        ma.sa.pvData = VarPtr(m_arr)
        ma.sa.rgsabound0.cElements = 2
        ma.ac.dCur(0) = 0
        ma.ac.dCur(1) = 0
    End If
    m_vt = vbEmpty
End Sub

Now, your code can be written as:

Sub ExcelNoCrash()
    Dim a() As Variant
    ReDim a(2, 2)
    a(0, 0) = 1
    
    Dim dict As New Dictionary
    Dim aw As New ArrayWrapper
    
    aw.ClaimFrom a
    dict.Add 0, aw
    
    Set aw = Nothing
    
    Dim b As Variant
    dict(0).ReclaimTo b
    
    b(1, 1) = 4
End Sub

Obviously, the above is just a mockup. The class can be written with zero dependency on LibMemory.

[mias1wot]

That's awesome!

Though your initial GetArrayByRef method doesn't require to pass the control back to the array to use it again later as the current class leaves the initial array completely empty (nothing). Example:

Sub Example()
    Dim a() As Variant
    ReDim a(2, 2)
    a(0, 0) = 1
    
    Dim dict As New Dictionary
    Dim aw As New ArrayWrapper
    aw.ClaimFrom a ' `a` is nothing now.
    dict.Add 0, aw
    
    Dim b As Variant
    
    For i = 0 To 2
        dict(0).ReclaimTo b ' dict(0).m_arr is nothing now.
        b(2, i) = i ' Error on second cycle.
        
        ' Do smth with b...
        
        ' Without passing ref back from b to dict(0), the next cycle will pass empty array to b.
        'dict(0).ClaimFrom b
    Next
End Sub

I thought about combining your previous code with the new one to make the class more versatile at least for my personal use:

' ArrayWrapper_v2
Option Explicit

Private m_arr As Variant


Public Sub SetArray(ByRef arr As Variant)
    m_arr = LibMemory.GetArrayByRef(arr)
End Sub

Public Function GetArray()
    GetArray = LibMemory.GetArrayByRef(m_arr)
End Function

Public Function GetArrayByRef(ByRef arr As Variant)
    GetArrayByRef = LibMemory.GetArrayByRef(arr)
End Function

Sub Example_2()
    Dim a() As Variant
    ReDim a(2, 2)
    a(0, 0) = 1
    
    Dim dict As New Dictionary
    Dim aw As New ArrayWrapper_v2
    aw.SetArray a ' `a` will continue to exist and point to the same array as `aw.m_arr`.
    dict.Add 0, aw
    
    Dim b As Variant
    
    For i = 0 To 2
        b = dict(0).GetArray
        b(2, i) = i
        
        ' Do smth with b...
        
        ' No need to pass ref back from b to dict(0), the latter contains the same ref.
    Next
End Sub

a, aw.m_arr, b - all of them contain the same array reference now.
Now it contains functionality to save array reference for accessing it later when using collection and the original GetArrayByRef method to copy by ref instantly when working directly with arrays.

[cristianbuse]

I get the appeal to do this but like I wrote above, it is dangerous.

Apologies, I was thinking of another scenario, so I edited my reply. Imagine if the class is passed as a return to the caller method and the array gets deallocated:

Sub Example_3()
    Dim aw As ArrayWrapper_v2
    Set aw = Example_2
    
    Dim b As Variant
    
    b = aw.GetArray
    b(2, 0) = 1
End Sub

Function Example_2() As ArrayWrapper_v2
    Dim a() As Variant
    Dim i As Long
    ReDim a(2, 2)
    a(0, 0) = 1
    
    Dim dict As New Dictionary
    Dim aw As New ArrayWrapper_v2
    aw.SetArray a ' `a` will continue to exist and point to the same array as `aw.m_arr`.
    dict.Add 0, aw
    
    Dim b As Variant
    
    For i = 0 To 2
        b = dict(0).GetArray
        b(2, i) = i
        
        ' Do smth with b...
        
        ' No need to pass ref back from b to dict(0), the latter contains the same ref.
    Next
    Set Example_2 = aw
End Function

As soon as Example_2 returns, the array gets deallocated, and all the ByRef references point to invalid memory. SO, Example_3 causes a crash.

[cristianbuse]

I think passing control is a good and safe way of avoiding making copies. However, i did mention there is another way, but it is more complicated because it requires side-memory allocation. It is actually the reason why the repo has the MemAlloc method. I never had the time to post the latest code, which I wrote in 2023, because it still needs lots of refining and it wasn't a priority for me.

[mias1wot]

Brilliant example!

It means that returning ArrayWrapper_v2 class object is forbidden - it can be used only within the same method or inner ones, but cannot be passed to the caller.

I was thinking about modifying the ArrayWrapper newly added methods to pass the array to the object one and then return its address to the original array (otherwise I get "Automation type not supported error" - present in the code below as a comment).

''' Part of ArrayWrapper_v3 class.

Public Function SetArray(ByRef arr As Variant)
    ClaimFrom arr
    'arr = LibMemory.GetArrayByRef(m_arr) ' "Variable uses an Automation type not supported in Visual Basic" error.
    SetArray = LibMemory.GetArrayByRef(m_arr)
End Function

Public Function GetArray()
    GetArray = LibMemory.GetArrayByRef(m_arr)
End Function

Sub Example_7()
    Dim aw As ArrayWrapper_v3
    Set aw = Example_6
    
    Dim b As Variant
    
    b = aw.GetArray
    b(2, 0) = 1 ' No crash.
End Sub

Function Example_6() As ArrayWrapper_v3
    Dim a() As Variant
    ReDim a(2, 2)
    a(0, 0) = 1
    
    Dim dict As New Dictionary
    Dim aw As New ArrayWrapper_v3
    
    a = aw.SetArray(a)
    dict.Add 0, aw
    
    Dim b As Variant, c As Variant
    b = aw.GetArray
    c = aw.GetArray
    
    a(1, 1) = 8 ' `b` and `c` are not modified.
    b(1, 1) = 4 ' `a` is not modified, although `b` and `c` are.
    Set Example_6 = aw
End Function

But I bumped into a problem: a and b seem to not point to the same memory - they are not modified together.

[cristianbuse]

Yes, it's very tricky to have them point to the same memory while always taking into account all scopes (including global) and possible memory leaks, or worse - wrong data being used. That is why I am more and more conviced that having a single array that is passed around and claimed/reclaimed is the cleanest and simplest workaround.

I will dwell on this in the next few weeks. I have another big project that I want to roll out first. Many thanks for your feedback. I will keep this issue open as a reminder that I need to address this repo at some point.

Cheers

[mias1wot]

Thank you for your time and dedication into VBA. Your results are really cool and at the end of the day, your code does work and allows to copy an array by ref (no idea why not make it default like in other languages). Also thanks for the dictionary workaround. If you have time in the future, that would be great to see what you can do with these challenges :))