Best Approach for dynamic variables?

[ruribev]

Hello!

I am creating a CSV editor for files with millions of records, so I ran into the following problem:

Because a CSV file can have a variable number of columns (my variables), I end up creating different structures for each number of variables like the following:

public struct data1
{
	public float f1;
}

public struct data2
{
	public float f1;
	public float f2;
}

public struct data3
{
	public float f1;
	public float f2;
	public float f3;
}

And keep going for 100 variables and It works with great performance, but it seems like a weird approach and I'm not sure if there's some other solution that I'm not looking at right now.

Any help with this is greatly appreciated!

[nijnstein]

You can just use only the largest struct and cast to that as long as your base type is kept to float in the nativearray supplied and you dont access the fields not mapped (they will contain the next record which is very handy while debugging).

If you use a nativearray and fill the array with all data, then u could use a construct like:

((data1024*)&((float*)NativeArrayUnsafeUtility.GetUnsafeBufferPointerWithoutChecks())[ record_index * fieldcount ])->f1  (or [0].f1)

This is far from pretty though but wraps up nicely in a utilty function.

y

[ruribev]

Thanks a lot I will be using this.

All the best,
Rodrigo

[nijnstein]

Future work should include utilities for this but you always run into compile time constraints with fixed structures, which youre almost required to use in burst.. its difficult to get to a solution for any situation that is also performant and i hope that input like yours will lead to one

[nijnstein]

Another option would be including a simple codegen for structures based on the inputs set for the script in blast

[ruribev]

I will be exploring using vcodegen just for the performance

[ruribev]

I found the following article:
http://graphicdna.blogspot.com/2014/03/fast-casting-of-c-structs-with-no.html

Not sure how efficient is but is handy for me the following:

	[Serializable]
	[StructLayout(LayoutKind.Explicit)]
	public struct data1
	{
		[FieldOffset(0)]
		public float4 F1;
	}
	
	[Serializable]
	[StructLayout(LayoutKind.Explicit)]
	public struct data2
	{
		[FieldOffset(0)]
		public float4 F1;
		[FieldOffset(1)]
		public float4 F2;
	}
	
	[Serializable]
	[StructLayout(LayoutKind.Explicit)]
	public struct data3
	{
		[FieldOffset(0)]
		public float4 F1;
		[FieldOffset(1)]
		public float4 F2;
		[FieldOffset(2)]
		public float4 F3;
	}
***
	[Serializable]
	[StructLayout(LayoutKind.Explicit)]
	public struct data256{***}

	[StructLayout(LayoutKind.Explicit)]
	public struct data
	{
		[FieldOffset(0)]
		public data1 Data1;
		[FieldOffset(0)]
		public data2 Data2;
		[FieldOffset(0)]
		public data3 Data3;
                ***
		[FieldOffset(0)]
		public data256 Data256;
	}

And when using it I go for the largest data and casting to the one I wanna use later:

	    NativeArray<data2> data_array = new NativeArray<data2>(DataCount, Allocator.Persistent);

            for (int i = 0; i < DataCount; i++)
            {
            	data inputData=new data();
	            inputData.Data256 = new data256()
                {
	                F1 = 100f * i,
	                F2 = 101f * i
                };
	            data_array[i]=inputData.Data2;
            }

[nijnstein]

This is how unity defines their fixed arrays and although there is a explicit data copy the compiler will likely optimize it away and because of the way slower interpretation later it probably wont matter much for performance.

I think at some point for ease of use it should generate cs code for matching structures to data in design|compile time. Because anything fixed is almost never gonna fit anyway and those using runtime modification could easily use an approach like yours or directly access the buffer like in my method. I should add some methods to hide the pointer logic for mapping structures a bit.

Maybe a dialog in unity would help to copy paste the structure's code from after setting some script to it? That would work for some use cases.