Limit of array values

[geohaupt]

Is the limit of one series within the array 29849? When I create data with more values I get the error that "Uncaught TypeError: Cannot read property 'x' of undefined" and no globe is created than.

[redbar0n]

I would really like an answer to this. I am planning on plotting approx. 19 million datapoints onto the globe, and if there is such a limitation then that would be meaningless..

[mrdoob]

You're likely going to run out of memory.

In the near future I would like to update this project so the geometry gets generated in the vertex shader. That way 19 million data points would be feasible. In that case JSON.parse() will likely become the bottle neck :)

[redbar0n]

I see. So there's no way I could split the data and feed it into the globe one segment at a time? Because all 19 mill datapoints needs to be kept in memory for rendering the globe, at the same time anyways?

Actually, I might be able to join datapoints on the same location beforehand, so that would decrease the number of points I would have to plot. I've estimated I will have on average around 70 points per location, thus reducing the size from 19 mill to 271 429 datapoints. Do you think the globe is capable of taking in that many points?

Are you aware of any limit in the array values the globe is able to take in, or is it entirely dependent on the machine running the browser?

[redbar0n]

@geohaupt In the example JSON given, population909500.json, I tried joining together the 3 series into one. That seemed to work fine. So it might indicate that the globe is able to take in 29849*3 = 89547 data points in one series. However, the coordinates in the 3 series were duplicate coordinates, but with slightly different magnitudes at some points. I noticed that when you input the same coordinates several times into the same series, the globe will just display their highest magnitude, and discard the rest of the magnitudes (not adding them together). So if the globe is able to actually display 89547 data points at the same time, is not confirmed.

[redbar0n]

@geohaupt When I took that one conjoined series of 89547 data points, and copy pasted the content two more times into the same series, it became 89547*3 = 268 641 data points. The globe would take some more seconds to load, but would still display nicely. However, it would start lagging a little bit. When I increased to 6 times the original amount, 89547 * 6 = 537 282, then it took 100% of my CPU, 800-900 MB of memory, and stalled indefinitely. It seems I ran out of available memory, like @mrdoob said.

Reducing again to 4 times the original amount, 89547*4 = 358 188 data points. I still ran out of memory, and got a message about having to kill the process.

So it seems the limit is around 268 641 data points, maybe up to 300 000, for my computer, at least. My computer specs are: Macbook Pro 2,4 GHz Intel Core 2 Duo, 4 GB 1067 MhZ DDR3 RAM, NVIDIA GeForce 320M 256 MB, OS X 10.9.4.

PS: I also got the error you were getting when I had more than one data series, with different amount of data points in them. I think it expects data series with the same amount of data points.

[codefrenzy]

Hi @redbar0n ,

Were you ever able to get around the issue you found where this error is thrown if you have more than one data series with different amounts of data points in them? I'm encountering the same issue - #28.

Thanks,
Annie

[redbar0n]

Hi @codefrenzy,

no, I'm afraid that the only solution I found was to limit the total amount of data points plotted. Maybe it's possible to pre-process the list of data-points so one can limit the amount of data-points but still preserve the original ratio. I think perhaps that's what I did, if I remember correctly. After all, it's not really necessary to plot all data points, just the ratio. And the height of the bars on the globe can be scaled to whatever height you want. So the difference won't be noticable at all.

PS: I don't think there's a problem with plotting more than one data series with data points. In the worst case, you can always just merge them before just plotting one conjoined series.

Hope this was of any help.

cheers, and good luck :)

[yeshiqing]

Hello @redbar0n ,
I meet the same problem.
your 89547 data points means [ latitude, longitude, magnitude] equals one data or 3 data??
and what do you mean that "895476 = 537 282 data points"?
and you suggest merge them ,is that mean "globe.addData" for several time instead of on time??
" globe.addData(data, {
format: 'legend'
});
globe.createPoints();
globe.animate();"

[redbar0n]

Hi @pageye123

your 89547 data points means [ latitude, longitude, magnitude] equals one data or 3 data??

I wrote: "In the example JSON given, population909500.json, I tried joining together the 3 series into one. That seemed to work fine. So it might indicate that the globe is able to take in 29849*3 = 89547 data points in one series."

If I remember correctly, this means that 89547 data points means that [ latitude, longitude, magnitude] equals 3 data. That's since I was joining the 3 series (one series each for lat, long, magn into one large conjoined series). So there are 29849 latitudes, and the same amount of longitudes, and the same amount of magnitudes.

and what do you mean that "895476 = 537 282 data points"?

It seems that github showed 895476 = 537 282, when I wrote 89547 * 6 = 537 282 (without the spaces surrounding the multiplication sign). Now with spaces it should be clear.

and you suggest merge them ,is that mean "globe.addData" for several time instead of on time??

No, not really. If I remember correctly I pre-processed (merged) the data points from 3 series into one series. This is done outside the flow of the normal program. I think I just used Sublime Text editor to edit the arrays, or wrote a small script to edit the json file so that it had one large series of data instead of 3 series of data. Then, in the JavaScript code on the globe, I did globe.addData on that series of data, from the edited json file.

Hope this may help.