World: r3wp

Maxim, I am not sure if doing that will have a continuous disk space. 
Why don't we have something like this: make %file 100000000

usually it will depend on disk fragmentation.  I'm not sure all OS 
allow you to force a contiguous disk area. 

on any OSes which allow it, it could be a good suggestion for R3.

Actually, I am making a NoSQL in R3 for Chinese Social Network Analysis. 
I have more than 20,000,000 records, and every records might need 
a few KB.

you should post a CC wish ticket for it.

(and its a good idea to give some details as to why you need (in 
the CC ticket) this in order to give persepective on the ticket)

When data are not much, my NoSQL was very fast, I called it Lightning 
DB. Now with so much data, It's very slow, and I call it Snail DB. 
:-(

20,000,000 * 1kb *  n   =  a massive file!  :-)

OK, I will post this wish in CC.

are you sure its related to fragmentation?

Well... I am 95 % sure. By the way, Every column of every table has 
its own one or two huge files. I didn't put all the data in a single 
file.

As far I remember It's working with R2 if you write the %file past 
the end, it's filled with #{00].

I remember having asked the same feature in R3 one year ago at least 
(in the R3 chat)

Are you sure it must be in one file?

Thanks again, Jerry, for pushing the resource usage limits of R3. 
http://issue.cc/r3/1799is about reallocating a 256+ MB map! to an 
even larger map!, and then getting a slowdown probably because of 
virtual memory use. We really need tasks so this can go on in the 
background :)

More likely a 512+MB map.

I think the problem was a realloc from 256+MB to 512+MB, which would 
temporarily have 768+MB in memory, plus a hash table recalculation.

If a single R3 value slot is 128bit and a map needs two value slots 
for each (key, value) pair:
(128 / 8) * 2 * (2 ** 24) / (1024 ** 2) == 512.0

Weird, I did a similar calculation and got 256. I should revise my 
comment.

Oh right, I put a 16 in there when it should have been a 32.

so should maps allow to be pre-allocated like series?

They are already.

m: make map! n

Well, at least that allocates _something_ :)

hehe

:-)

So is it about the initial sufficient prediction of programmer allocating 
enough of memory, or is there some artificial limit for the map size?

its about the fact that some things have to be arrays in ram, and 
if you don't make them big enough to begin with eventually, you have 
to live with this sort of "cleanup'

REBOL didn't crash so I'd assume it did its job correctly.  
but a few tests might prove that something is not optimal.

Why do guys need so large map array? Don't you remember Bill Gates 
once said, that 640KB is enough for everyone? :-)

this seems to prove that pre-allocating maps does indeed store all 
the space required:

>> stats
== 921576
>> m: make map! 20000000
== make map! [
]
>> stats
== 909353688

Jerry, you might want to try the above and see if the wait occurs 
on the 21 millionth item.

(20 millionth item + 1)

I'm with Brian in that most likely the only thing "not optimal" is 
the amount of RAM in Jerry's system.

yep!

if the pre-allocation fixes the setup. then there's no bug in REBOL.

Preallocation won't help with insufficient memory :)

moving to SSD disks might help a bit :-)

Well, it would, actually. It would still be slow to use, but not 
as slow. Reallocation takes even more memory.

Yes, of course.

This is assuming virtual memory. I wouldn't even be able to test 
on this system; it only has 1GB of RAM. My main system would work 
though.

I was hinting at the fact that this probably won't matter much if 
we are talking about a 512M system :)

>> dt [m: make map! [] repeat i to-integer 2 ** 24 [poke m i i]]
== 0:01:22.254896
;; 2393MB resident

>> dt [m: make map! n: to-integer 2 ** 24 repeat i n [poke m i i]]
== 0:00:48.329933
;; 1026MB resident

I suggested preallocation in a comment. You might want to chime in 
with that code :)

I would probably make it significantly large than the number of expected 
pairs.

Round up :)

;; Storing 2^24 in a map with 2^25 preallocated

>> dt [m: make map! to-integer 2 ** 25 repeat i to-integer 2 ** 24 
[poke m i i]]
== 0:00:33.695578
;; 1538MB resident

~30% faster.

Still incredibly slow.

Are you doing those measurements in a fresh console each time? Remember, 
process allocation from the OS matters too.

Yes :)