World: r3wp

moving to SSD disks might help a bit :-)

Well, it would, actually. It would still be slow to use, but not 
as slow. Reallocation takes even more memory.

Yes, of course.

This is assuming virtual memory. I wouldn't even be able to test 
on this system; it only has 1GB of RAM. My main system would work 
though.

I was hinting at the fact that this probably won't matter much if 
we are talking about a 512M system :)

>> dt [m: make map! [] repeat i to-integer 2 ** 24 [poke m i i]]
== 0:01:22.254896
;; 2393MB resident

>> dt [m: make map! n: to-integer 2 ** 24 repeat i n [poke m i i]]
== 0:00:48.329933
;; 1026MB resident

I suggested preallocation in a comment. You might want to chime in 
with that code :)

I would probably make it significantly large than the number of expected 
pairs.

Round up :)

;; Storing 2^24 in a map with 2^25 preallocated

>> dt [m: make map! to-integer 2 ** 25 repeat i to-integer 2 ** 24 
[poke m i i]]
== 0:00:33.695578
;; 1538MB resident

~30% faster.

Still incredibly slow.

Are you doing those measurements in a fresh console each time? Remember, 
process allocation from the OS matters too.

Yes :)

Figured :)

With speedstepping switched off and the REBOL process pinned to a 
single core.

And ...

What antagonist language of the day do we want to counter-benchmark?

Let's go with Python.

~10 times faster (without pre-allocation). Well ...

Guess there's some room for improvement :)

How fast without the repeat, but with preallocation, comparing Python 
and R3? Remember, Python is compiled.

Bytecode compiled, with a really slow VM.

Bytecode compiled is still compiled, and it is likely that bytecodes 
specific to loops are being used. I am interested in the comparison 
of the preallocation of the map! type versus the Python equivalent, 
whatever that is.

I don't know of a way to preallocate Python dicts.

Python dicts are probably not allocated in a large chunk.

Well, afair they use open addressing in their implementation, so 
I guess they will.

But then, no idea about the impl details.

As an aside, path notation (`m/(i): i`) is slightly (6%) faster than 
POKE in this case.

POKE has to look up the function twice: Once from the word, the nect 
time in the action's datatype. Path evaluation at least knows what 
it's doing, so it doesn't have to figure it out.

the next time in the datatype's action list.

There must be some algorithm issue in R3 map!. When I have 21,000,000 
key-value pairs in a map, accessing it becomes very slow. Using " 
mymap/:key " to get a value takes 0.2 sec.

Did you read the notes suggesting that you might be running out of 
physical memory (RAM)?

I hope the lack of enough physical memory is the reason. I have 2GB 
RAM in my PC. I will get my MacBook Pro this evening. It will have 
8GB RAM. I will test this in my Mac.

Well, 2GB will be a close call for 21M entries.

Have a look at your memory/swap consumption, that'll probably help 
you identify if that's a problem.

Yeah, I hope we will have 64-bit REBOL for Mac soon. Analizing social 
networking data without enough physical memory is a pain.

Particularly on OSX (or Windows 7/Vista), since the OS itself uses 
a lot of RAM.

Any serious data handling with 32b is a pain :)

Hm, but yes. There might actually be something seriously off about 
>>2^24 entries.

Initialising a map with 21M entries just took insanely long for me. 
Investigating.

Andreas, would you post a ticket in CC on this? You probably can 
describe the issue better than me.

If I can pin down an issue, I will.

On my system it had to allocate virtual memory for the process from 
the OS, and swap memory in RAM to the VM so it would have room to 
allocate the map in the working RAM. It took as long as I would have 
expected it to take given that circumstance.

As soon as you start swapping, all bets are off.

An empty map! of 22,000,000 entries took nearly 1GB of RAM on its 
own, and that doesn't include memory for any strings, blocks or structures 
that you might add to the map after it is allocated.

I used DP instead of DT, and it gave me all the details.

Can't tell you about the nature of your virtual memory, though.

Well, it technically could, but it doesn't :)