World: r3wp

How fast without the repeat, but with preallocation, comparing Python 
and R3? Remember, Python is compiled.

Bytecode compiled, with a really slow VM.

Bytecode compiled is still compiled, and it is likely that bytecodes 
specific to loops are being used. I am interested in the comparison 
of the preallocation of the map! type versus the Python equivalent, 
whatever that is.

I don't know of a way to preallocate Python dicts.

Python dicts are probably not allocated in a large chunk.

Well, afair they use open addressing in their implementation, so 
I guess they will.

But then, no idea about the impl details.

As an aside, path notation (`m/(i): i`) is slightly (6%) faster than 
POKE in this case.

POKE has to look up the function twice: Once from the word, the nect 
time in the action's datatype. Path evaluation at least knows what 
it's doing, so it doesn't have to figure it out.

the next time in the datatype's action list.

There must be some algorithm issue in R3 map!. When I have 21,000,000 
key-value pairs in a map, accessing it becomes very slow. Using " 
mymap/:key " to get a value takes 0.2 sec.

Did you read the notes suggesting that you might be running out of 
physical memory (RAM)?

I hope the lack of enough physical memory is the reason. I have 2GB 
RAM in my PC. I will get my MacBook Pro this evening. It will have 
8GB RAM. I will test this in my Mac.

Well, 2GB will be a close call for 21M entries.

Have a look at your memory/swap consumption, that'll probably help 
you identify if that's a problem.

Yeah, I hope we will have 64-bit REBOL for Mac soon. Analizing social 
networking data without enough physical memory is a pain.

Particularly on OSX (or Windows 7/Vista), since the OS itself uses 
a lot of RAM.

Any serious data handling with 32b is a pain :)

Hm, but yes. There might actually be something seriously off about 
>>2^24 entries.

Initialising a map with 21M entries just took insanely long for me. 
Investigating.

Andreas, would you post a ticket in CC on this? You probably can 
describe the issue better than me.

If I can pin down an issue, I will.

On my system it had to allocate virtual memory for the process from 
the OS, and swap memory in RAM to the VM so it would have room to 
allocate the map in the working RAM. It took as long as I would have 
expected it to take given that circumstance.

As soon as you start swapping, all bets are off.

An empty map! of 22,000,000 entries took nearly 1GB of RAM on its 
own, and that doesn't include memory for any strings, blocks or structures 
that you might add to the map after it is allocated.

I used DP instead of DT, and it gave me all the details.

Can't tell you about the nature of your virtual memory, though.

Well, it technically could, but it doesn't :)

I used process explorer.

allocating 22M entries leads to 929MB resident, for me

Yup, that was the nearly part :)

Well, at around 2^24 entries things start to deteriorate noticeably.

Long before we even reach 2GB resident.

So probably Jerry was right from the beginning after all :)

The OS and other apps are also resident. Anything over 512MB would 
push things on a 2GB system running OSX.. I wish my main system was 
running - it has 4GB, which is plenty even on Win7.

Believe me when I say I have sufficient amounts of free RAM :)

Only the R3 process looks like it doesn't even want to go beyond 
1.9GB resident :)

Are you aware that on a 32 bits system, not the full 4 GB is available, 
because that's the size of the total address space?

Yes.

A considerable part of that is needed by the OS for its kernel space. 
2 GB is a bit low for that limit on the main OSes, but it makes you 
think that there's a sort of limit like that in R3

64-bit host system here.

Yes, but R3 is 32 bits, so its address space is at most 4 GB and 
could be 2 GB if for example one bit of the address is used as a 
flag

Of course.

But it's easy to confirm that this is not the case for R3.

You said it doesn't seem to want to go beyond 1.9 GB resident

For a single map, yes.

So maybe a 31 bits limit for a map

>> m5: make map! 22'000'000 
** Internal error: not enough memory
** Where: make
** Near: make map! 22000000

>> stats
== 3890639112

So obviously no PAE hacks, but we get 4GB usable address space for 
the R3 process, at least.