World: r3wp

Particularly on OSX (or Windows 7/Vista), since the OS itself uses 
a lot of RAM.

Any serious data handling with 32b is a pain :)

Hm, but yes. There might actually be something seriously off about 
>>2^24 entries.

Initialising a map with 21M entries just took insanely long for me. 
Investigating.

Andreas, would you post a ticket in CC on this? You probably can 
describe the issue better than me.

If I can pin down an issue, I will.

On my system it had to allocate virtual memory for the process from 
the OS, and swap memory in RAM to the VM so it would have room to 
allocate the map in the working RAM. It took as long as I would have 
expected it to take given that circumstance.

As soon as you start swapping, all bets are off.

An empty map! of 22,000,000 entries took nearly 1GB of RAM on its 
own, and that doesn't include memory for any strings, blocks or structures 
that you might add to the map after it is allocated.

I used DP instead of DT, and it gave me all the details.

Can't tell you about the nature of your virtual memory, though.

Well, it technically could, but it doesn't :)

I used process explorer.

allocating 22M entries leads to 929MB resident, for me

Yup, that was the nearly part :)

Well, at around 2^24 entries things start to deteriorate noticeably.

Long before we even reach 2GB resident.

So probably Jerry was right from the beginning after all :)

The OS and other apps are also resident. Anything over 512MB would 
push things on a 2GB system running OSX.. I wish my main system was 
running - it has 4GB, which is plenty even on Win7.

Believe me when I say I have sufficient amounts of free RAM :)

Only the R3 process looks like it doesn't even want to go beyond 
1.9GB resident :)

Are you aware that on a 32 bits system, not the full 4 GB is available, 
because that's the size of the total address space?

Yes.

A considerable part of that is needed by the OS for its kernel space. 
2 GB is a bit low for that limit on the main OSes, but it makes you 
think that there's a sort of limit like that in R3

64-bit host system here.

Yes, but R3 is 32 bits, so its address space is at most 4 GB and 
could be 2 GB if for example one bit of the address is used as a 
flag

Of course.

But it's easy to confirm that this is not the case for R3.

You said it doesn't seem to want to go beyond 1.9 GB resident

For a single map, yes.

So maybe a 31 bits limit for a map

>> m5: make map! 22'000'000 
** Internal error: not enough memory
** Where: make
** Near: make map! 22000000

>> stats
== 3890639112

So obviously no PAE hacks, but we get 4GB usable address space for 
the R3 process, at least.

On pre-Vista server versions of Windows you used to be able to change 
the balance to 1GB OS, 3GB apps, as an install opton. This is likely 
done dynamically now.

The famous /3GB switch :)

Recently Carl increased the limit on maps - it used to be less than 
500,000 pairs - but if there is a hard limit now it is unlikely to 
be increased for the 32bit builds.

Do you have a CureCode reference for that handy?

He even wrote a blog about it.

Here are my current findings:
https://gist.github.com/1ece58c8302b2d1f734b

Ah, of course!

A hard-coded allocation progression would have been my current guess 
:)

http://issue.cc/r3/1344

http://www.rebol.net/r3blogs/0295.html

We clearly have a different concept of "recently" :)

On this project, anything in the last year is recently.

Disqualifying that ticket and blog :)

Which this wasn't

I'm having AltME freezes again.

A single map! is limited to 2 ** 26 - 1 entries. Or ~3GB memory.