World: r3wp

Only the R3 process looks like it doesn't even want to go beyond 
1.9GB resident :)

Are you aware that on a 32 bits system, not the full 4 GB is available, 
because that's the size of the total address space?

Yes.

A considerable part of that is needed by the OS for its kernel space. 
2 GB is a bit low for that limit on the main OSes, but it makes you 
think that there's a sort of limit like that in R3

64-bit host system here.

Yes, but R3 is 32 bits, so its address space is at most 4 GB and 
could be 2 GB if for example one bit of the address is used as a 
flag

Of course.

But it's easy to confirm that this is not the case for R3.

You said it doesn't seem to want to go beyond 1.9 GB resident

For a single map, yes.

So maybe a 31 bits limit for a map

>> m5: make map! 22'000'000 
** Internal error: not enough memory
** Where: make
** Near: make map! 22000000

>> stats
== 3890639112

So obviously no PAE hacks, but we get 4GB usable address space for 
the R3 process, at least.

On pre-Vista server versions of Windows you used to be able to change 
the balance to 1GB OS, 3GB apps, as an install opton. This is likely 
done dynamically now.

The famous /3GB switch :)

Recently Carl increased the limit on maps - it used to be less than 
500,000 pairs - but if there is a hard limit now it is unlikely to 
be increased for the 32bit builds.

Do you have a CureCode reference for that handy?

He even wrote a blog about it.

Here are my current findings:
https://gist.github.com/1ece58c8302b2d1f734b

Ah, of course!

A hard-coded allocation progression would have been my current guess 
:)

http://issue.cc/r3/1344

http://www.rebol.net/r3blogs/0295.html

We clearly have a different concept of "recently" :)

On this project, anything in the last year is recently.

Disqualifying that ticket and blog :)

Which this wasn't

I'm having AltME freezes again.

A single map! is limited to 2 ** 26 - 1 entries. Or ~3GB memory.

You should put that in a comment to #1344.

Already did. Also added a (rather unlovingly formatted) comment to 
the map! docs.

(http://www.rebol.com/r3/docs/datatypes/map.html)

Here's a plot of the DT for each "put" operation (m/(i): i) for 19M 
puts:
http://bolka.at/2010/rebol3/tmp/raw19.png

Will do a properly sampled one tomorrow.

>> to-integer #{ffff}

== 65535  ; how can I make it -1,  is there a to-signed-integer function?

I believe that integer!s in R3 are signed 64-bit integers.

>> to-integer #{ffffffffffffffff}

== -1

They are, but that's the question is different... we should be able 
to do such a conversions somehow. Don't know how at this moment.

>> sext32: func [x] [32768 xor x - 32768]
>> sext32 to-integer #{ffff}
== -1

something like:
>> shift to-integer #{FFFF0000 00000000} -48
== -1

:)

sorry, should be sext16

>> sext16: func [x] [32768 xor x - 32768]
>> sext16 to-integer #{ffff}
== -1

>> shift shift to-integer #{FFFF} 48 -48
== -1

>> shift shift to-integer #{0001} 48 -48
== 1

My version seems to be a little bit faster:
>> sext16_v1: func [x] [32768 xor (to integer! x) - 32768]
>> sext16_v2: func [x] [shift shift to integer! x 48 -48]
>> dt [sext16_v1 #{ffff}]
== 0:00:00.00001
>> dt [sext16_v2 #{ffff}]
== 0:00:00.000009

Your persone is a little bit saner as well :)

version*

Well, another idea for the map! performance degradation: the actual 
hash function used is flawed for >>2^24 keys.

Another indication: plain CHECKSUM computes a 24-bit CRC

I'm now quite confident that this is the underlying problem.