World: r3wp

Steeve, I couldn't find how to in help, so I just asked. :-)

Public messages moved. I'll move the private messages when I figure 
out how to list them.

[VBS] Humm, as i stated, performances are bad currently with append, 
it took 1 minute to add 10000 values (166 values by second).

[VBS] But to fetch 10000 values randomly, it took only 0.01 sec (fast)

[VBS] Did you realize that 1 000 000 values can be fetched in a loop 
without bloating the memory in only 1 SEC

[VBS] SORRY, i made a wrong read, it took 1 sec to fetch 10 000 values 
not 1 000 000 but it's quite good too, no ?       -_-'

@BrianH : What are microprocesses ? you mean something like very 
lightweight processes that don't share common memory?

(lightweing in sense that they can be created and destroyed quickly 
and that there can be many of them)

Steeve, your performance measurements are about dead on with what 
Tretbase performance is.

http://www.rebol.net/cgi-bin/rebdev-web.r?msg=577

Here is a fetch of 10,000 records:

>> s: now/precise db/get test d t: now/precise
== 22-Jan-2009/12:13:34.534-6:00
>> difference t s
== 0:00:00.973


Pretty much about  1 second - similiar to what your seeing.  I don't 
think we can get much more performance out of REBOL than that.

Paul: is tretbase already intended for use or still in development?

It is still in development Janko but progressing.

Tretbase is going to be ported to C though.

to C, interesting

this is maybe a good idea to develop something in gihert level language 
to figure out all bits and pieces and algoritms, and then when you 
have a good picture port it into c

Paul, in fact i had the same result than you, i wrote 1 sec to give 
an arround :)

gihrt = higher

Yeah I figured that Steeve.

Janko, the problem is that REBOL at least in R2 doesn't give us the 
control of memory and such as I would like.

did you try 10 000 random access ? is was in that case

Here is the total experience I did Steeve:

>> d: []
== []
>> loop 10000 [append d random 100000]

== [33695 15852 80380 7196 57367 71232 73853 47912 70304 20415 65171 
70632 77767 89530 45257 65786 34935
 70620 90566 92441 23445 26...
>> db/get test d

== [[44 #{00000354} [paul tretter 41]] [45 #{00000368} [paul tretter 
41]] [46 #{0000037C} [paul tretter
41]] [55 #{00000430} [paul ...
>> s: now/precise db/get test d t: now/precise
== 22-Jan-2009/12:13:34.534-6:00
>> difference t s
== 0:00:00.973

Notice my records are only three fields each.

Obviously the size of the record will impact the timing.

But mine is also returning not just the whole record but also the 
change id tag.

Steeve, don't let Carl's promise of RIF stop you from working on 
VBS.  I have heard that promise for years  now.

Hum currently, it's a problem for me, he said that his RIF conception 
have better perfs than VBS.

He said too that i  have not done the best choices (missing cache 
for example).

He said too than doing such a scheme is complex (beyond my understanding 
???)

I don"t want to work on a thing which will be uselless when R3 will 
be released.
He said better perfs with RIF, so RIF works currently...
I wonder why rebdev is not using it...
I'm a little bit disapointed...

I think we just don't know all the capabilities that we can harness 
with REBOL.  Regarding caching, I know that you have to tune how 
much you send to a read/write request for optimal port usage.  Carl 
is going to know the underlying details better than we would.

But the real attribute that we would need to take advantage of is 
registers and we can't do that.

At least not that has been published as far as I know.

hm.. I would be a little disapointed to :(  ... but when will RIF 
be available then.. isn't time in general problem with RT

RIF is not going to come right now. That is probably a feature for 
3.1.

So perhaps in 2 or 3 years, but it has excellent perfs currently 
according to him, But Carl doesnt use it in rebdev, very coherent...

Steeve, I read his last message as: rebdev doesn't need it yet as 
the current model would easily scale to over 100000 messages.

yes perfs will be here, but what about the memory overhead ?

his loading the full database in memory, in 2 different blocks (the 
second one rebuild and sorted each time you connect)

so i can make an estimation of the memory used for 100 000 messages, 
wait a minute...

I take 16 bytes for storing a value in memory (like for R2) i don't 
 know if it's more in R3.

An unique message has a structure (a block!) of a minimum size of 
80 bytes, so i take 110 bytes (for a small message of  30 characters 
in average).

for a block containing 100 000 messages we have 100 000 * (16 + 110) 
= 12 Mb

So for the 2 blocks , we have 24 Mb, just to deal with 100 000 messages.

It's growing fast, because rebdev store many events too (remove, 
private message, move, etc...).
and it will be connected to devbase too.

I think that  thousands of messages  and events will be added per 
month when the community will have access to rebdev.

So his current implementation will resist 2 or 3 months before reaching 
tens of Megabyte in memory usage.

And when it will go over 100 000 messages (in 4 or 6 month) the perfs 
will blow !!!!!

i made a wrong statement

the second block uses references not a copy of the message, so that 
the second block has a size of 100 000 * 16 = 1,5 Mb

So that it reduces the size of the memory overhead by 2.

i can do a prediction, it will end like with Altme, using 50 Mb of 
memory after one year of usage

And you have the light version currently, the console version.

It will be worst with the GUI if Carl doen't change his conception

A couple things to realize:

- The chat client can change easily, independent of R3, and does 
change daily.

- Our priority is on making the public release happen ASAP. Chat 
reaching its limits is months down the road at worst. We have time.

Right now the prioirty for the people usng R3 needs to be testing 
R3, in prep for its release.

For some of us, outside of work time of course.

The chat can be changed now because it's tiny enough, 

But 50% of his conception is wrong and has to be rewritten to deal 
with direct access in files instead of loading all in memory.

Yes i read carefully his script, 50% of it must be rewritten now...

More Carl will developp the GUI and the console version with wrong 
choices (to my mind) more it will be painfull to rewrite all later.

50% of it must be rewritten now
 - Not *now*, later. After the public release of R3. Priorities.

The chat database will be tiny enough for months. We aren't even 
done with the feature set. The internals can change easily. There 
are less than 700 messages, and we don't start having problems til 
100000. I think we have time for this.

I only reacted with your false statement, that it's easy to modify 
the chat to kill the memory overhead.
You didn't studied it carrefully enough.

I know there is no chance that Carl and other gurrus follow my advices.
But I know i'm right, i hope you will not regret it later...