World: r3wp

Paul, in fact i had the same result than you, i wrote 1 sec to give 
an arround :)

gihrt = higher

Yeah I figured that Steeve.

Janko, the problem is that REBOL at least in R2 doesn't give us the 
control of memory and such as I would like.

did you try 10 000 random access ? is was in that case

Here is the total experience I did Steeve:

>> d: []
== []
>> loop 10000 [append d random 100000]

== [33695 15852 80380 7196 57367 71232 73853 47912 70304 20415 65171 
70632 77767 89530 45257 65786 34935
 70620 90566 92441 23445 26...
>> db/get test d

== [[44 #{00000354} [paul tretter 41]] [45 #{00000368} [paul tretter 
41]] [46 #{0000037C} [paul tretter
41]] [55 #{00000430} [paul ...
>> s: now/precise db/get test d t: now/precise
== 22-Jan-2009/12:13:34.534-6:00
>> difference t s
== 0:00:00.973

Notice my records are only three fields each.

Obviously the size of the record will impact the timing.

But mine is also returning not just the whole record but also the 
change id tag.

Steeve, don't let Carl's promise of RIF stop you from working on 
VBS.  I have heard that promise for years  now.

Hum currently, it's a problem for me, he said that his RIF conception 
have better perfs than VBS.

He said too that i  have not done the best choices (missing cache 
for example).

He said too than doing such a scheme is complex (beyond my understanding 
???)

I don"t want to work on a thing which will be uselless when R3 will 
be released.
He said better perfs with RIF, so RIF works currently...
I wonder why rebdev is not using it...
I'm a little bit disapointed...

I think we just don't know all the capabilities that we can harness 
with REBOL.  Regarding caching, I know that you have to tune how 
much you send to a read/write request for optimal port usage.  Carl 
is going to know the underlying details better than we would.

But the real attribute that we would need to take advantage of is 
registers and we can't do that.

At least not that has been published as far as I know.

hm.. I would be a little disapointed to :(  ... but when will RIF 
be available then.. isn't time in general problem with RT

RIF is not going to come right now. That is probably a feature for 
3.1.

So perhaps in 2 or 3 years, but it has excellent perfs currently 
according to him, But Carl doesnt use it in rebdev, very coherent...

Steeve, I read his last message as: rebdev doesn't need it yet as 
the current model would easily scale to over 100000 messages.

yes perfs will be here, but what about the memory overhead ?

his loading the full database in memory, in 2 different blocks (the 
second one rebuild and sorted each time you connect)

so i can make an estimation of the memory used for 100 000 messages, 
wait a minute...

I take 16 bytes for storing a value in memory (like for R2) i don't 
 know if it's more in R3.

An unique message has a structure (a block!) of a minimum size of 
80 bytes, so i take 110 bytes (for a small message of  30 characters 
in average).

for a block containing 100 000 messages we have 100 000 * (16 + 110) 
= 12 Mb

So for the 2 blocks , we have 24 Mb, just to deal with 100 000 messages.

It's growing fast, because rebdev store many events too (remove, 
private message, move, etc...).
and it will be connected to devbase too.

I think that  thousands of messages  and events will be added per 
month when the community will have access to rebdev.

So his current implementation will resist 2 or 3 months before reaching 
tens of Megabyte in memory usage.

And when it will go over 100 000 messages (in 4 or 6 month) the perfs 
will blow !!!!!

i made a wrong statement

the second block uses references not a copy of the message, so that 
the second block has a size of 100 000 * 16 = 1,5 Mb

So that it reduces the size of the memory overhead by 2.

i can do a prediction, it will end like with Altme, using 50 Mb of 
memory after one year of usage

And you have the light version currently, the console version.

It will be worst with the GUI if Carl doen't change his conception

A couple things to realize:

- The chat client can change easily, independent of R3, and does 
change daily.

- Our priority is on making the public release happen ASAP. Chat 
reaching its limits is months down the road at worst. We have time.

Right now the prioirty for the people usng R3 needs to be testing 
R3, in prep for its release.

For some of us, outside of work time of course.

The chat can be changed now because it's tiny enough, 

But 50% of his conception is wrong and has to be rewritten to deal 
with direct access in files instead of loading all in memory.

Yes i read carefully his script, 50% of it must be rewritten now...

More Carl will developp the GUI and the console version with wrong 
choices (to my mind) more it will be painfull to rewrite all later.

50% of it must be rewritten now
 - Not *now*, later. After the public release of R3. Priorities.

The chat database will be tiny enough for months. We aren't even 
done with the feature set. The internals can change easily. There 
are less than 700 messages, and we don't start having problems til 
100000. I think we have time for this.

I only reacted with your false statement, that it's easy to modify 
the chat to kill the memory overhead.
You didn't studied it carrefully enough.

I know there is no chance that Carl and other gurrus follow my advices.
But I know i'm right, i hope you will not regret it later...

I didn't dispute your statement, just its urgency. Priorities.

700 messages in 3 weeks, with 5 regular users, you're according my 
point.

Many eyes make bugs shallow. We need more developers, which means 
we need a public release. Priorities. We can fix chat *after* the 
public release. As it is, getting the file store in place is a more 
pressing need. That way we can release and track the source. Once 
the source is released we can make fixes as needed. The priority 
needs to be on testing R3 itself, not the chat client. We aren't 
even going to start the GUI chat client until after the R3 public 
release.

It's good to see a little realism about the scope of R3 when Carl 
posted this in RebDev:

Regarding RIF, it will be in 3.1 (not 3.0) and it will provide a 
lot of the 

virtualness" needed for what I think your goal is (to have very large 
blocks for large datasets)."

From what I understood, RIF would have several advantages compared 
to a mezz-level approach :

- fast indexed storage engine in C (probably btree indexes as sqlite 
storage layer was considered for RIF)

- no LOAD/MOLD overhead (direct reading/writing of values in memory)

I wonder if it would preserve bindings? I can't see how...

I guess that the drawback is that RIF would probably be hidden in 
the Core part, so not open sourced.

Or it could be a UDT.

Probably if R3.1 takes another 3-4 years to be done ;-)

Just be thankful R3 is not a 100 year language like Paul Graham's 
ARC.

Steeve, as long as you're using a decent OS, any database should 
be kept all in memory regardless of size. Otherwise, you need a DBMS 
and a DBA.

Yes, some DBAs can do amazing things wih disks (disk tuning is a 
good selection criterion for a DBA)

Gabriele, it's your opnion not mine. we shouln't have to use DBMS 
or other external products (having to install them) for rebol applications 
which are standalone and not multi-user.

The file scheme of Rebol can do the job with a simple scheme wrapper 
(to hide complexity).

This has been requested by the community since long time ago. If 
you don't see the interest, good for you but let the other hope.

I have big long cutting knifes in my kitchen ! Who wants one ? It 
can help you finding who is right or wrong in a more funny (gore) 
manner ;-)

By the way, as Brian say, RebDev can stay as it is for a moment. 
It's simple Rebol data, so it could be easily transform to whatever 
new storage mechanism is needed when the time will require.

And, I don't think the client need to load the whole msgs db as most 
of the time (99.9%) user just read the lasts msgs. So It can be changed 
to cache last 10000 msgs and will only deal with the full db if user 
ask it to do (ie : search).


And to finnish, 24MB is not much for 100'000 msgs. I would not bother 
until it reach 200MB (It's what FF3 take after half a day of surfing) 
so I have 800'000 msgs left :-)

About protocols: Carl says that the basic foundation is not going 
to change, so he wants to see some protocols written. He wants to 
start with something simple like Finger. Here's your chance to contribute!