World: r3wp

So perhaps in 2 or 3 years, but it has excellent perfs currently 
according to him, But Carl doesnt use it in rebdev, very coherent...

Steeve, I read his last message as: rebdev doesn't need it yet as 
the current model would easily scale to over 100000 messages.

yes perfs will be here, but what about the memory overhead ?

his loading the full database in memory, in 2 different blocks (the 
second one rebuild and sorted each time you connect)

so i can make an estimation of the memory used for 100 000 messages, 
wait a minute...

I take 16 bytes for storing a value in memory (like for R2) i don't 
 know if it's more in R3.

An unique message has a structure (a block!) of a minimum size of 
80 bytes, so i take 110 bytes (for a small message of  30 characters 
in average).

for a block containing 100 000 messages we have 100 000 * (16 + 110) 
= 12 Mb

So for the 2 blocks , we have 24 Mb, just to deal with 100 000 messages.

It's growing fast, because rebdev store many events too (remove, 
private message, move, etc...).
and it will be connected to devbase too.

I think that  thousands of messages  and events will be added per 
month when the community will have access to rebdev.

So his current implementation will resist 2 or 3 months before reaching 
tens of Megabyte in memory usage.

And when it will go over 100 000 messages (in 4 or 6 month) the perfs 
will blow !!!!!

i made a wrong statement

the second block uses references not a copy of the message, so that 
the second block has a size of 100 000 * 16 = 1,5 Mb

So that it reduces the size of the memory overhead by 2.

i can do a prediction, it will end like with Altme, using 50 Mb of 
memory after one year of usage

And you have the light version currently, the console version.

It will be worst with the GUI if Carl doen't change his conception

A couple things to realize:

- The chat client can change easily, independent of R3, and does 
change daily.

- Our priority is on making the public release happen ASAP. Chat 
reaching its limits is months down the road at worst. We have time.

Right now the prioirty for the people usng R3 needs to be testing 
R3, in prep for its release.

For some of us, outside of work time of course.

The chat can be changed now because it's tiny enough, 

But 50% of his conception is wrong and has to be rewritten to deal 
with direct access in files instead of loading all in memory.

Yes i read carefully his script, 50% of it must be rewritten now...

More Carl will developp the GUI and the console version with wrong 
choices (to my mind) more it will be painfull to rewrite all later.

50% of it must be rewritten now
 - Not *now*, later. After the public release of R3. Priorities.

The chat database will be tiny enough for months. We aren't even 
done with the feature set. The internals can change easily. There 
are less than 700 messages, and we don't start having problems til 
100000. I think we have time for this.

I only reacted with your false statement, that it's easy to modify 
the chat to kill the memory overhead.
You didn't studied it carrefully enough.

I know there is no chance that Carl and other gurrus follow my advices.
But I know i'm right, i hope you will not regret it later...

I didn't dispute your statement, just its urgency. Priorities.

700 messages in 3 weeks, with 5 regular users, you're according my 
point.

Many eyes make bugs shallow. We need more developers, which means 
we need a public release. Priorities. We can fix chat *after* the 
public release. As it is, getting the file store in place is a more 
pressing need. That way we can release and track the source. Once 
the source is released we can make fixes as needed. The priority 
needs to be on testing R3 itself, not the chat client. We aren't 
even going to start the GUI chat client until after the R3 public 
release.

It's good to see a little realism about the scope of R3 when Carl 
posted this in RebDev:

Regarding RIF, it will be in 3.1 (not 3.0) and it will provide a 
lot of the 

virtualness" needed for what I think your goal is (to have very large 
blocks for large datasets)."

From what I understood, RIF would have several advantages compared 
to a mezz-level approach :

- fast indexed storage engine in C (probably btree indexes as sqlite 
storage layer was considered for RIF)

- no LOAD/MOLD overhead (direct reading/writing of values in memory)

I wonder if it would preserve bindings? I can't see how...

I guess that the drawback is that RIF would probably be hidden in 
the Core part, so not open sourced.

Or it could be a UDT.

Probably if R3.1 takes another 3-4 years to be done ;-)

Just be thankful R3 is not a 100 year language like Paul Graham's 
ARC.

Steeve, as long as you're using a decent OS, any database should 
be kept all in memory regardless of size. Otherwise, you need a DBMS 
and a DBA.

Yes, some DBAs can do amazing things wih disks (disk tuning is a 
good selection criterion for a DBA)

Gabriele, it's your opnion not mine. we shouln't have to use DBMS 
or other external products (having to install them) for rebol applications 
which are standalone and not multi-user.

The file scheme of Rebol can do the job with a simple scheme wrapper 
(to hide complexity).

This has been requested by the community since long time ago. If 
you don't see the interest, good for you but let the other hope.

I have big long cutting knifes in my kitchen ! Who wants one ? It 
can help you finding who is right or wrong in a more funny (gore) 
manner ;-)

By the way, as Brian say, RebDev can stay as it is for a moment. 
It's simple Rebol data, so it could be easily transform to whatever 
new storage mechanism is needed when the time will require.

And, I don't think the client need to load the whole msgs db as most 
of the time (99.9%) user just read the lasts msgs. So It can be changed 
to cache last 10000 msgs and will only deal with the full db if user 
ask it to do (ie : search).


And to finnish, 24MB is not much for 100'000 msgs. I would not bother 
until it reach 200MB (It's what FF3 take after half a day of surfing) 
so I have 800'000 msgs left :-)

About protocols: Carl says that the basic foundation is not going 
to change, so he wants to see some protocols written. He wants to 
start with something simple like Finger. Here's your chance to contribute!

Carl: "most of the scheme structure has been solid and unchanged 
for more than a year. The main disruption was Unicode, but that's 
been done for a while too. I think we should pick a few very simple 
protocols, maybe one like Finger, and j ust show how they are done... 
as an example."

No Didec to work as you wish, rebdev client as to be rewritten completly, 
it's not a minor change.

What was the proposition for change for RebDev? If it would be upon 
me, I would accept SQLIte as small, robust RDBMS for R3, via plugin 
(to stay external). Wait, we don't have plugins :-)

Steeve, check your private messages.

Steeve, there isn't much code in the RebDev client, by design it 
wouldn't be much work to completely rewrite it. Everything at that 
scale is a minor change - that is the nature of the scale.

In any case, we can do this later.

Steeve: there are opinions, and there are facts. It is a fact that 
you can't be faster / more efficient than the OS. It is your opinion 
that you want to be less efficient.

Gabriele, you're out of my point, i don't say it can be faster than 
the OS.

Today I read acrticle about DB  market. There are going to be some 
specialised DBs in future, and the needs diversify and scenario usages 
too. E.g. IBM has SolidDB in-memory DB, which uses async syncrhonisation 
to DB2 or Informix or something like that ...

Yes it's cool, it's something i try to do with Virtual blocks (blocks 
which are synchronised with files). it"s a tiny scheme (script size 
less than 5 kb).
It doesn't allow sorted blocks currently but it could be.

I am also sure DB's will specialise and already do (couchDB, bigtable..), 
and I plan to make few simple specialised storages to for some my 
projects.

Steeve, exactly as I said, it is your opinion that you don't want 
to be efficient. It is my opinion that I want most of the time.

We have not the same defintion of efficience. 

The efficience of an application is a grid which contains several 
criteria. 

When you choose an implementation. All these criteria can't  have 
the best value together

Hiding this complexity by saying i'm efficient, and you are not, 
without saying you choose to lower some criteria, only reveals one 
thing, you're pretentious.

Can anyone give a range of time where rebol 3 will be released? If 
not, can I just be notified by email when it is?? I'm sick of logging 
onto rebol.com and checking the blogs. I've been doing this for at 
least a year and I'm sick of it.

sick of it

  seems a bit extreme.... there are other good reasons to check in 
  here.


But that aside, how about simply setting up an RSS feded off Carl's 
blog, and set it to pull once a month or so?

Feed.