World: r3wp

Yes Pavel, that is the way it looks to me also regarding the vector.

Yeah, vector! doesn't work yet. There are a lot of tickets for vector!, 
all deferred. That to-block bug is not there, but not unexpected.

Brian, I think we need to address the lack of list in R3.  Maybe 
a list like handling of block data that still returns a block.

at the performance level of list.

I think that would be an excellent thing to address with a user-defined 
datatype.

People in general didn't use list!, because of the bugs and because 
block! was good enough for most uses. We haven't felt its lack for 
a year now. Most of the advantage of list! in regular code was handled 
by allowing insert and remove at the head of a block! to expand or 
contract the block, just like it does at the end, without a block 
copy. Just by making block! more efficient for inserts and removes, 
we have made list! even less necessary.

For that matter, you can make blocks even faster by getting rid of 
list!, just by enabling more optimizations.

The real advantage of list! wasn't the speed though, it was the alias 
persistancy. That was the source of the bugs in list! too.

I tested vector! extensively about one and half year ago. Not much 
changed, I think. The BIGGEST problem with vector! is that it's zero 
based.

Trust me, vector! has bigger problems than that right now. I expect 
that it will not end up 0-based though, even with all of the advantages 
that 0-based vectors give us. It is likely that we will instead have 
0-based versions of PICK and POKE that work on all series.

Brian, all the vector! tickets are mine, so I know very well about 
vector! problems :) I'm not sure about zero based PICK and POKE, 
I would probably prefer PICKZ and POKEZ, but I can get used to it.

Yes, those would be good names for the aformentioned 0-based versions 
of PICK and POKE. PICK0 and POKE0 have also been suggested as names, 
though I prefer *Z. They could be natives (or mezzanines?) that call 
PICK and POKE internally. I think the only reason we don't have them 
already is that PICK is an action!, and the naming disagreement.

Are blocks stored in memory as linked fragments?

no

How can inserts at the head and tail be done without moving the block, 
then?

you asked for blocks not for lists

Yes, Brian talks about blocks above

where ?, i don't think so

I´m not talking about links for each item, but for fragments determined 
during use

uh ? he just said that the performance have been improved with block 
operations, so that lists are no more requested. He didn't say that 
blocks are working like lists now

I didn´t say that, either. But there must be something going on if 
you can insert without moving the memory

where did he said that ?

One screen up

this ? "Just by making block! more efficient for inserts and removes, 
we have made list! even less necessary."

The only way I can think of to keep the memory contiguous is advanced 
use of the hardware MMU, which would lead to partially used memory 
pages at the start and end of each block

>> a: []
== []

>> dp [append a 1]
== make object! [
    timer: 32
    evals: 11
    eval-natives: 4
    eval-functions: 1
    series-made: 2
    series-freed: 1
    series-expanded: 1
    series-bytes: 464
    series-recycled: 0
    made-blocks: 1
    made-objects: 0
    recycles: 0
]


as you can see the block has been expanded, which means copied in 
another place

That would contradict Brian´s statement even about R2

Don't see when he stated that, anyway we just have to wait his return

¨People in general didn't use list!, because of the bugs and because 
block! was good enough for most uses. We haven't felt its lack for 
a year now. Most of the advantage of list! in regular code was handled 
by allowing insert and remove at the head of a block! to expand or 
contract the block, just like it does at the end, without a block 
copy. Just by making block! more efficient for inserts and removes, 
we have made list! even less necessary.¨

So it already worked that way for appends in R2

Is that profiling accurate Steeve?

>> dp [a: [] b: []]
== make object! [
    timer: 15
    evals: 12
    eval-natives: 3
    eval-functions: 1
    series-made: 1
    series-freed: 0
    series-expanded: 0
    series-bytes: 432
    series-recycled: 0
    made-blocks: 1
    made-objects: 0
    recycles: 0
]

>>

Shouldn't made-blocks indicate 2 blocks were created?

no to Kaj, no to Paul :-)

To kaj: Brian is talking about lists, not blocks. Your test do not 
append or insert data in the input blocks, it's not relevant.

To Paul: No, the 2 blocks are created before entering in the dp function, 
dp doesn't create them.

>> dp []
== make object! [
    timer: 14
    evals: 8
    eval-natives: 3
    eval-functions: 1
    series-made: 1
    series-freed: 0
    series-expanded: 0
    series-bytes: 432
    series-recycled: 0
    made-blocks: 1
    made-objects: 0
    recycles: 0
]

see, a block is always created by dp

If the block needs to be expanded because there isn't enough allocated 
the system does a block copy. If there *is* enough allocated, as 
when you preallocate using make block!, then the system doesn't have 
to do a block copy. That is R2 and R3.


What is new in R3 is that the "head" pointer of a block doesn't have 
to point to the beginning of  the allocated memory, just like the 
"tail" pointer in R2. This means a remove from the head of the block 
just shifts the pointer over one in R3, while in R2 you had to copy 
over the rest of the block contents to shift it towards the head 
of the allocated memory. Preallocated memory can also exist before 
the head of the block contents in R3. This means that there is no 
difference in overhead between inserts at the head or the tail of 
a block in R3.


In theory, inserts inside the block in the first half could be more 
efficient because you would only have to shift from the nearest end, 
not the tail. I don't know whether this optimization has been implemented.


Block operations in general could be faster because with no list! 
type we wouldn't have to special-case as much code, so we could make 
our code much faster through more aggressive optimization.


Btw, I submitted a tweak to DP to make it more accurate by subtracting 
its own overhead. It still has some variance though - have to tweak 
the native to fix that. Plus there is the extreme variance caused 
by Windows.

I hope this answers your question, Kaj.

Back to map Brian's note words-of returns all keys is easy to overlook, 
but this is only way how to traverse thru map (when no next is at 
hand) THX Brian

There is a proposal (looking likely to be implemented) to have FOREACH 
work on object! and map! types. The word list syntax would be restricted, 
but you could do your traversal that way. In the meanwhile you have 
WORDS-OF to get the keys in a block, VALUES-OF to get the values 
in a block, BODY-OF object! to get both in a block (map! proposed 
too) and TO-BLOCK of map! to get both in a block. It works, but the 
FOREACH proposal would create fewer intermediate blocks.

THX again, is it possible something like "select" ie <,>,between 
etc?

Of course FOREACH of map! would operate in the order that TO-BLOCK 
map! would return the keys and values at that moment. In the long 
run you would have to consider the order of FOREACH map! to be non-deterministic 
between calls. The map! type has no inherent ordering, so position 
and sorting are meaningless for it.

hashed doesn't mean ordered?

It doesn't. SELECT works on object! and map!, but there is no comparison 
except equality for those types.

Hashed means indexed (in the database sense), not ordered.

my fault I missmatch rebol select with SQL like select

IE if you expect quite large number of keys would you have any chance 
to "narrow" like "sql between"

Map! is not indexed in the REBOL sense, ironically enough: no position.

OK now I have more clue, all this is toward key/value datastore preferably 
diskbased something I believe Paul is trying in R3

REBOL is a lower-level language than SQL. No set operations (for 
map!, ar least). Those kinds of SQL operations need to be done procedurally 
in REBOL, using block copies and such.

We will have REBOL Indexed Files (RIF) for the disk-based datastore.