World: r3wp

There is memory allocation optimisation both for value slots (as 
seen in the growing block of integers example above), and elsewhere.

So a single allocation is not enough to deriive the underlying algorithm.

And a binary is just a type of string, so yes, the value slot contains 
a pointer to the actual data.

Value slots are always 16 bytes long in current REBOL versions (Says 
Gabriel).

It seems reasonable to assume this will increase if REBOL goes 64bit 
(speculates Sunanda)

so then a block of integers  ie: [1234432   345   45345   5435   
2345  5435353]  .. .
is the most efficient way to store?

Storing a bunch of integers in a binary should be more efficient, 
only one value slot used, and each integer takes only 32bits.

I guess that's my point.. i need to use 32 bits to store a single 
integer??

What precision do you need in your integers ?

You could make 8-bit or 16-bit, or ... 13-bit integers if you wanted. 
It's more work, but possible.

i can use bits.. where 00= 0   01 = 1  10 = 2 .. 11 = 3  etc.

Yes, how many bits per integer are needed ? What's the highest number 
? Any negative numbers allowed ?

( 0 = 0 rather)

(ie. how many unique integer values are needed ?)

no negatives..  and a MAX of 32 bits is more than enough for the 
largest number

A value slot is 16 bytes, so  a single integer takes 16 bytes -- 
it has to live in a value slot.

16 bytes.. that seems large

I guess that's where the number of words limitation kicks in?

No, the slot size is fixed so it can fit a whole lot of different 
types in it, eg. time! values.

word limit is something else -- to do with the number of unique (across 
all contexts) words. Not related to value slots as far as I know.

Im thinking I should do this in Assembly  ;)

Well, what is your largest number ?

ahh .. ok..  that's a fairly heavy price to pay (relatively speaking) 
just so I can store non binary data types

That dictates the number of bits needed to represent all your numbers.

For compact representation of large integer sets, I often uses the 
format:
    [10x4 50 76x1000] 
ir REBOL pairs! (each taking 1 value slot) meaning
     [10 11 12 13  50  76 77 78 ..... 1076]
This works for me for large blocks of semi-sparse integers.

There is a script for handling this format at REBOL.org -- look for 
rse-ids.r

Im guessing my largest number is probably around 2 million... approx

You have to know what your largest number is. Otherwise your software 
will break with overflow errors.

well..  say 21 bits

Well 24bits (3 bytes) --> 2^24 = 16777216 unique values.

Ok, so you could pack each 21 (or 24) bits into a binary.

No wastage.

so is that a block of binaries then?

No, a single binary.

yeah for each... but i need to group them

You have to write the access methods to index into your binary though.

(So for that I recommend just using 3 bytes.)

Ok, well you can have several binaries, why not ?

Anton, I think, is suggesting you do something like this:
my-list: make binary! 25000

Then use insert or append to store 3-byte values to the binary string 
you have created.

That way you need only 1 value slot plus the length of the binary 
string.

oops -- anton typed faster than me :-)

ahh ok..    ie: [a4b   4°¢  ÑAg]   etc ?

That would be a block of binary....each binar would take a slot
Anton is suggesting this sort of approach:
x: make binary! 25000
>> loop 18 [insert x to-char random 255]
== #{5E218289FC8B65B86A1C597C232F9E8C79}
Just one binary string to hold the data.

is there a function to convert an integer to 3 bytes?

ahh.. nice

No, you must make the functions to convert  3-byte-int <--> integer!

>> bin: #{000005000006000007}

>> repeat index 3 [bin-int: copy/part skip bin (index - 1 * 3) 3 
print [index mold bin-int to-integer bin-int]]
1 #{000005} 5
2 #{000006} 6
3 #{000007} 7

binary! is indexed per byte, so if we use 3 bytes per value, we must 
multiply the index by 3 to find the correct value.

We must do this in our functions which should allow retrieving, changing 
and inserting the value.

would there be a performance trade off as opposed to just a block 
of integers? (finding/ appending etc.)

Note that converting integer <-> binary is a little tricky. But we 
can use a struct! (re-use a shared struct for optimal performance).

Yes, there is lost performance. Obviously, you must go via your access 
functions each time you store and retrieve.

(This is addressed by Rebol3's vector! datatype, which stores different 
values this way.)

well, performance is the primary concern