r3wp [groups: 83 posts: 189283]
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

World: r3wp

[Core] Discuss core issues

Terry
20-May-2007
[8156]
yeah... I think the conclusion is ... don't worry about the number 
of bytes (and thus mem) when using plain integers with my index block, 
as it's much smaller than the dictionary anyways.. and unless Im 
shown otherwise, the crawling of it (find/ foreach, append) should 
be about as fast as any other method, right?
Anton
20-May-2007
[8157]
It should be faster and simpler to just use integers. When you want 
to cut down the size of your 2 million integers (=30MB), you can 
then look at implementing 3-byte integers packed in a binary.
Terry
20-May-2007
[8158]
exactly
Anton
20-May-2007
[8159x2]
Here's a couple of conversion routines.
struct: make struct! [int [int]] none

; convert integer -> 3-byte binary

integer-to-3-byte-binary: func [integer [integer!]][
	struct/int: integer
	copy/part third struct 3
]

; convert 3-byte binary -> integer

binary-3-byte-to-integer: func [int3 [binary!]][
	struct/int: 0 ; just make sure all bytes are zero
	change third struct int3
	struct/int
]
	
; test

my-bin: integer-to-3-byte-binary 2000000
;== #{80841E}
my-int: binary-3-byte-to-integer my-bin
;== 2000000
Terry
20-May-2007
[8161x3]
ok.. that's cool.. thanks
Even though each integer uses 16 bytes, there's some compaction by 
using the smallest integers with the most commonly used dictionary 
strings
I'll add these functions to the dictionary.. 

dict: [{integer-to-3-byte-binary: func [integer [integer!]][
	struct/int: integer
	copy/part third struct 3
]}]


now whenever i want to use that function.. i can represent it with 
a single integer.. ie:  do pick dict 1
Oldes
20-May-2007
[8164x2]
Terry, I would not make own binary storage as there should be new 
datatype dictionary! in R3 which is exactly what you need... if I 
understand it well
I cannot see hash! in this list http://www.rebol.net/r3blogs/0076.html
so it will be probably replaced
Terry
20-May-2007
[8166]
and that could be reduced further.. 

dict: [

{fetch: func [index][do pick dict index]}

{integer-to-3-byte-binary: func [integer [integer!]][
	struct/int: integer
	copy/part third struct 3
]}

]

do pick dict 1
fetch 2
my-bin: integer-to-3-byte-binary 2000000
Oldes
20-May-2007
[8167]
The new vector! datatype will be usefull as well to store large block 
of same data type
Terry
20-May-2007
[8168]
Im happy with R2 ;)
Oldes
20-May-2007
[8169]
yes... you can make it in R2 using index: [1 2 3]  and for R3 you 
just replace:  index: make vector! [integer! 24 [1 2 3]]
Terry
20-May-2007
[8170x3]
Now as for compressing the dictionary.. it seems that smaller strings 
grow with compression?
>> a: compress "a"
== #{789C4B04000062006201000000}
>> a: compress "aaaa"
== #{789C4B4C4C4C040003CE018504000000}
>> a: compress "aaaaa"
== #{789C4B4C04020005B401E605000000}
Oldes
20-May-2007
[8173x2]
compress is uzing zlib compression... just insetad of checksum at 
the tail (last 4 bytes) is used size of the source string
the size is used to make a result buffer for decompress:
>> decompress rejoin [#{789C4B4C} #{04020005B401E6} #{00000000}]
** Script Error: Not enough memory
>> decompress rejoin [#{789C4B4C} #{04020005B401E6} #{FF000000}]
== "aaaaa"
Dockimbel
20-May-2007
[8175]
Terry, your 'integer-to-3-byte-binary conversion function is dependent 
on platform endianness (third struct! gives you an endian-dependent 
result), watch out for that.
Anton
20-May-2007
[8176]
You can blame me for that, I wrote it quickly.
btiffin
21-May-2007
[8177]
Terminology question;  I know I could probably RTFM, but sometimes

Ask A Friendly Human is more fun.  What is the correct terminology 
for the global

REBOL context.  I'm describing (or trying to at least) the parse 
dialect "copy" versus

the REBOL "copy".  Is there a one word term for the "no context" 
context?  Or is
the REBOL global namespace
 good enough (and not too confusing to new rebols)?
Anton
22-May-2007
[8178]
Yeah, we all call it the global context.
btiffin
22-May-2007
[8179]
Thanks Anton.
Jerry
22-May-2007
[8180]
Is there a function which can flat a block. For example,
>> FLAT [ 1 2 [ 3 4 ] 5 6 ]
== [ 1 2 3 4 5 6]
Graham
22-May-2007
[8181]
probably not what you want ...

>> to-block form  [ 1 2 [ 3 4 ] 5 6 ]
== [1 2 3 4 5 6]
Chris
22-May-2007
[8182]
flatten: func [block [any-block!]][

    parse block [

         any [block: any-block! (change/part block first block 1) :block | 
         skip]

    ]
 
   head block
]
Henrik
23-May-2007
[8183x2]
and with greater speed but less accuracy:
>> load form [1 2 [3 4] 5 6]
and now I should of course have read what Graham posted...
BrianH
23-May-2007
[8185]
Don't assume that the parse is slow - you may be surprised.
Anton
23-May-2007
[8186]
Yes, and I wouldn't recommend the LOAD FORM way, it looks too brittle.
Henrik
24-May-2007
[8187x2]
does SORT/COMPARE not work in combination with /SKIP? it seems to 
happily ignore /SKIP here.
seems that using /ALL does not help.
Gregg
24-May-2007
[8189]
>> b: [1 2 3 4 5  5 4 3 2 1  7 6 5 4 3  3 4 5 6 7]
== [1 2 3 4 5 5 4 3 2 1 7 6 5 4 3 3 4 5 6 7]
>> bb: sort/compare/skip copy b 2 5
== [1 2 3 4 5 5 4 3 2 1 3 4 5 6 7 7 6 5 4 3]
>> bb: sort/compare/skip copy b 4 5
== [5 4 3 2 1 1 2 3 4 5 7 6 5 4 3 3 4 5 6 7]
Henrik
25-May-2007
[8190]
what if the record size is 2, consists of a binary and an object 
and you want to sort on a value in the object?
Sunanda
25-May-2007
[8191]
Here's one way -- though it assumes (for simplicity) that the binary 
is a string of equal length in all keys:


data: reduce ["z" make object! [key: 1] "y" make object! [key: 2] 
"z" make object! [key: 2]]
sort/all/skip/compare  data 2 func [a b][
     return (join a/1 a/2/key) < (join b/1 b/2/key)
    ]
probe data
Henrik
25-May-2007
[8192x5]
hmm... funny, I did exactly that code, but it would never recognize 
the b object in the compare function. will have to try again...
oh wait a minute. I see now :-)
I guess I don't after all. I still can't get it to recognize 'b. 
If I try to probe 'b inside the function, sorting just stops. probing 
'a works fine.
and the block checks out fine. it is properly arranged.
found the bug. (and I need more coffee)
Sunanda
25-May-2007
[8197]
There were problems with using all three refinements [ /all/compare/skip 
] together in earlier versions of REBOL. I believe they are fixed 
now.
Henrik
25-May-2007
[8198]
I think the compare feature is just a bit underdocumented.
Rebolek
25-May-2007
[8199x2]
Script error: REPLACE has no refinement called ONLY
I can use CHANGE/ONLY FIND ...
What do you think, skould I RAMBO it as a wish?
Henrik
25-May-2007
[8201]
rebolek, yeah, I think it would make sense to have that.
Rebolek
25-May-2007
[8202]
It's mezanine, so I'll try to figure out some fix and post it.
Henrik
25-May-2007
[8203]
cool
Rebolek
25-May-2007
[8204x2]
Hm, was easier than I thought :))
Posted