callbacks
[1/24] from: moliad::gmail at: 29-Oct-2007 18:33
hi guys,
I have been trying to use callbacks today and well, its a totally random
experience so far.
my rebol func actually gets called, but its interface, how I spec it, is
irrelevant... whatever I write nothing changes.
I can even write a call-back spec which is empty, change the types,
anything... the rebol func being called never reacts differently.
anyone done anything with callbacks and can give me a few pointers?
another strange issue, in some examples, I see callbacks defined like so:
test: make routine! [
a [int]
b [int]
c [callback! [int int return: [int]]]
return: [int]
] test-lib "test"
but I get an error about the 'CALLBACK! word.
If I replace it with 'CALLBACK then the error goes away, but what gets
called in the actuall callback is oblivious to anything I write in the spec
. indeed I can even add as many args in the rebol func and they just get
pumped with random values (probably raw mem addresses). The other strange
thing is, in my specific useage, the first and second value which get passed
to the rebol func never change, and type also has nothing to do with routine
spec nor function spec!
help!
I'm using the 2.7.5 view pro (almost the last unofficial release ;-) on
windows
-MAx
[2/24] from: moliad::gmail at: 29-Oct-2007 18:46
well,
seems I sent this post 5 minutes too soon.
As part of my last ditch attempts, I just started moving thing around
related to my dll's library documentation and includes files, expecting
crashes and MS errors... and curiously, it seems, something is not calling
the proper callback in the first start... so it seems the issue does not lie
with REBOL finally...
if I change the spec of a completely unrelated callback, the interface is
changing... so it seems, the dll is not calling what its supposed to,
according to its specs and its init funcs...
thanks anyways.
-MAx
On 10/29/07, Maxim Olivier-Adlhoch <moliad-gmail.com> wrote:
[3/24] from: moliad:gm:ail at: 29-Oct-2007 20:21
ATTENTION:
This is a report about a bug I am having with the callback system within
rebol. Your experience may be different, but this just happened to me. Its
good to know, so I am posting this on the Mailing list, so someone searching
about callbacks might find it.
its a long and detailed post, sorry, but this just couldn't be explained
properly within a simple 3 line post without quite a few questions popping
up .
note, the code below is simplified and does not run by itself, it just
illustrates the exact problem without clutter.
here we go:
this code (the extended version of this code) crashes whenever the func-a is
called by the external lib!
;-----------------------------------------------
rebol []
my-lib: load/library %some-bogus-lib
do-init: make routine! [
"pass the supplied arguments as callbacks to lib."
number [long] "this shall always be set to 0 (void* = NULL)"
spec-a [callback [long ]]
spec-b [callback [char]]
spec-c [callback [char*]]
] my-lib "doInit"
func-a: func [value [integer!]][print value]
func-b: func [letter [char!][print letter]
func-c: func [sentence [string!]][print sentence]
do-init 1 :func-a :func-b :func-c
;-----------------------------------------------
I was getting an "illegal memory read at address 0x00000001", hard core
windows crash. (but thankfully not a blue screen :-)
I thought I had my callbacks in bad order, but no, it would effectively fire
off the wrong callbacks if I changed the order supplied to do-init, so I had
to put things back... I was a bit stumped.
then I looked at the address MS was giving me... hum ...the number looked
too clean... like an enum value, so I could figure out a callback value
actually was being supplied somewhere.
I tried many things, tripple-checked everything but finally, to have the
code working, I had to switch the callback specs, without changing the
actuall callback submission order, cause remember, its actually calling the
proper callback function itself!
what's happening, is that its in fact (for some VERY ODD reason) using
spec-c and being a pointer to string, its trying to reference mem addres at
value!
within the lib, func-a really IS being called, its just REBOL which is
trying to apply the wrong trampoline specification, in order to convert the
mem address into a rebol string. Cause in fact, func-a shouldn't be
referecing the address, it should just use the integer directly and supply
that to func-a.
the above illustrates the change relative to the example above.
;-----------------------------------------------
rebol []
my-lib: load/library %some-bogus-lib
do-init: make routine! [
"pass the supplied arguments as callbacks to lib."
number [long] "this shall always be set to 0 (void* = NULL)"
spec-c [callback [char*]]
spec-b [callback [char]]
spec-a [callback [long ]]
] my-lib "doInit"
func-a: func [value [integer!]][print value]
func-b: func [letter [char!][print letter]
func-c: func [sentence [string!]][print sentence]
do-init 1 :func-a :func-b :func-c
;-----------------------------------------------
so beware, there is some oddity with the callback system.
now I'm about to use Ladislav's get-mem powertool, cause I am receiving
variable length arrays of varying types.. :-)
I should be having loads of fun tonight :-)
-MAx
[4/24] from: anton::wilddsl::net::au at: 30-Oct-2007 15:10
Hi Max,
If I remember correctly, callbacks have less supported
types available to use than routines.
I'm not sure, but string might not be supported,
so that might be the problem.
And yes, you could probably work around it using
get-mem.
Regards,
Anton.
Maxim Olivier-Adlhoch wrote:
[5/24] from: moliad:gmai:l at: 30-Oct-2007 0:44
hi anton,
note that within the routine, I am using char* which is one of the 3 types
supported within callbacks... within the callback func, whatever you put as
the argument type is irrelevent. it totally ignores the interface you try
to impose.
I just discovered that the callbacks also ignore the error trapping
completely, and this is very hard to detect! nothing happens, the function
just quits silently... I'll try putting an attempt within the func
another detail I have been trying to understand is that the GC is totally
trashing the stack or something while callbacks are being called.
Unless I use recycle/off, at the first recycle occurence... rebol crashes.
once I am done with the callbacks I can put recycle/on, but at that point, I
dont reclaim any ram which was lost. in fact, it really seems like the GC
in rebol is often useless, the moment something makes it peak. often, an
app just grows to incredible sizes in the simplest of uses, even without
view.
-MAx
On 10/29/07, Anton Rolls <anton-wilddsl.net.au> wrote:
[6/24] from: moliad:gm:ail at: 30-Oct-2007 1:35
hi all,
well, seems the instability really is related with the GC being sort of
pre-emptive hehe it thinks the ram is not usefull anymore... but we are
inside the callback itself and bam... it crashes. the moment we play with a
callback's arguments, setting one to none or even within a function which
is called by the callback...
so I had to allocate a Global word and set one of the callback's values to
that word, as the first operation of the callback... I even had to convert
the interface, set one to char* so rebol's GC would track its pointer use...
from that point on, the whole system became totally stable, even with
recycle/on being called repeteadly.
because I set one of the interace's values to a char* I had to extract the
address of the string and use two other methods from peekpoke to retrieve
the original value which was a double float array to start with!
so, overall, the callback is not for the faint of heart, but if you are
patient, and tweak here and there, you can eventually get you want. note
that you will not do much, if you do not also use the EXCELLENT
peekpoke.rpower library over at ladislav's site. Cause in many cases,
the callbacks
use much more complex datastructures than what is available out of the box
in rebol.
I am able to convert double float variable length arrays being sent to me
and other nice things, but without the peekpoke lib, I'd be quite lost
indeed.
-MAx
On 10/30/07, Maxim Olivier-Adlhoch <moliad-gmail.com> wrote:
[7/24] from: moliad::gmail::com at: 30-Oct-2007 2:59
I'm about to go mad!
turns out the char* trick below, also trips up the address when some values
are being passed. but not always. So I had to stop that. It also seems
further testing might invalidate my previous observations.
it seems the use of the memory? function within the peekpoke.r file causes
the GC to go wild and eventually crashes rebol when callbacks are being
used. so far this is pretty constant.
if I do recycle/off, no crash occurs, if I let recycle/on and remove the
call to memory? there are no crashses either (but then I'm pretty much at
step 0!).
damn, this is just too frustrating!
the damned thing, is that a part from the GC crashing rebol, all is working
flawlessly.
any ideas/experience welcome
-MAx
On 10/30/07, Maxim Olivier-Adlhoch <moliad-gmail.com> wrote:
[8/24] from: sqlab::gmx::net at: 30-Oct-2007 9:59
Did you try to use char-array in order to be on the safe side ?
http://www.rebol.com/article/0141.html
Maxim Olivier-Adlhoch wrote:
[9/24] from: moliad::gmail::com at: 30-Oct-2007 14:36
hi,
I saw that ... but althouh it might fix the issue, my problem is that I have
no idea about the length of the returned data before the call... and it can
be huge. I could have to make 100kb or even larger copies.
in the end, if that's the only thing that makes the system stable, I'll end
up doing this (but I still have to check if it really alleviates the
problem, cause its not totally clear why and how the GC trips up like that).
I'm still open to ideas about why this all happens in the first place and
how to solve it.
-MAx
On 10/30/07, sqlab <sqlab-gmx.net> wrote:
[10/24] from: lmecir::mbox::vol::cz at: 31-Oct-2007 23:50
Maxim Olivier-Adlhoch napsal(a):
> hi,
> I saw that ... but althouh it might fix the issue, my problem is that I have
<<quoted lines omitted: 16>>
>>>
>>
a couple of notes:
1) CALLBACK! versus CALLBACK - this is a recent R2 change that might
have been unintended
2) MEMORY? versus GC: the MEMORY? function does not protect the memory
against GC. You need to make sure the memory is protected by other
means. (see http://www.rebol.com/docs/library.html#Garbage )
HTH
-Ladislav
[11/24] from: moliad:gmai:l at: 31-Oct-2007 23:35
aha, I was hopping you'd pop in.
I was thinking today, that the GC might be trying to de-allocate memory from
the struct, which is not its own.
maybe if I reuse the structs used withing the memory? func or nullify the
pointer just before leaving the memory? func.
I'll checkout the link you gave me to get other ideas.
-MAx
On 10/31/07, Ladislav Mecir <lmecir-mbox.vol.cz> wrote:
[12/24] from: moliad:gma:il at: 1-Nov-2007 0:14
well, that was it.
I have read that little paragraph a while back, but had forgotten about the
[save] option on struct spec.
its just badly placed within that doc... its part of the routine!
description... it really should have been put under the struct! or maybe as
part of its own topic "Preventing Garbage Collection"
It was pretty easy to update address-as-struct so it creates a safe struct.
(adding it in the memory? struct was not enough for some reason).
thanks ladislav.
-MAx
On 10/31/07, Maxim Olivier-Adlhoch <moliad-gmail.com> wrote:
[13/24] from: moliad:gm:ail at: 1-Nov-2007 0:24
hi Lad,
although its fixing the crashing, safe structs now act as a huge memory
leak. its about half the amount of doing recycl/off... in my tests so far.
-MAx
On 11/1/07, Maxim Olivier-Adlhoch <moliad-gmail.com> wrote:
[14/24] from: lmecir:mbox:vol:cz at: 1-Nov-2007 6:49
Maxim Olivier-Adlhoch napsal(a):
> hi Lad,
>
> although its fixing the crashing, safe structs now act as a huge memory
> leak. its about half the amount of doing recycl/off... in my tests so far.
>
> -MAx
>
Certainly. You should protect the memory "using other means".
The ADDRESS-AS-STRUCT function as written by me does not protect the
memory against GC intentionally. It should not be necessary to protect
the memory against GC, if the memory is already protected "by other
means". Example:
a: head insert/dup copy "" "1234567890" 4
b: head insert/dup copy "" "abcdefghij" 4
ma: memory? string-address? a 40
mb: memory? string-address? b 40
; no problem occurs, since
; MA is protected by 'a
; MB is protected by 'b
recycle
probe as-string ma
; unprotecting MA
a: none
; now this is a problem, since MA isn't protected
recycle
probe as-string ma
As you can see, the problem was caused by the fact, that I "unprotected"
the memory, not in the ADDRESS-AS-STRUCT function. Warning! Another way
how to unprotect MA might have been just to do e.g.:
insert tail a "1234567890"
, since it causes memory reallocation.
Does this make sense to you?
-L
[15/24] from: moliad:gma:il at: 1-Nov-2007 0:57
Hi Ladislav,
yes, I understand the whole issue quite clearly. I know exactly what is
going on. I was just mentioning, that the [save] option when used in struct
is akin to half the memory leak of shutting the GC off completely.
in this case, REBOL was trying to free ram for which, it was not the
original allocator.
IMO it seems the space used for the original struct is being lost somewhere,
since we are changing its pointer AFTER its been allocated, and telling
REBOL to not recyle it after.
-MAx
On 11/1/07, Ladislav Mecir <lmecir-mbox.vol.cz> wrote:
[16/24] from: moliad:gma:il at: 1-Nov-2007 1:01
hi all,
this is the FINAL mail about this topic from me, and its a bit surprising.
Its a happy conclusion, at least.
I did a merge of two of the functions within peekpoke.r added a few tricks
to it and called it memcopy. I include it here and I hope ladislav will
include it withing the official distribution. there are two reasons why
this function is a complement to what is already there:
* it solves the issue of the [save] being a memory leak completely. in fact,
I do not use [save] in the function at all. this was the solution I
intended to test tonight as a fix to the GC issue. and its completely and
100% GC friendly. In fact, the GC really does recycle the structs itself
and the original memory address is in fact allocated by an external source
(best of both worlds).
* its MUCH faster than calling the memory? function using [save] option
within struct even though the basis of the memcopy function uses all of the
same code! don't ask me why, I've been splitting my head enough for the
last 3 days. ;-)
in my tests which use this new function, the application in a tight loop
(with no console activity) doing requests on the relavance database (shown
at devcon 2007) was 20% faster, but this includes the actual query itself,
which takes most of the time, so its possible that the function by itself,
is like twice as fast (I don't have time for further benchmarking).
so, Ladislav, if you want to add it, do so, I'll be happy if I did a little
part in your powertool library :-)
-MAx
On 11/1/07, Maxim Olivier-Adlhoch <moliad-gmail.com> wrote:
[17/24] from: moliad:gmai:l at: 1-Nov-2007 1:04
oops, forgot the func... here it is :-)
note, this is a stand-alone function, it does not expressly rely on any of
the funcs within peekpoke.r
memcopy: func [
{copies a region of ram given at address with sizeof length.}
address [integer!]
length [block!]
/local struct-struct buffer addr
] [
address: make struct! [address [integer!]] reduce [address]
struct-struct: make struct! compose/deep [ struct [struct! [[save]
(spec)]]] none
addr: copy third struct-struct
change third struct-struct third address
buffer: copy third struct-struct/struct address head insert/dup copy []
[c [char]] length
change third struct-struct third addr
buffer
]
-MAx
On 11/1/07, Maxim Olivier-Adlhoch <moliad-gmail.com> wrote:
[18/24] from: lmecir:mbox:vol:cz at: 1-Nov-2007 7:20
Maxim Olivier-Adlhoch napsal(a):
> Hi Ladislav,
> yes, I understand the whole issue quite clearly. I know exactly what is
<<quoted lines omitted: 6>>
> REBOL to not recyle it after.
> -MAx
Are you telling me, that it is a C routine, which allocates the memory?
If that is so, then Rebol GC isn't likely to deallocate it. My suspicion
is, that something else happens:
1) Your Rebol code allocates some memory and "gives" it to the C library.
2) Your Rebol code "unprotects" the memory it allocated for the C library.
3) The C library fills the memory by some data and gives the memory back
to a callback function.
4) The GC frees the memory.
5) Your callback function accesses the collected memory causing the crash.
If you want to just copy the contents of the memory, then this is the
simplest/fastest way:
memcopy: func [
{copies a region of ram given at address with sizeof length.}
address [integer!]
length [integer!]
/local struct
] [
address: make struct! [address [integer!]] reduce [address]
struct: make struct! compose/deep [s [char-array (length)]] none
change third struct third address
struct/s
]
The advantage of this is, that the memory copy *is* protected, but there
is still a chance, that the GC may release the memory sooner, than the
MEMCOPY function copies it, in which case the crash still may happen.
[19/24] from: gregg:pointillistic at: 1-Nov-2007 10:11
Hi Max,
I did something similar a few years ago. Not callback related, but
filling a struct with data allocated by an API (IIRC).
get-dereferenced-data: func [
{Given a pointer to memory, copy the target data into a REBOL struct.}
pointer [struct!] "LPINT structure whose /value is the data pointer"
struct-def [block!] "The struct you want returned with data"
/local struct data orig-pointer result
] [
struct: make struct! compose/deep/only [ ; make wrapper struct
sub [struct! (struct-def)]
] none
orig-pointer: third struct ; store original inner pointer
change third struct third pointer ; change inner pointer to ref'd data
data: copy third struct/sub ; copy data from the inner struct
change third struct orig-pointer ; restore inner pointer
result: make struct! struct-def none ; make result struct
change third result data ; change data in result struct
struct: data: orig-pointer: none
result
]
-- Gregg
[20/24] from: moliad:gma:il at: 5-Nov-2007 22:31
hi all,
I'm sorry to say I submitted something odd as the memcopy function.
its actually only 0.5% slower than ladislav's but has two differences,
*its returns a binary type
*it should be 100% GC safe
here is the proper version:
memcopy: func [
{copies a region of ram given at address with sizeof length.}
address [integer!]
length [integer!]
/local struct-struct buffer addr
] [
address: make struct! [address [integer!]] reduce [address]
struct-struct: make struct! compose/deep [ struct [struct! [ (head
insert/dup copy [] [c [char]] length)]]] none
addr: copy third struct-struct
change third struct-struct third address
buffer: copy third struct-struct/struct address
change third struct-struct third addr
buffer
]
I realized that its VERY similar to what gregg sent in when I looked at his
submission. its probably equivalent, in fact.
-MAx
On 11/1/07, Gregg Irwin <gregg-pointillistic.com> wrote:
[21/24] from: lmecir:mbox:vol:cz at: 6-Nov-2007 11:16
Hi Max,
> hi all,
>
> I'm sorry to say I submitted something odd as the memcopy function.
>
> its actually only 0.5% slower than ladislav's but has two differences,
> *its returns a binary type
> *it should be 100% GC safe
>
regarding the speed: according to my measurements:
include %peekpoke.r
include %timblk.r
a-binary: #{000102030405060708090a0b}
max-memcopy: func [
{copies a region of ram given at address with sizeof length.}
address [integer!]
length [integer!]
/local struct-struct buffer addr
] [
address: make struct! [address [integer!]] reduce [address]
struct-struct: make struct! compose/deep [ struct [struct! [ (head
insert/dup copy [] [c [char]] length)]]] none
addr: copy third struct-struct
change third struct-struct third address
buffer: copy third struct-struct/struct address
change third struct-struct third addr
buffer
]
lad-memcopy: func [
{copies a region of ram given at address with sizeof length.}
address [integer!]
length [integer!]
/local struct
] [
address: make struct! [address [integer!]] reduce [address]
struct: make struct! compose/deep [s [char-array (length)]] none
change third struct third address
struct/s
]
t-max: time-block [max-memcopy address 12] 0,05
t-lad: time-block [lad-memcopy address 12] 0,05
t-max / t-lad - 1 * 100 ; == 187
, so the speed difference is 187% not 0.5% (here).
Regarding your second premise supposing your Memcopy is GC safe.
Actually, due to the fact that it "consumes" more memory than my Memcopy
version, it is more likely to crash the interpreter when accessing a "GC
unprotected" memory. As I said, the source of the problem is the fact,
that you most probably allocate a chunk of memory using Rebol and give
it to a C routine not protecting the memory against GC. That is most
likely the true reason of the crash instead of the poor Memcopy
function, which just happens to be the victim obtaining a GC-candidate
memory.
-L
[22/24] from: moliad:gma:il at: 6-Nov-2007 10:15
Lad,
I allocate nothing, its a callback. the lib calls my func. I never build a
struct except to supply the callbacks upon loading the lib, and that is
safe.
Maybe its possible the gc could recycle in the window of time in between
change pointer address and doing the copy, but if this is the case, both
versions I guess suffer the same fatality. maybe adding a recycle/off
recycle/on pair within the func could prevent this possibility.
previously, anytime the GC would pick up the struct it would crash. Delay
was no issue, the stuct really is left corrupted, my guess is that the GC
attempts a free() on the memory pointed to by address. There are a few
reasons why structs could actually be allocated manually, outside of the
normal recycling pool, and since there is a GC managing the struct rebol
wrapper anyways, the rule that everything eventually gets freed is still
applicable.
wrt speed, my tests where within my whole environment. I'm sorry if a pure
test shows you're function almost twice as fast, and I insinuated otherwise.
just a question, can you add a to-binary to your memcopy and see the speed
impact that has? cause your function returns a string. I don't expect it
to be major, but with REBOL I don't assume anything anymore. I'm just
curious.
-MAx
On 11/6/07, Ladislav Mecir <lmecir-mbox.vol.cz> wrote:
[23/24] from: lmecir:mbox:vol:cz at: 7-Nov-2007 6:16
Hi Max,
> I allocate nothing, its a callback. the lib calls my func. I never build a
> struct except to supply the callbacks upon loading the lib, and that is
> safe.
Sorry for not believing you, it looked so improbable, but you were
right! Rebol indeed recycled the memory it did not allocate.
> Maybe its possible the gc could recycle in the window of time in between
> change pointer address and doing the copy, but if this is the case, both
> versions I guess suffer the same fatality. maybe adding a recycle/off
> recycle/on pair within the func could prevent this possibility.
>
There indeed is a problem with your memcopy version. The code below
reliably
crashes the console, when copied and pasted to it:
include %peekpoke.r
include %timblk.r
a: #{00010203040506070809}
aa: string-address? a
time-block [memcopy-max aa 10] 0,05
time-block [memcopy-max aa 10] 0,05
After inserting recycle/off; recycle/on pair as follows:
memcopy-max: func [
{copies a region of ram given at address with sizeof length.}
address [integer!]
length [integer!]
/local struct-struct buffer addr
] [
recycle/off
address: make struct! [address [integer!]] reduce [address]
struct-struct: make struct! compose/deep [
struct [struct! [(head insert/dup copy [] [c [char]] length)]]
] none
addr: copy third struct-struct
change third struct-struct third address
buffer: copy third struct-struct/struct address
change third struct-struct third addr
recycle/on
buffer
]
the crash was still there.
> previously, anytime the GC would pick up the struct it would crash. Delay
> was no issue, the stuct really is left corrupted, my guess is that the GC
<<quoted lines omitted: 5>>
> wrt speed, my tests where within my whole environment. I'm sorry if a pure
> test shows you're function almost twice as fast, and I insinuated otherwise.
+187% in speed means 287% in total ;-)
> just a question, can you add a to-binary to your memcopy and see the speed
> impact that has?
to-binary is not necessary, as-binary suffices and it is faster. The
speed difference between a version returning string and a version using
as-binary to return a binary is almost unmeasurable. Here it is:
memcopy: func [
{copy a region of memory given at address with the given size}
address [integer!]
size [integer!]
/local s c
] [
address: make struct! [address [integer!]] reduce [address]
s: make struct! compose/deep [c [char-array (size)]] none
change third s third address
c: s/c
change third s #{00000000}
as-binary c
]
-L
[24/24] from: moliad::gmail at: 7-Nov-2007 10:39
hi,
wrt the crashing, strangely, I was having problems yesterday also, I don't
know why this appeared suddenly. its possible I corrupted the version I had
been using reliably just prior to sending a strange version last week.
last week I had done tests which lasted well above 5minutes of non-stop
querying (at over 50 queries a second) and all was well, this week it seems
only your version is safe, so I'll stop trying to fix mine and use yours all
the time.
I'm just happy this whole debate ended up in exploring and fixing this
unpredictable REBOL behaviour. It seems that now, finally, we have an easy
and 100% safe way of using external data within REBOL.
I noticed in your latest version, you change the address back to 0. My
guess is this is the way rebol handles things inside, it assumes NULL * as
an unallocated struct, so the GC stops trying to free it. This was to be
my next test.
-MAx
Notes
- Quoted lines have been omitted from some messages.
View the message alone to see the lines that have been omitted