World: r3wp

Most of REBOL's memory usage is that data. The mezzanine code is 
that data. For that matter, I am being asked to implement delayed 
modules so that non-native code can be shared in its source form 
for as long as possible, delaying loading it into regular REBOL data 
unless absolutely necessary.

but the loaded function is now immutable in R3, so it can be shared, 
as long as there are no "static" or global variables in the code. 
as opposed to a string being hacked away within that function.

Only if the series and other data structures in its code block are 
marked copy-on-write.

yep. that's what I meant by static variables.

Cool, this sounds doable, with some tweaks we haven't thought of, 
of course. Getting back to the new module importer now.

it would also be nice to have something like GDC in the thread API. 
 something specifically designed for short-running bursts.  something 
like....

draw this image, read this web page, etc. this way, an app could 
divide a process into parrallel tasks, linked in a port or something 
so that you just need to wait on completion of some or all tasks.

breakup rendering of an sequence into several threads and instead 
of messaging long running threads and having to build a complex application... 
you just something like

render-sequence: func [data frames] [ 
	repeat frame frames [
		dispatch 'pic-renders img: image-render data frame
	]  
	wait pic-renders 
	dispatch 'encode-movie do-encode %movie.mp4 img
]


the thing here is that the dispatch setup is external to the source 
code using bursts, and could be configured to use more or less threads.

you've got two independent bursts with a dependency

As for callbacks - I surely don't understand the issue properly, 
but in overall - we are a messaging language. We should aim for message 
passing based kernel, as Amiga was, and as QNX is. Let's use events, 
ports for IPC, when appropriate ...


As for tasks - my understanding is, that Carl still plans on threads 
internally. Didn't BrianH said yesterday, that in modern/future multicore 
architectures OS tasks are better way to go?

Carl - if you do tasks, we might get Cheyenne ported to r3 :-)

note here that we DON'T wait for the second dispatch to finish, we 
just make sure its got all the frames before  dispatching it.


then you could have a burst config which allocates different number 
of threads based on dispatch labels... maybe there's 4 rendering 
threads and only one encoding thread.

but you could dispatch that  function  too...

render-scene: func [shots][
	foreach shot shots [
		dispatch render-sequence shot/data shot/length
	]
]


so here you just send the whole scene to render, and it would only 
use the allocated number of 'pic-render threads allowed accross all 
shots.  :-)

Max - we should "copy" Amiga/QNX, and not a Python - simply put - 
REBOL uses its own OS-like advanced mechanisms - ports, devices. 
Codecs are turning back to ports too. And as for tasks and IPC, we 
should not just wrap OS, but use the REBOL way of doing things once 
again ....

with a few polling function it could be really nice:


completed-bursts 'pic-renders  ; reports progress done on overall 
pic-renders for all shots.

cancel-bursts 'pic-renders ; stops all pic-renders and bursts which 
depend on them.

interrupt-bursts 'pic-rendrs ; puts bursts (and all depencies) into 
a wait list, to be restarted later.

this is NOT os way, its an api over a thread model which doesnt require 
the programer to know about tasks and deal with messaging, IPC, bla 
bla.


in the vast majority of cases, threads don't need to be persistent, 
and is in fact a burden to have to manage.

using OS threads is the only way we can use multiple hardware threads 
in the same application.  otherwise, they will be completely different 
processes, and that makes a BIG difference in memory consumption, 
especially on GUI Oses which open up a lot of libs on startup.

Max - I was not talking about your proposal, that sounds OK. I was 
just referring to your Python example, where it seems to be just 
simple OS wrapper ....

What we probably should do is designate a PORT type for intertask 
messaging, and make it easy to use.

Yes please. :-)

Max, where is this parser for C header files that you're releasing/released 
today?

I'm hip deep in it right now...  I'm implementing the last "feature" 
which is the ability to format command arguments differently than 
the original C function parameters.


this will allow templating for extensions, just like in C++, and 
will also allow us to put litterals in the spec, so that one doesn't 
need to provide ALL parameters from the REBOL side.

hip deep = 50% done?

this C header file:

//---------------------------------
// r3t_integer_add
//
// test: print [ r3t-integer-add 1 0 " > expecting: " 1]
// test: print [ r3t-integer-add 2 2 " > expecting: " 4]
// test: print [ r3t-integer-add 2 3 " > expecting: " 5]
// test: print [ r3t-integer-add 0 0 " > expecting: " 0]
// command-format: [object!]
extern int r3t_integer_add(int a, int b)


will tell the tool, to provide an object interface to the function 
rather than to expect two integers.

or do you have reallly long legs?

the engine, without this feature was working 100% .  but its now 
undergoing open-heart surgery so its not currently working... no 
point in releasing just right now.

there is also a lot of work to do in order to provide support for 
more datatypes, but that can be done concurrently by a few of us, 
once the tools is public.

I didn't release the other version, because this formatting changes 
the generator algorythm, so that any work people would do to extend 
the tool, would have to be re-coded.

I also want to add A LOT of comments to make the source more litterate. 
 in fact I expect the source to contain more comment bytes than code.

What file did Carl say contains the new documentation?

not aware of any.

reb-ext-lib.h seems to be it

#define RXI_SET_FIELD(a,b,c,d)      RXI->set_field(a,b,c,d)

Maxim, well, not too literate... or it gets in the way and starts 
looking like XML.

G: that's it.

its just a massive amount of comments, the reason being that I want 
this tool to be easy to maintain in the long run and or it to be 
easy to modify for people who have special needs.

things are explained everywhere so that little is left "hanging in 
the air" .  I want cause and effect to be explicit everywhere in 
the source.

you can always save/load the code to strip it  ;-)

Max - please define, WHEN are you going to be ready with your "last" 
feature, because judging upon your Glass project, which should be 
delivered months ago, we can only see a constant teasing, but nothing 
conrete being delivered ...

it should be done today.

Wow, I was gone for a few days.  Really exciting developments!

the assimilator is getting quite powerfull, we can now specify litteral 
values in the alternate command formats!


this allows C functions to be called with some of the arguments filled-in 
by default and requires one-less argument from the REBOL side of 
things.

and since we can specify as many alternate command formats as we 
want per C function, we can build a library of parameter-less rebol 
commands which call the C functions with usefull values built-in 
to the extension.

Has anyone else considered what needs to be done for intertask communications 
... are we thinking maybe something like a RPC on the Rebol side?

http://www.library.ipc.ac.nz/

Nasa's deep space one IPC library http://www.cs.cmu.edu/~IPC/

Looks like this stuff was originally designed to help various legs 
communicate with each other ...  ( multi-legged robotic landers crawling 
across alien landscapes )

Sounds like a dinosaur approach ...

I've thought about it on and off. There are a million models to choose 
from. I don't think RPC is the way to go at all though. I'm for channels 
and messages. On a larger scale, I like the tuplespace model.

I'd like a tuplespace implementation too - I've had a good experience 
using GigaSpaces Jini/JavaSpaces implementation.  Very intuitive 
(and simple in terms of API) concept.

Go for it guys

Unless someone else wants to volunteer ?

as a testament to what is now possible with the new assimilation 
engine, just look at how insanely flexible it is at allowing you 
to provide just about any interface to a command...

using this information in the C header...

// command-template: [obj [object!]] prefix: obj- suffix: 3
// command-name: rxt-iadd
// command-template: [a [integer! decimal!] b [integer!]]
// command-template: [obj [object! x y]] prefix: xy-
// command-name: plus10
// command-template: [10 [litteral!] value [integer!]]

extern int r3t_integer_add(int a, int b)


;--------------------------------------

one ends up with this command interface... all calling the same C 
function.

    rxt-module: {REBOL [
        Title: {r3-test-extension extension}
        Name: r3-test-extension
        Type: extension
        Exports: []
    ]
    rxt-obj-r3t-integer-add3: command [ obj [object!]]
    rxt-rxt-iadd: command [ a [integer! decimal!] b integer!]
    rxt-xy-rxt-iadd: command [ obj [object!]]
    rxt-plus10: command [ value integer!]

    rxt-r3t-integer-mult: command [ a [integer! decimal!] b [integer! 
    decimal!]]

;----------------------------------------


note that in this setup, auto-exports is set to false and an rxt-prefix 
was added to every command name automatically.

also note that the command templates are not required, they just 
allow you to manipulate how a C function is mapped to REBOL.  there 
are still many features to come, but right now I am working on finishing 
the last phase of assimilation which is the actuall extensigon C 
source code generation.