The REBOL async:// tutorial - take 1
[1/24] from: maarten:vrijheid at: 3-Mar-2004 15:48
See below. Enjoy it. Republish it.
==================================================
The REBOL async:// tutorial - Take 1
Maarten Koopmans, code samples and root protocol by Gabriele Santilli
What is async:// anyway? It is an asynchronous TCP root protocol that can
be used to drive REBOLs hidden async TCP features. This allows network code
to be processed automagically upon certain network events, such as getting
a connection, ready to read/write, and closing a conection.
Once you have set the code to be called upon these events, things will go
by themself if you're in an event loop. This is much like GUI events are
processed automagically when you call 'view.
The first thing you need to do is get the async:// root protocol by
Gabriele. Download it at http://www.rebol.it/giesse/async-protocol.r
Note that this version supports binary transfers only, /lines support is
being worked on.
Now let's start with a simple client script, downloading images in the
background in REBOL/View. First the code:
handler: func [port [port!] state [word! error!] /local tmp cmd] [
if error? :state [print mold disarm state return true]
switch state [
connect [
; do HTTP request
insert port {GET /fg/anen.jpg HTTP/1.0^M^JHost:
www.3dwallpaper.com^M^J^M^J}
false
]
read [false]
write [false]
close [
; get data
data: copy port
close port
;print copy/part data find data "^M^J^M^J"
data: to binary! find/tail data "^M^J^M^J"
other/image: attempt [load data]
other/text: ""
show other
false
]
]
]
port: open async://www.3dwallpaper.com:80
port/awake: :handler
view layout [
across me: box 100x100 random 255.255.255 0:00:00.5 feel [
engage: func [f a e] [
if a = 'time [
me/color: random 255.255.255
show me
]
]
]
other: box 100x100 255.255.255 "Downloading image..." Return
Area 208x100 "You can type here while downloading."
]
Here is how it works: you open an async port using normal URI syntax in the
middle of the script. You can see this in the lines:
port: open async://www.3dwallpaper.com:80
port/awake: :handler
The second line sets the callback function. This function is the crucial
part for async:// based implementations, as it provides code that can be
executed for all the events. So, let's take a look a the handler function!
It starts with the following piece of code:
handler: func [port [port!] state [word! error!] /local tmp cmd] [
if error? :state [print mold disarm state return true]
As you can see, we get two parameters: The port that we can read/write to
and the state. The state can be an error! or a word!. If it is an error we
need to clean up and the above code shows briefly one way how this can
be done on
the second line. If it is a word it can have any of four values:
- 'connect, signalling that we have just acquired a connection (client-side
only)
- 'read , signalling that we can have data to read
- 'write , signalling that we can write data (again)
- and 'close , signalling that the other side has closed the connection.
So what we do next in our handler is a switch based on the state, executing
different code for each possible state.
switch state [
connect [
; do HTTP request
insert port {GET /fg/anen.jpg HTTP/1.0^M^JHost:
www.3dwallpaper.com^M^J^M^J}
false
]
read [false]
write [false]
close [
; get data
data: copy port
close port
;print copy/part data find data "^M^J^M^J"
data: to binary! find/tail data "^M^J^M^J"
other/image: attempt [load data]
other/text: ""
show other
true
]
]
As you see, we simply do a HTTP GET for an image, and write it to file and
the view it. Upon 'connect we immediately make our request. Why not in the
'write event? The 'write event is triggered everything we have something
written and we canw rite more. So the first time we always have to write
from another event!!!
Now, once the data is inserted we return false. This indicates that we
shouldn't be removed from the system/ports/wait-list list, where all ports
are stored (at least async:// and event:// ones). As you can see we did
return true on error.
The read and write states COULD be used to do some reading from data or
write more, but as this is a very simple request-response, that's not
necessary, so they just return false. So... the other side returns the data
and closes the connection. Upon close we simple read it ALL (again this
could have been done in 'read using a temporary buffer) and close the port.
We update the image and.... done!
Now for a simple server:
First we add a listening server port to the system/ports/wait-list, like:
either error? try [listen: open/no-wait tcp://:8000] [
port: open async://localhost:8000
port/awake: do handler
] [
listen/awake: func [l /local p] [
print "Got connection."
p: first listen
remove find system/ports/wait-list listen
port: make port! [scheme: 'async sub-port: p]
open port
port/awake: do handler
false
]
insert tail system/ports/wait-list listen
port: none
]
As you can see, its awake function convert the accepted port to an async
one and sets the handler. So what is the handler then?
handler: [ use [ buffer ][
buffer: copy []
func [port [port!] state [word! error!] /local tmp cmd] [
if error? :state [print mold disarm state return true]
switch state [
connect [print "Connected." false]
read [
append buffer copy port
while [tmp: find buffer newline] [
cmd: copy/part buffer tmp
remove/part buffer next tmp
do-cmd cmd
]
false
]
write [false]
close [print "Peer closed connection." close port true]
]
]
]
]
The first thing to notice is the fact that we use 'use to create a context
that returns a function! value. This function (and only this particluar
value) has access to its buffer. By doing the handler block in the server
part above, every accepted port gets a copy of theis function value with
its own "static" buffer space. A very simple but effective trick.
Now, on to the events. 'Connect does nothing, as we are already connected.
This happened when the awake function of our TCP server port was called. We
chose 'write and 'close also to do nothing. Why? We only want to receive a
command and then do it. The reading is done in 'read, we append what we
have read to the buffer. If we see a newline, we execute it and clear the
buffer until the newline. And then we return false. The fact that we return
false implies that we may read more than one connection.
Also note that 'close does nothing, but *if* a port is closed, it also
closes. Of course you can use 'close for all sorts of clean-up. Now if you
this code
do-cmd: func [cmd] [
other/color: attempt [load cmd]
show other
]
view layout [
across me: box 100x100 random 255.255.255 0:00:01 feel [
engage: func [f a e] [
if a = 'time [
me/color: random 255.255.255
if port [insert port join me/color newline]
show me
]
]
]
other: box 100x100 random 255.255.255 Return
Area 208x100
]
halt
to the above, you have a simple GUI with a command server.
You are now ready to play with async:// and enhance the code adding
substates to 'read or 'write to implement advanced protocols. Its just more
of the same. I hope you keep the use [ ] [ ] trick in mind for server
handlers, I found it a very easy way to add extra substates this way.
Now.... start coding!
[2/24] from: maarten:vrijheid at: 3-Mar-2004 16:13
Of course you have to do the async-protocol.r first before trying the
sample code.....
Maarten Koopmans wrote:
[3/24] from: petr:krenzelok:trz:cz at: 3-Mar-2004 17:58
Maarten Koopmans napsal(a):
Hi,
that stuff looks really excelent! Strange thing is, that even if the
code looks pretty straightforward, I don't understand every detail of
it, but that is what there are questions for :-)
> handler: func [port [port!] state [word! error!] /local tmp cmd] [
> if error? :state [print mold disarm state return true]
<<quoted lines omitted: 3>>
> insert port {GET /fg/anen.jpg HTTP/1.0^M^JHost:
>www.3dwallpaper.com^M^J^M^J}
That one - raw tcp stream, right? I wonder if some kind of dialect (set
of functions) could be produced to handle that ugly MJMJMJ and GET
commands etc., as e.g. read/custom allows. Just a theoretical question,
if it even would be worth it, nothing more
> false
> ]
<<quoted lines omitted: 3>>
> ; get data
> data: copy port
OK, so that one does not block right? And it is so just because we are
inside handler function, which is being called once some event happens
on port, so theoretically some data should be awaiting us. I just wonder
it comes in 'close switch part. We get here once other side closed
connection? So if I understand it correctly, we read it in parts, but
'read part does nothing, rebol is internally buffering data (how large
data is rebol able to buffer easily that way?) and once other side
closes connection, we can read it by copy (which will not block, even if
no-wait was not used), and whole data is being read out of the port
buffer at once? Well, I hope I am still on track :-)
>Now for a simple server:
>
>First we add a listening server port to the system/ports/wait-list, like:
>
Why? Is that needed? Am I right thinking it is because of View? Once
View starts, it adds event-port into wait-list and if we want to process
all various events properly, we have to go via wait-list?
> either error? try [listen: open/no-wait tcp://:8000] [
> port: open async://localhost:8000
> port/awake: do handler
>
Above code somehow escapes my understanding :-) So if we are not able to
open listen port (because e.g. we are already listening), we open
connection on localhost to that port? What is that good for?
> ] [
> listen/awake: func [l /local p] [
> print "Got connection."
> p: first listen
> remove find system/ports/wait-list listen
> port: make port! [scheme: 'async sub-port: p]
>
that is something I never understood. That is why I was not able to
further more deeply adapt Sterling's proxy.r script. It contained way
too much port subport and proxy (as a port :-) stuff for my brain to
swallow :-) 'p is assigned first connected client. It does not contain
any sub-port, yet it can communicate. IIRC someone said, that sub-port
contains real communication port. But I don't understand the difference,
even without sub-port, I am able to send data here and there and I can
see it buffered in port/state. What is sub-port then?
> open port
> port/awake: do handler
<<quoted lines omitted: 3>>
> port: none
> ]
so overall - it is clever - once first event happens on listening port,
we remove it from wait-list, reassign handler and insert it back into
wait-list. That sounds like nice constructor/init method in OOP :-)
>As you can see, its awake function convert the accepted port to an async
>one and sets the handler. So what is the handler then?
<<quoted lines omitted: 24>>
>part above, every accepted port gets a copy of theis function value with
>its own "static" buffer space. A very simple but effective trick.
cool!
Thanks for your tutorial, very educative!
-pekr-
[4/24] from: ptretter:charter at: 3-Mar-2004 12:27
Thanks Maarten and Gabriele!
Paul Tretter
[5/24] from: maarten:vrijheid at: 3-Mar-2004 19:35
First, realize that we didn't knock this off in two afternoons. So it is
OK to hit your head against the wall.
>> handler: func [port [port!] state [word! error!] /local tmp cmd] [
>> if error? :state [print mold disarm state return true]
<<quoted lines omitted: 11>>
>commands etc., as e.g. read/custom allows. Just a theoretical question,
>if it even would be worth it, nothing more
You could use the builtin crlf as a replacement.
>> false
>> ]
<<quoted lines omitted: 17>>
>no-wait was not used), and whole data is being read out of the port
>buffer at once? Well, I hope I am still on track :-)
That copy doesn't block, because it gets diverted to the copy in the
async:// handler IIRC. Also, async:// does all low-level input and
output buffering.
And async opening + async dns://
>>Now for a simple server:
>>
<<quoted lines omitted: 6>>
>View starts, it adds event-port into wait-list and if we want to process
>all various events properly, we have to go via wait-list?
It can be done many ways indeed. Pick your own example.
>> either error? try [listen: open/no-wait tcp://:8000] [
>> port: open async://localhost:8000
<<quoted lines omitted: 6>>
>open listen port (because e.g. we are already listening), we open
>connection on localhost to that port? What is that good for?
Gabriele? I missed it, it looks like setting up a tunnel to me.
>> ] [
>> listen/awake: func [l /local p] [
<<quoted lines omitted: 14>>
>even without sub-port, I am able to send data here and there and I can
>see it buffered in port/state. What is sub-port then?
The real connecttion :-) Anyway... this is the way to go. If you take a
look at the renewed xml-rpc you will find an add-server function that
abstract this away nicely.
>> open port
>> port/awake: do handler
<<quoted lines omitted: 11>>
>we remove it from wait-list, reassign handler and insert it back into
>wait-list. That sounds like nice constructor/init method in OOP :-)
Go clean your mouth!
>>As you can see, its awake function convert the accepted port to an async
>>one and sets the handler. So what is the handler then?
<<quoted lines omitted: 33>>
>>
>cool!
I know! That's why I thought I'd share it with you all.
>Thanks for your tutorial, very educative!
>
Put it to work. async:// is so clever. Carl explained it Eastern 2003
in a private chat, then I started experimenting and Gabriele turned that
into async://
It took some more experimenting, and finally we have a tutorial.
--Maarten
[6/24] from: g:santilli:tiscalinet:it at: 3-Mar-2004 19:58
Hi Maarten,
On Wednesday, March 3, 2004, 7:35:46 PM, you wrote:
>>> either error? try [listen: open/no-wait tcp://:8000] [
>>> port: open async://localhost:8000
<<quoted lines omitted: 9>>
>>
>>
MK> Gabriele? I missed it, it looks like setting up a tunnel to me.
That was a quick demo, first time you launch the script it works
as a server, second time it works as a client and connects to the
server. You just need to run the script twice...
Regards,
Gabriele.
--
Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer
Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/
[7/24] from: maximo:meteorstudios at: 3-Mar-2004 13:58
I find this/last week to be a milestone in the rebol world.
we can now start an async xml-rpc server by browsing a remote web page.
Our world is getting soooo coool.
we now even have a tutorial to understand the async protocol... I am thinking of using
it for my liquid net module.
-MAx
---
You can either be part of the problem or part of the solution, but in the end, being
part of the problem is much more fun.
[8/24] from: atruter:labyrinth:au at: 4-Mar-2004 10:41
Well done guys, and thanks for taking the time and effort to not only
release this but *explain* it! ;)
Regards,
Ashley
[9/24] from: tbrownell:veleng at: 4-Mar-2004 1:36
Excellent.. finally some light in this dark hole of tcp.
I have a couple of questions though.. in the handler we have connect,
read, write close.. how can i do this...
connect []
read [ (read the buffer, which in this case is "2 + 2", and having
processed this to a result string of "4"...)]
write [... send the result back?]
close []
And my second question.. what does it take to handle multiple clients,
and have the server continue to serve, rather than shutting down when
the client disconnects?
Thanks
Terry
[10/24] from: maarten:vrijheid at: 4-Mar-2004 11:02
Terry Brownell wrote:
>Excellent.. finally some light in this dark hole of tcp.
>I have a couple of questions though.. in the handler we have connect,
<<quoted lines omitted: 4>>
>write [... send the result back?]
>close []
terminate the string by a newline.
inbuf your input buffer from the use context:
read [
append inbuf copy port
if found? find inbuf newline [
do copy/part inbuf find inbuf newline
remove/part inbuf -1 + index? find inbuf newline
]
]
>And my second question.. what does it take to handle multiple clients,
>and have the server continue to serve, rather than shutting down when
>the client disconnects?
>
Nothing. Every connection gets a client port with this handler from the
server port. That's the beauty of it. And multiplexing is done by a
state machine based on the event per port, hence the name "async".
--Maarten
[11/24] from: maarten:vrijheid at: 4-Mar-2004 11:03
add a false after the read code :-)
Terry Brownell wrote:
[12/24] from: tbrownell:veleng at: 4-Mar-2004 2:21
I was misunderstood.. what I meant was, once I get some stuff FROM read.. and
do some magic with it (ie: get the result of 2 + 2).. how do I send the RESULT
back to the client via WRITE?
I dont have a problem with getting or processing READ.
Terry
Terry Brownell wrote:
>Excellent.. finally some light in this dark hole of tcp.
>I have a couple of questions though.. in the handler we have connect,
<<quoted lines omitted: 4>>
>write [... send the result back?]
>close []
terminate the string by a newline.
inbuf your input buffer from the use context:
read [
append inbuf copy port
if found? find inbuf newline [
do copy/part inbuf find inbuf newline
remove/part inbuf -1 + index? find inbuf newline
]
]
>And my second question.. what does it take to handle multiple clients,
>and have the server continue to serve, rather than shutting down when
>the client disconnects?
>
Nothing. Every connection gets a client port with this handler from the
server port. That's the beauty of it. And multiplexing is done by a
state machine based on the event per port, hence the name "async".
--Maarten
[13/24] from: tbrownell:veleng at: 4-Mar-2004 2:24
Using the server example in the doc, the server stops once the client
disconnects?
Terry
[14/24] from: tbrownell:veleng at: 4-Mar-2004 2:51
ok, in answer to my own question, you can use "insert port" at anytime
while processing script.
TB
Terry Brownell wrote:
[15/24] from: maarten:vrijheid at: 4-Mar-2004 12:44
Terry Brownell wrote:
>ok, in answer to my own question, you can use "insert port" at anytime
>while processing script.
>
>TB
>
>Terry Brownell wrote:
>
Yes, but you need to do that once, that will trigger the 'write vent
that can take over. Typically I insert the first few chars of an out buffer.
--Maarten
[16/24] from: maarten:vrijheid at: 4-Mar-2004 12:45
Terry Brownell wrote:
>Using the server example in the doc, the server stops once the client
>disconnects?
>
>Terry
>
No, it closes the connection with this particular client only.
--Maarten
[17/24] from: Christophe:Coussement:mil:be at: 4-Mar-2004 15:11
Hi Maarten,
I do not have very much experience when coming to network protocol, so please excuse
the triviality of my question :).
The fact is I could for sure use this 'async protocol for a project we are now developing.
I tried the following, but without any result. I try to load a %dghr.jpg file situated
in the same dir as %async-protocol, and try to connect to localhost. What did I wrong?
;>>>>>>>>>>>>>>>>>>> code ---
do %async-protocol.r
handler: func [
{ your comment here
RETURN: }
port [port!] "comment"
state [word! error!]
/local tmp cmd
][
;$debug/var/stop 'state
if error? :state [print mold disarm state return true]
switch state [
connect [
insert port {GET logo.jpg HTTP/1.0^M^JHost:127.0.0.1^M^J^M^J}
false
]
read [false]
write [false]
close [
data: copy port
close port
;$debug/var/stop 'data
data: to binary! find/tail data "^M^J^M^J"
other/image: attempt [load data]
other/text: ""
show other
false
]
]
]
port: open async://127.0.0.1:80
port/awake: :handler
view layout [
across me: box 100x100 random 255.255.255 0:00:0.5 feel [
engage: func [f a e][
if a = 'time [
me/color: random 255.255.255
show me
]
]
]
other: box 100x100 255.255.255 "Downloading image ..." return
area 208x100 "you can type here while downloading"
]
;--- code <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
TIA!
==christophe
[18/24] from: warp:reboot:ch at: 4-Mar-2004 15:56
I think you need a slash before "logo.jpg" and a space after "Host:"
-> insert port {GET /logo.jpg HTTP/1.0^M^JHost: 127.0.0.1^M^J^M^J}
On 4 mar 2004, at 15:11, Coussement Christophe wrote:
> insert port {GET logo.jpg HTTP/1.0^M^JHost:127.0.0.1^M^J^M^J}
Will
[19/24] from: maarten:vrijheid at: 4-Mar-2004 16:09
Hi Christophe,
>I do not have very much experience when coming to network protocol, so please excuse
the triviality of my question :).
>The fact is I could for sure use this 'async protocol for a project we are now developing.
>
Good.
>I tried the following, but without any result. I try to load a %dghr.jpg file situated
in the same dir as %async-protocol, and try to connect to localhost. What did I wrong?
>
What are you trying to do? Fetch the file from a file or a webserver?
>;>>>>>>>>>>>>>>>>>>> code ---
>do %async-protocol.r
<<quoted lines omitted: 19>>
> ;$debug/var/stop 'data
> data: to binary! find/tail data "^M^J^M^J"
Do you receive data? What does your webserver log say?
> other/image: attempt [load data]
> other/text: ""
<<quoted lines omitted: 18>>
>]
>;--- code <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
It looks like you are re-using the exact code I mailed, which (I just
checked) works. So it looks like you'll have to debug using your
webserver and debugger in the script to see what gets done and what
doesn't. Sorry I can't help any more, but the code looks fine (an exact
copy of the sample code except for the URI).
--Maarten
[20/24] from: g:santilli:tiscalinet:it at: 4-Mar-2004 16:51
Hi Maarten,
On Thursday, March 4, 2004, 12:44:17 PM, you wrote:
MK> Yes, but you need to do that once, that will trigger the 'write vent
MK> that can take over. Typically I insert the first few chars of an out buffer.
It is also possible to insert a big string all at a time (after
read or in any other occasion); async:// will care about sending
it out a little chunk at a time.
Regards,
Gabriele.
--
Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer
Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/
[21/24] from: tbrownell::veleng::com at: 4-Mar-2004 9:41
async:// Close connection
When I use the following example from the tutorial, my server stop whenever a client
disconnects with this message...
async protocol loaded
Got connection.
Peer closed connection.
>>
rebol []
do %async-protocol.r
either error? try [listen: open/no-wait tcp://:3000] [
port: open async://localhost:3000
port/awake: do handler
] [
listen/awake: func [l /local p] [
print "Got connection."
p: first listen
remove find system/ports/wait-list listen
port: make port! [scheme: 'async sub-port: p]
open port
port/awake: do handler
false
]
insert tail system/ports/wait-list listen
port: none
]
handler: [ use [ buffer ][
buffer: copy []
func [port [port!] state [word! error!] /local tmp cmd] [
if error? :state [print mold disarm state return true]
switch state [
connect [print "Connected." false]
read [
append buffer copy port
while [tmp: find buffer newline] [
cmd: copy/part buffer tmp
remove/part buffer next tmp
do-cmd cmd
]
false
]
write [false]
close [print "Peer closed connection." close port true]
]
]
]
]
do-cmd: func [cmd] [
other/color: attempt [load cmd]
show other
]
view layout [
across me: box 100x100 random 255.255.255 0:00:01 feel [
engage: func [f a e] [
if a = 'time [
me/color: random 255.255.255
if port [insert port join me/color newline]
show me
]
]
]
other: box 100x100 random 255.255.255 Return
Area 208x100
]
halt
Terry Brownell wrote:
>>Using the server example in the doc, the server stops once the client
>>disconnects?
>>
>>Terry
>>
>>
>
No, it closes the connection with this particular client only.
--Maarten
[22/24] from: Christophe:Coussement:mil:be at: 11-Mar-2004 13:33
Re: The REBOL async:// tutorial - take 1
Hi Maarten and Will
Sorry for this late answer: I was out of office for a while...
Thanks for helping me !
> >I tried the following, but without any result. I try to load
> a %logo.jpg file situated in the same dir as %async-protocol,
> and try to connect to localhost. What did I wrong?
> >
> What are you trying to do? Fetch the file from a file or a webserver?
Fetch from the file.
I just wanted to try the code running on localhost.
> >;>>>>>>>>>>>>>>>>>>> code ---
> >do %async-protocol.r
<<quoted lines omitted: 24>>
> >
> Do you receive data? What does your webserver log say?
Well, do I have to work on a webserver?
I thought %async-protocol.r serviced a sort of server, listening on specified port ?
> > other/image: attempt [load data]
> > other/text: ""
<<quoted lines omitted: 28>>
> (an exact
> copy of the sample code except for the URI).
anyway, wet i get is the generated window, but with no image loading.
After placing a debug mark at the beginning of 'handler, I can see the function isn't
evaluated once.
I think I'm confused about the context of the use of this protocol :(
Could you provide me with a concrete use case ?
TIA
==christophe
[23/24] from: maarten:vrijheid at: 11-Mar-2004 14:34
The specific example you try is a client that connects to a webserver to
fetch the image. If you have no webserver running (no valid url...): no
image!
async:// is just like tcp:// , but now with calllbacks instead of
waiting yourself. Does this help you?
--Maarten
Coussement Christophe wrote:
[24/24] from: Christophe:Coussement:mil:be at: 11-Mar-2004 15:12
Hi Maarten,
It's all clear now ;)
Thanks !
==christophe
Notes
- Quoted lines have been omitted from some messages.
View the message alone to see the lines that have been omitted