World: r3wp

the ability to remove/override/extend the set of chars considered 
"special"  in  string splitting

like " ?

some sql dialects have the handy abilty to set the current delimiter

the abilitty to "recognize all datadypes not just a frw like  integer! 
tag! in string parsing

yes Graham quote is one that can be hard to work around or show up 
unexpectedly

comma and space  seem to be others

block parsing .. you can't match specific integers

Graham - there is going to be a LIT keyword to match integers ....

this may bot be a well formed thought but ... the ability to directly 
feed a port into parse

maybe something like

parse open txcp://where.ever:555 rule

where the closing port would determine when if ever  parse was finished

prt: open ...
parse/port prt [  ...
	insert  prt something
...]

hmm, continuous parse ... there was my request for it long time ago, 
and IIRC even Carl said, that it might be usefull. Imagine for e.g. 
encoders ... You read stream from file in chunks, and you evaluate. 
The problem is, when you need to backtrack, you would need to cache 
the stream. Dunno, if something like that is possible at all ...

I don't like the result of none if parsing strings.


parse/all ",,," [(block: copy [] ) any [thru "," copy a to "," (append 
block a)] to end ]
should give  
>> block
== ["" ""]
and not
== [none none]

My suggested improvement to parse would be a trace (or debug) refinement:
    trace-output-word: copy [] 
    parse/trace string rules trace-output-word 

I'm not entirely sure how it would work. That would depend in part 
on how parse works internally, and so what trace points are possible. 
But, as a minimum, I'd expect it to show me each rule that triggers 
a match, and the current position of the string being parsed.
 parse would  append trace info to the trace-output word


Otherwise, parse is too big a black box for any one other than very 
patient experts.

Buffered parse - that should be added to the Parse Project  DocBase 
page. It's a big one, though.

Streamed parsing with backtracking : sure, it's possible, I'm doing 
that in postgresql driver since 2001 and more recently in the experimental 
async mysql driver release last year. (It's not done in a easily 
reusable way, thought).

Tracing parse: IMHO, it would more efficient to add a PARSE mezz 
for parse rules debugging purpose. (That requires to emulate PARSE 
command, which is not a difficult task.).

Tracing support would be good to add, but start it and stop it with 
the TRACE native.

Tomc, parsing an open port has been a wish of mine for years. In 
theory you could handle backtracking with buffering.

BrianH: how do you know how much to store in cache and whent o flush 
it?

Just cache the whole thing ;-)

Well, when PARSE enters a block it saves the position at the start 
of that block. If you have to backtrack that is as far back as you 
would need to cache. PARSE could optimize this by not saving backtracking 
info unless there is an alternate later on in the block. You could 
then minimize caching in some cases by rearranging your rules to 
use as little backtracking as possible, and none at the top level.

:-) Very usefull for xy MB streamed video :-)

I was thinking streamed XML, but yes :)

Interesting, Brian. I was going to suggest:

DISPENSE: Parse command to mark points in the data which don't need 
backtracking past. Parse can use

  this information to dispense with older buffer data no longer needed. 
  Otherwise it holds and accumulates the data.

  This would be used for very large or unbound length data streams. 
  eg. internet radio.

Also, I was thinking of parsing file ports opened with seek mode.

exactly ....

Yes, that's another mode, suitable for files (but not internet radio).

I was thinking about Amiga like datatypes, done in REBOL. Such decoders 
could be slow though ....

Interesting, but you wouldn't need DISPENSE if your rules don't have 
alternates to backtrack to (statically determinable).

Yeah, I hadn't thought of that.

Perhaps there are cases where alternates are not a good method of 
determining when to dispense buffer data ?

Of course "statically determinable" means that you wouldn't be able 
to modify the rule block that PARSE is currently working on (which 
would likely crash PARSE anyways).

Well, if you have no alternate, you have no backtracking, so you 
can dispose on the way.

What about REVERSE ?

Ah, that would require buffering. Darn.

And also set-words..

We're back to Robert's "cache the whole thing" :(

and Anton's DISPENSE when you know you ain't goin' backwards from 
here.

And crashing PARSE if you DISPENSE something that you need to go 
back to. That might be better done with incremental parsing.

Like this:
    parse port rule1
    ; cache gone
    parse port rule2 ; picks up where the previous left off

Yes, I was just thinking what would happen if you var:, DISPENSE, 
then :var afterwards.

Should DISPENSE update the index of var (and any other vars) when 
the internal parse index shortens ?

Does your incremental parse example assume that there is enough data 
in the cache to complete each rule ?

Maybe we should study Doc's postgresql driver :)

Oh, set-words and get-words would not work the same with R3 ports. 
You wouldn't be able to use them the same way in the code blocks 
for instance. This is because for R3 ports the position is a property 
of the port rather than the port reference, so those set-words would 
be setting the word to the numeric index of the current position 
of that port, and the get-word would correspond to seeking to that 
index. In the code blocks the words would only contain integer! values 
- you would need a separate reference to the port for them to make 
sense.

The new port model would make PARSE on ports completely different. 
You would only be able to parse seekable ports if you want to use 
set-words and get-words, and you might just be able to rely on the 
internal port caching. This might be easier than we thought.

In theory you could even do something like block parsing on event 
ports, like SAX pull. Same seekable restrictions apply - no backtracking 
or position setting or getting unless the port supports seeking.