World: r4wp

State of Red's PARSE: 


It's not on the roadmap because it's too low-level for the bird view 
(maybe I need to add it anyway?). Moreover, PARSE is not (for now) 
useful for the internal construction of Red/System and Red, so from 
that perspective, it's quite low-priority.


OTOH, it is quite simple to implement in Red/System, and users could 
see that as a sign of good progress, so I probably need to schedule 
it for a weekend to implement a R2-level PARSE (with a couple of 
features from R3's PARSE) and a few more days to test and debug it.


Also, Gabriele is interested in implementing a "compiled" PARSE version 
for Red, but unfortunately, Red has not yet all the features that 
Gab needs for it (mainly object! support). So, he's waiting on me 
to get Red ready first.


As currently, object! support is much more important to implement 
(for completing the context/binding model of Red and enabling ports) 
than PARSE, you might get Gab's version first. Also if it's fast 
enough, I wouldn't need to make a Red/System version then.


Last but not least, I don't agree with 100% of the changes/addition 
in PARSE for R3, so I would need to review R3 parse and make a "cleaned-up" 
version for Red. Also, Topaz has some interesting improvements on 
PARSE what I might want to include in Red's version too, so that 
needs a bit of preliminary review work.


So, as you can see, it's hard to give you a precise timeframe, I 
guess we would have a first version of it during Q2 2013.

If your review of R3's PARSE generates some criticisms, be sure to 
let us know. I am already looking at Topaz for ideas, your ideas 
would likely be useful too.

Some of the design choices for R3's PARSE were the result of inevitable 
side effects of its implementation model. Some of those choices would 
have been different with a different implementation model, particluarly 
with a compiled dialect. If you had a compiled dialect with rule 
functions, for instance, most of the operations would be converted 
to those functions and the dialect would be very lean indeed.

For instance, there is at least one bug report for R3 related to 
increasing the number of delimiters that should be rejected because 
implementing it would drastically reduce the readability of code 
written in any Rebol family language. If Red implemented that proposal 
it would be a bad thing for Red as well for the same reasons.

 -- my proposal is to write a special AI filter that checks for each 
 word user writes and resists to evaluate it if it reduces the readibility 
 of code. IMHO we surely can't say that ony specific delimiter reduces 
 readibilty and anything else a stupid programmer might cobble together 
 with the ones left doesn't  <-ironic .. I hoped the comma wars won't 
 have to continue in Red also :(

Some of the new operations were added to reduce bugs by having clean 
implementations of things that PARSE users frequently need to do 
but usually get wrong, such as CHANGE, INSERT, REMOVE, IF, THEN, 
AND, NOT, QUOTE and DO. The modification ops are the ones that lead 
to the most frequent R2 parse errors.

(I say this because your proposal is that red and r3 should have 
the same syntax rules for the same datatypes, like words)

Well, word syntax in R3 needs some work. There are tickets about 
some of the problems already.

I am not 100% if I understood your text correctly. Did you say that 
the bug report was about "increasing the number of delimiters which 
should be rejected" or that the bug that was "about increasing the 
number of delimiters" (well I don't know what this means, the only 
delimiter now is space and even nospace at brackets etc) and the 
bug report should be rejected?

The latter, the ticket needs to be rejected, I was wrong. There is 
another ticket about adding support for Unicode whitespace as delimiters, 
and we should probably try to figure out how to implement that one 
(though the zero-width spaces are a bit iffy).

aha, I misunderstood it then.. yes these unicode whitespaces are 
problematic.. I often have problems with them when trying processing 
data in bash from various documents

We can switch to !REBOL3 is you want to continue this discussion 
:)

I am just going to sleep, no problem

BrianH: compiled dialect is what Gabriele wants to implement. It 
should indeed be efficient in Red. About the review, I think we should 
get it organized somehow, I know that Gabriele has also some words 
to say about it too. But this can wait for a few more weeks.

It's a subject I've given a bit of thought to, especially when we 
were working on R3's PARSE dialect. The problem is that without the 
dynamism PARSE is only as powerful as a regular PEG parser, since 
regular PEG parsers are compiled. With the dynamism PARSE can be 
strictly more powerful than PEG. If your compiled dialect doesn't 
replicate the dynamic rule abilities it will be less powerful (though 
faster, so that's a tradeoff). Actually, that rule function trick 
doesn't need to be compiled, it can work with an Rebol-style interpreted 
dialect - I was thinking of doing it for R3 as an experiment.

implementations of things that PARSE users frequently need to do 
but usually get wrong, such as CHANGE, INSERT, REMOVE, IF, THEN, 
AND, NOT, QUOTE and DO

 this is where I *must* disagree. THEN seems to be infrequent, CHANGE, 
 INSERT and REMOVE are not frequent either according to my experiences 
 (nor they should be frequent), DO is a typical "one user feature".

Well, the modifiers are infrequent in your experience because (based 
on our arguments) you avoid modification of the source in parse rules 
on principle, and don't like it much outside of parse rules. So it's 
not surprising that *you* don't do it much. For that matter, *you* 
would be a poor judge of what people normally do badly in PARSE or 
DO dialect code because you just aren't that used to doing things 
wrong :)

I agree about THEN though (that was a Carl thing), and to some extent 
DO. For DO in particular I have seen people request help doing stuff 
like that frequently enough, but they either give up or are capable 
enough to not get it wrong.

DO is basically the semantic block-parse equivalent to Fork's request 
to incorporate TRANSCODE into PARSE. People would use it for mixed 
DO-other dialect code, but there is probably a better way to do it.

What can we do, Gabriele wanted it and it seemed like a good idea 
at the time. He convinced me.

What I see as a bad idea about DO is that it looks too specific to 
be of general use

It seems useful for dialects, and dialects are one of the main purposes 
of block parsing, and supposedly of Rebol. Beyond that it doesn't 
seem to be useful. It doesn't hurt to have it in there. Security 
was one of my initial concerns when Gabriele first proposed it, but 
R3's greater control of binding can be enough to deal with that issue 
so it's about as harmless as any method of implementing a dialect 
that allows code execution.

It also isn't difficult to do (Do_Next is a core function) and isn't 
likely to cause a conflict as a keyword because why uses 'do as a 
rule name or temporary variable? END is a worse keyword than DO.

why -> who

I hope you guys are considering the longer (hopefully not so long) 
term series streaming (from ports?) wrt to what you consider worthwhile 
functions in parse. I would think that some the in-place changing 
functionality is desirable for that kind of situation. I guess, copying 
might not even be an option if there is not significant buffering 
available. Am I totally off with this kind of thinking?

That is something I have given some thought to. I don't know about 
Red's port model, but for R3's port model we should be able to use 
seeking to implement backtracking and position setting. This should 
be reasonably quich in buffered port models.

Copying would be easier than modification, especially since modification 
wouldn't even apply for many port schemes, while copying just results 
in a value in memory.

You're right - I don't know what I was thinking. What I had in mind 
was one or more back-to-back parsing/transformation stages and it's, 
clear now that you pointed it out that each stage would just parse 
(if needed) data as it is copying from the port or a previous stage, 
to fill its respective buffer.

Kaj, yes, I would say regarding the SDL wav XP bug that it is appearing 
as a timing bug.  Hope to look at it more this weekend.

Great!

Doc, how about merging dyn-lib-emitter into the upcoming release? 
I think it would make use and development a lot easier

I've tested that the remaining issues we had with the R3 bridge are 
due to bugs in R3, not dyn-lib-emitter

Would it be possible to implement { } multi-line strings in LOAD 
before the release?

Another crash:

red>> function [] [()]
Segmentation fault

LOAD doesn't parse file! type:

red>> %/x

*** Script error: action 42 not defined for type: 2

If it would do that, the interpreter could execute the same scripts 
as the compiler unmodified, assuming that any #include files are 
already compiled into the interpreter

The error seems to have something to do with path interpretation 
again, as it happens when a / is present

Paul, I think I have found the sound problem on Windows

It seems I need to open a window before sound is initialised properly, 
no matter if you want a window or not. DirectSound is said to need 
to be associated with a window handle. So it wouldn't apply to other 
sound systems, which probably explains why it works on WINE

Sounds like a poor design choice on the part of the DirectSound team.

Yes, and it's not abstracted away in SDL

Ka, there's no file! type yet in Red, even if some parsing rules 
are already included in compiler's lexer. So the path error for %/x 
is currently correct.


However, adding file! is not a big deal and as you already have provided 
some file access functions, I think I'll add it for the 0.3.2 release.

Ka => Kaj.

FYI, the runtime lexer will parse #include as a word! currently, 
I could change that though, if it's a blocking issue for some scripts 
(but I don't want to spend too much energy in a tokenizer that will 
be replaced in the next weeks).

Multi-line strings: looking into it to see if it's easily doable.

BTW, the segfault above has been fixed.

Is it a known issue for console:
red>> print  4 * 4
16
red>> print 10 / 2
10
== / 2

*, +, - works, but / not.

Good catch!