World: r3wp

Actually, that is exactly what we need. One problem though:

>> transcode "a,"
== [make error! [
        code: 200
        type: 'Syntax
        id: 'invalid
        arg1: "word"
        arg2: "a,"
        arg3: none
        near: "(line 1) a,"
        where: [transcode]
    ] ""]

I've requested that this be changed to:
>> transcode "a ,"
== [a ","]

putting off the error until the next call. That change would be enough 
to do LOAD/else.

All it would require is to make "," a delimiter in the scanner - 
a trivial change for Carl.

Sorry, two messages back: "a ," -> "a,"

in the second example.

Pekr;  I think a convo we had on the blog wayy back was partly why 
Carl relaxed TRANSCODE  ;)

We went through a phase of trying to name the junk! datatype, and 
very real issues were raised about where the lexical scan should 
restart after.  Then someone discussed the linguistic analytic potential 
and another very real concern about polluting the semantics of REBOL 
datatypes was raised.  Carl may have been afraid we'd all run off 
and produce "any old crap that works", where I aim to provide a console 
that takes "any old crap" and works.  ;)  Leaving Gabriele and others 
free to use the ever cool true semantic meaning of data and code 
and leave junk! that floats for analysis of real world information 
sans REBOLification.


Part of that conversation lead to and is included in http://www.rebol.net/cgi-bin/r3blog.r?view=0078#comments

BrianH: ... then I expect you will propose the change to Carl, to 
enable load/else? :-)

any known rebol datatype should be recognised, and the rest should 
be assigned under junk! or similiarly named datatype. What is AI 
about that?

 Telling the different between a REBOL value and "the rest" is AI. 
 If you can without AI, then "the rest" is defined with a specific 
 parse rule... then you can just use string parsing, and it's not 
 a human language anymore but a specific language you just designed.

Right, I misunderstood LOAD/else proposal, I thought you could supply 
your own parsing rules to be used from the point where LOAD fails. 
But I still can't see the point of using LOAD on non-REBOL data, 
instead of string parsing or LOAD/next. LOADing a whole book would 
most probably blow out REBOL words limit. If it all boils down to 
splitting untrusted input into whitespace separated tokens, why no 
just use PARSE data "" ? Is there a real benefit for the programmer 
of having LOAD not stop on syntax errors?

Doc; no not really a benefit to programmers and rebol-by-nature types; 
 The goal is non-rebol types and creating ease of use that attracts 
those types to become rebol-by-nature.  But I've sure wasted a lot 
of good rebol time already, so ... 

Go Doc Go and Do Gabriele Do    Ok ... and Yay Brian!    ;)

Pekr, I have already proposed the TRANSCODE change, and I can start 
the implementation of LOAD/else asap (which will likely be after 
the release). LOAD/else will work without the TRANSCODE fix, but 
will work better with the fix.

About VPS: Looks like it could be Linode for me. Slicehost is pretty 
on the surface, but it appears that customers are a bit happier with 
Linode. Also Slicehost was recently purchased by Rackspace, and some 
customers reported a lower service quality after that. That might 
be a bad sign.

doc, its faster to let rebol do what is does best in binary and let 
us take over on those tokens it can't recognise.  just ignoring comma 
would have allowed me use scientific data more than once directly 
 without having to dare understand parse intricacies.


remember that parse IS NOT EASY.  I didn't use it for over 6 years, 
cause everytime I dared, it just blew up in my face.  this would 
provide a simple entry point for more people to support DSL and leverage 
the rich datatype system in rebol, without the need to be a guru 
level reboler.

the point is not to replace REBOL's syntax, but allow REBOL to cope 
better with the rest of the world's data, IMHO.

IMO, designing a good DSL is far more difficult than writing parse 
rules.


I think that ppl here are underestimating the complexity of implementing 
a jump-over-foreign-data feature that would work in the general case. 
REBOL syntax is not based only on whitespaces, but on delimiters 
too : double-quotes, curly and square brackets, parenthesis,.... 


So it's perfectly valid to not use whitespaces at some places, like 
in : "either conditon [true][false]". So what should LOAD do in case 
of, e.g. : "either condition [f,o,r,e,i,g,n,0,1][false]" ? IMHO, 
besides reporting a syntax error at "f," , there's no much point 
returning [either condition "[f,o,r,e,i,g,n,0,1][false]"]. And if 
you think that LOAD could just return [either condition ["f,o,r,e,i,g,n,0,1"][false]], 
then you just jumped over the complexity by creating some syntax 
rules in your mind, but LOAD can't do that. Once LOAD has passed 
the syntax error point, it has no sure way to determine where the 
foreign data ends and where REBOL correct syntax starts again. That 
would require at least, an AI engine (like Gabriele stated above). 
I'm not even sure that all possible cases could be covered that way.


Btw, does such feature exists in any other programming language? 
I've personally never seen or read about such feature elsewhere, 
maybe for a good reason.

load-foreign works well to me in R3.

alpha: charset [#"a" - #"z" #"A" - #"Z"]
chars: union alpha charset [#"0" - #"9" "-_"]
word: [alpha any chars]


probe load-foreign "either condition [f,o,r,e,i,g,n,0,1][false;]" 
[
		;**keep words with comma as trings
	[word some [#"," any chars]]		[keep &]	
		;**remove ";" at the end of words
	[word #";"]				[keep to-word head remove back tail &]	
]

>> [either condition ["f,o,r,e,i,g,n,0,1"] [false]]

I still vote for foreign!  as a datatype.  Sure there will still 
be reasons for throwing errors on evaluation on foreign! but I don't 
like the fact that the errors occur on load.

Give linguists a tool that out-of-the-box can load a poem and let 
them count characters, words, and transitions.


Let REBOL pros worry about the hassles of evaluation.   Re: where 
the foreign data ends; Skip AI, just go with delimiters.  Whitespace 
or quotes or any of the braces etc.  And if a missing quote causes 
a script to fail ... well bad on the coder for not testing before 
release.  If the English professor gets wonky counts, well let them 
figure it out, but give them the wonky counts not a LOAD failure. 
 IMHO it really would open the door to so many more uses and users. 
 


And with Steeve's example ... build something like that into the 
product.  Don't make poor Professor Keating try and figure out why 
most of the Dead Poet's prose can't be analysed with REBOL's nifty 
cool block data functions.


And as long as REBOL had foreign! as a native datatype, writing a 
LINT would be dirt easy.

And I could LOAD %cobol.cob  and do all kinds of super cool preprocessing. 
 Or at least wait for Sunanda to write a super cool preprocessor 
and live vicariously through rebol of the year candidates.  ;)


REBOL could be the JavaDoc, EpyDoc, Doxygen, ..., ReST besting documentation 
super tool of tomorrow.

doc, the point is that I and others HAVE had ,many situations where 
this would have saved HOURS even given new reasons to use rebol at 
some jobs.  the point is not to allow new rebol dialects, its just 
that in MANY cases it would work..  we all know that this isn't a 
replacement for parse.  but allowing me to import external data through 
load is just faster than parsing.  You still have to tailor the extension 
to the dataset anyways.

We could load things like C code directly, and then parse it using 
block dialecting instead. way faster.

Ditto Maxim.   Adding ...  I don't look at this as being a overly 
good thing for rebols (although I do think we'd all benefit, save 
time and write a whole new breed of utilities), I think of it as 
a good thing for non-rebols (and us rebols with smaller brains and 
bigger laze).

ah, so we are at it once again ....

You guys write it, then we can talk about it.

it's incredible you think that "parse is not easy" and then you don't 
trust people telling you that what you ask for is silly at best.

parse *IS* easy. try to use regexps instead!

ok i give you load-foreign, don't kick me Gabriele :-)

context [
	stack: make block! 10
	foreign-rules: make block! 10
	out: end: value: pos: none
	push: func [type][
		stack: change/only stack out 
		out: make type 1 
	]
	pop:  to-paren [
		stack: back stack 
		out: append/only stack/1 out
	]
	set-&: [end: (misc/&: to string! copy/part pos end)]
	misc: context [
		&: none
		keep: func [&][append out &]
	]
	blanks: charset " ^-^M"
	rules: [
		  some [blanks | #"^/" (new-line out true)]	  
		| pos: foreign-rules
		| [#"]" | #")"] (print "missing [ or (" return stack/1)
		| #"[" (push block!) any [#"]" pop break | rules]
		| #"(" (push paren!) any [#")" pop break | rules]

  | (set [value pos] transcode/next/error pos append out value) :pos
	]
	
	set 'load-foreign func [
		text [string!] foreign [block!]
	][
		out: make block! 10
		stack: change/only clear head stack out
		clear foreign-rules
		foreach [rule code] foreign [

   append foreign-rules compose [(rule) set-& (to-paren bind code misc) 
   |]
		]
		append foreign-rules  [end skip]
		parse/all to-binary text [any rules]
		bind out 'system
		either 1 = length? out [first out][out]
	]
]

]

Steeve, you're just proving my and Gabriele's point. There's no way 
to implement a LOAD-FOREIGN that would work for any data. You have 
to provide parsing rules that are specific for the input data. Max 
and others are asking for a LOAD-FOREIGN that just works on any input 
data without providing any additionnal rules.

and if you have to provide the additional rules you might as well 
parse it from scratch.

i can generate strings by default each time there are errors by default, 
so that there is no need to provide rules.
It's a tiny modification

But strings are not good types for those  it would be better to simply 
add errors as objects in the loaded stream.

But after that you steel need to reparse the block to convert the 
errors in proper values.
SO, in the end, y

*steal

Ok, in the following version (much simpler) i just add foreign data 
as error objects in the stream.
Then you can reparse the result to manage errors as you want.

context [
	stack: make block! 10
	out: value: pos: none
	push: func [type][
		stack: change/only stack out 
		out: make type 1 
	]
	pop:  to-paren [
		stack: back stack 
		out: append/only stack/1 out
	]
	blanks: charset " ^-^M"
	rules: [
		  some [blanks | #"^/" (new-line out true)]	  
		| [#"]" | #")"] (print "missing [ or (" return stack/1)
		| #"[" (push block!) any [#"]" pop break | rules]
		| #"(" (push paren!) any [#")" pop break | rules]

  | pos: (set [value pos] transcode/next/error pos append out value) 
  :pos
	]
	set 'load-foreign func [
		text [string!] 
	][
		out: make block! 10
		stack: change/only clear head stack out
		parse/all to-binary text [any rules]
		bind out 'system
		either 1 = length? out [first out][out]
	]
]

probe load-foreign {
	either condition [
		f,o,r,e,i,g,n,0,1
	][
		false
	]
} 

>>[
    either condition [
        make error! [
            code: 200
            type: 'Syntax
            id: 'invalid
            arg1: "word"
            arg2: "f,o,r,e,i,g,n,0,1"
            arg3: none
            near: "(line 1) f,o,r,e,i,g,n,0,1"
            where: [transcode parse load-foreign ...]
        ]
    ] [
        false
    ]
]

>> load-foreign "either condition ]f,o,r,e,i,g,n,0,1[]false["
missing [ or (
== none


Not very usefull output. Remember that we are talking here of the 
general case where input can be *any* data.

i gave the design, it's simple to ehance

what is good to return in that case ?

It depends on the rules chosen to interpret the string content. That's 
the point, there's no simple way to implement a general case load-foreign 
that would return a block! of mixed REBOL values and "foreign" values 
as string. It's not possible to define a general set of rules because 
"foreign" can be just anything. LOAD just returns an error! when 
the syntax is incorrect and that's the best thing it can do.

doc, never said it should accept anything...

just that I have seen it would work so many times as to not make 
it a useless feature.

its also a question of speed.  load is blighthingly fast.

If you can define a set of rules for your specific needs, then you 
can just use parse or Steeve's load-foreign for that (or ask someone 
to write it for you if you don't want to implement it). Everyone 
can have its own specific needs, I can't see how to define a set 
of rules for "foreign" data that would be or could be useful to most 
users. That would probably end up defining a few additionnal datatypes 
(or alternative syntax for existing datatypes).

Max, did you ever consider WHY load is so fast?

What makes you think that it would be so fast if it was able to do 
what you ask?

Because.  ;)   If REBOL had a foreign! datatype! could the lexer 
not just ... about to throw syntax error, make a foreign! and continue?


I still don't see why this is deemed a bad extension to REBOL.   
Sure it "stinks" to guru's but it opens doors to fishermen, contractors 
and linguists, imho.  And yes it could look like a load of gunk if 
a quote, paren, brace or bracket was misaligned.  So?


For code purity, I'm sure Steeve could spend a few minutes to write 
4 lines to scan a loaded series! and recursively highlight the foreign! 
data, providing a semblance of a lint procedure for those writing 
apps.  My push for foreign! has little to do with apps, and just 
about all to do with data, human real-world data.   For people using 
data for sums, control, etc... just use the lint scan to reject data 
blocks that have ANY foreign! substances??

Why not just load the bad data as a string! ?

should be simple to parse out string!s when a numeric datatype is 
expected...

Are you guys listening?

yep

I'm just a bit amazed at the lack of comprehension.  I agree with 
DocKimbel and I advise to think on his words longer.