World: r4wp

ok with me. I had some trouble with the compiler script last time 
I used it. Location of red or rsc compiler is changed by compiler, 
the active directory changes.

I have spend many hours today reviewing again Kaj's claims and looking 
into the code to see what is implemented. There are many public claims 
made by Kaj about Red that I need to correct, so these are my answers:

1) "I found out that not only does Red not support Unicode, it doesn't 
support Latin-1, not even on Windows" Red *does* "support" Unicode, 
Latin-1 "support" was not claimed in Red, except for the console 
script. I've put quotes around support word, because you're mixing 
up internal representation and I/O encoding formats.

2) "The printing backend doesn't fully support Unicode, either." 
It does, as proven by the test suite.

3) "So for most purposes, Red is currently ASCII only" Obviously 
not, it is for your specific purposes. Again, look at the Unicode 
tests.

4) "It was announced last year that Unicode support was implemented 
in a week. What I found first is that Unicode support is useless, 
and now I've found that only ASCII is really supported." Unicode 
support has been implemented exactly as stated in (will get back 
to that point later):
http://www.red-lang.org/2012/09/plan-for-unicode-support.html

5) "Yes, I think it's very dangerous to claim that Red has Unicode 
and Latin-1 support". Red *has* Unicode support, string! and word! 
value support Unicode, input Red scripts are Unicode, PRINT outputs 
Unicode characters. Latin-1 is used as an *internal* encoding format, 
I don't remember ever claiming that "Red supports Latin-1 for I/O" 
except for the console script (which is wrong, I agree). OTOH, I 
do remember thinking about supporting it at the beginning for printing, 
then I found it cumbersome to support in addition to Unicode mode 
and dropped it during the implementation.

So, about the console issue, the runtime lexer is able to parse Latin-1 
input but the input string gets internalized before being passed 
to the lexer using the UTF-8 loader, which chokes on MSDOS console 
incompatible codepages. For the Unix version, the console input being 
in UTF-8 by default, it passes the internalization, but crashes the 
runtime lexer.

So, currently, only 7-bit ASCII is safe to input in the console. 
This limitation has nothing to do with Red implementation or the 
interpreter, it's a console input issue, so generalizing it to whole 
Red is inaccurate and unfair.

Anyway, as I stated many times, the current runtime lexer is a temporary 
implementation, waiting to be replaced by a proper Unicode lexer. 
It shouldn't have survived more than a few weeks, hopefully, it should 
be soon gone. We'll switch then the console to the Windows Unicode 
API to get rid of the codepages hell. In the meantime, I will correct 
the console banner.

Wrt the Unicode plan (my blog entry link above), I would like to 
highlight only one sentence: "Conversion for input and output strings 
will be done on-the-fly between one of the internal representation 
and UTF-8/UTF-16." This is what have been implemented for Red input 
scripts (except from the console), and for outputting text on screen 
with the currently hardwired PRINT output support because the I/O 
sub-system has not been yet implemented in Red. The PRINT backend 
will be rewritten once we get ports/devices support. Also, the "on-the-fly" 
part (no intermediary buffer) should have hinted you that I could 
not implement encoders/decoders before I/O sub-system is done. This 
also means that the current encoding/decoding logic you've implemented 
these last days probably won't be useful for Red's I/O.

Kaj, it seems to me that you were confused by a few things:
- console script banner wrong statement (my fault)

- internal "Latin-1" naming (like in Python's internals) which might 
be misleading (there's no other closer naming in Unicode for one 
byte representation AFAIK, though some people call it "UCS-1", maybe 
we should adopt that too).

- "Unicode support" seems to imply to you that *all* possible Unicode 
encodings have to be supported (with encoders/decoders). It doesn't, 
having just one encoding supporting the full Unicode range (like 
UCS-4) is enough for claiming "Unicode support".

Furthermore:


Red can input some Unicode and print some Unicode. That's enough 
to support the test suite, but mostly useless in real life programs

 I must have missed the point in time when Red was declared beta. 
 :-) AFAICT, Red is not yet ready for real-life programs (Red/System 
 is though).

It's not even true that I/O is to be done, because I support it

 Reading how you put it, I just hope you still remember which version 
 is the official one? ;-)

Really, I don't feel like spending more time on this. Let's agree 
to disagree

I feel the joy in my Red work slipping away from me

I have showered Red with praise on many occasions. Why is it such 
a problem to have some criticism, too?

Well, I don't understand all the fuss. Of course, eventual false 
claims should be corrected, but it feels like selecting one feature, 
knowing we are not there yet, and trying it to push it forward, reshifting 
other priorities? If someone else would wait for dynamic libraries, 
and Doc now switched the focus, would we see another complaints from 
 someone else, that dynamic libraries are not yet fully supported?

If dyn-lib is around the corner, let's better ask Doc, what is imediatelly 
next focus of implementation. IIRC, it should be objects, needed-by 
and followed by I/O. And if Doc says, that more Unicode stuff comes 
with I/O implementation, what timeframe are we talking about? 2-3 
months? Well, not sure how much you are kept stuck with such a timeframe, 
but maybe even you can wait?

OTOH - it seems, that for now, you found/implemented interim solution? 
I believe that if Doc could add full support in 2-3 days, he could 
reshift the priorities, but if it really depends on I/O, then let's 
say - first things first ....

And as for the criticism - it is OK. It is just that we can feel 
you are not happy about the current state of Unicode support, and 
as I know Doc, he would like to please us all, so I can imagine it 
kind of creates some push on him :-)

There is as I read this a different issue. Dock want Red to be as 
complete as posible, Kaj wants it to officially useable. Kaj really 
needs UTF-8 (and or Latin-1) character support, for getting this, 
I guess this has to do with the Syllable operating system amongst 
others.

I would like Red to support time and random functions as natives 
and (Gregg is one of your mezz funcs REJOIN ? I want that too) be 
able to connect to a MySQL database so I can dump PHP for some webdevelopment.

Besdies that we all love to see a VID (like) solution for display 
and creating apps. 

We have to be patient agreed 100% amongst everybody? Where the roadmap 
mentions all things to progress Red, above things are not on that 
list. I want Red to have enough to make it useable in production 
and after that expand, imho that is the way to really attrackt more 
funding/enthousiast programmers and make sure current support does 
not fade/ loose interest.

I think, and hope, the conflict over this will fade soon, and everyone 
will be happy again. We're just all so anxious for the things we 
each need, and Doc has made such great progress that it seems like 
Red should be almost ready to use. I was going to post that REJOIN 
wasn't stable yet, but decided to test it on my latest console build 
and it seems to work great.

rejoin: func [

 "Returns a new string (or same series type as block/1) made from 
 reduced values."
	block [block!]
	/local op
][
	either empty? block: reduce block [block] [
		op: either series? first block [:copy] [:form]
		append op first block next block
	]
]

Why is it such a problem to have some criticism, too?

 There is no problem with that, and believe me, you can't be more 
 critical on Red that I am myself. But I find it really unfair to 
 paint a bad picture of whole Red because some features that are planned 
 are not yet implemented. 


I can't go faster than the music, to get I/O done with required encoders/decoders, 
I need to setup the ports/devices infrastructure. To do that, I need 
objects support done. Also, as shown by my entries on Red Trello 
page, error! (and typeset!) support and getting a Unicode runtime 
lexer are even more prioritary to make Red "more usable for real-world 
apps". Moreover, when you manage a lot of tasks, some of them marked 
as "important" that keep been postponed because of other more urgent 
ones, will at some point become "urgent" themselves. That is what 
is happening with shared libs support, which is a blocker for getting 
Red on Android and iOS. I'll also probably make a Java bridge prototype 
this week before getting back on other Red features.

Almost forgot about the Trello page. Do you use that page as an interim 
planning for what is actually being worked on? If so, and once you 
have few minutes of spare time (I can imagine you don't :-), maybe 
one sentence reference on the Roadmap page would make it a nice addition:

You can find actual tasks we work on on the following Trello page

 .... it would make feel ppl more dynamic about the project, as Roadmap 
 page is rather static. OTOH we can watch Git commits, twitter, but 
 still if one needs to know, what is an actual priority, and as far 
 as Trello page is being updated, it would make some sense. E.g. "Download" 
 addtion was imo very nic simplification for those starting with Red 
 ...

Nice to read about the JAVA bridge ....

btw - probably stupid question by me uneducated :-) Reading a Trello 
card, I can see task "Extend ELF emitter to support shared library 
generation" - does it mean linux is not supported? Or is ELF anything 
else, than Linux executables?

Andreas is working on Linux shared library support. ELF is not limited 
to executables.

Don't want to bother you with primitive questions, but - why in general, 
there is a need for shared library support, on mobile platforms? 
Can we just have - executables? IIRC, R/S code already worked under 
Droid? Well, I know that .apk is some other format (maybe a zip file 
with some files inside?). Is it that in order to have PROPER app 
support (e.g. via GooglePlay), we need to adhere to some rules, and 
one of those is, we need dynamic libraries?

The need is for bridging Red with the JVM and the objective-c runtime.

Red executables (like R3 ones or any other language compiled to native 
code) are not supported directly by mobile OSes. So, it needs to 
be loaded and called from Java or obj-c, hence the need for shared 
library support.


The experiments with Red/System on Android was limited because using 
such approach, you can't have access to the higher level API, can't 
access GUI and need an hack to get it loaded. My goal was just to 
check that Red/System toolchain was able to produce code that runs 
on Android.

Petr, I didn't ask anyone to switch focus, I just asked for information 
repeatedly. I eventually analysed and fixed the problem myself, with 
help from Peter. If that's not enough, I don't know what is

Doc, I keep repeating, I am not "painting a bad picture of whole 
Red", you make that up, blowing the issue out of proportion

I was genuinely disappointed by the lack of Unicode asymetry, and 
had good cause to expect it based on your publications. I made the 
best effort to gain insight into the situation and contribute a fix

I have been digging out this issue with Kaj privately, and it appears 
that several misunderstandings happened a few weeks ago setting up 
a chain of events that resulted in having an unpleasant (for us and 
others) exchange between me and Kaj in this channel.

Kaj raised 
an issue a few weeks ago and asked for information that I failed 
to give. This was my mistake and I'm sorry for that. We've dug out 
the root reasons for that and have set up better rules to avoid, 
as much as possible, such failure in the future. Kaj's dedication 
for Red is obvious to all, from day one, and I'm grateful to him 
for that.

Actually, after setting up this issue, we've come up with 
some interesting new plans and strategies for Red that we will unfold 
in the next weeks...so stay tuned. ;-)

Yay! Thanks for posting that Doc. I was having a hard time choosing 
which of you to keep as a friend.

And it's easier if you're both there, so I can stand on your shoulders.

Hehe :-) You might want then to take an oxygen mask with you, because 
we're going to jump very high. ;-)

Well, maybe I'll ride your coattails then.

As an aside, I needed a break last night, and looked at some Red 
mezz bits again. I realized that REMOVE-EACH was one of the missing 
natives I had written off for a while, assuming you would get to 
it. But I got impatient and wrote a quick mezz version. Fun stuff.

One of my goals is to have as much working at the mezz level as possible, 
so you don't have to write more natives in these early stages. Once 
we find actual cases that aren't efficient enough, then we can optimize 
them.

e.g., I did all the set functions (union, intersect, etc.), even 
though they aren't optimal. They work, and that will let us port 
code.

Sounds fair enough.

Great ... I can take those PERL books back.

I'm very happy with that, Gregg, I follow the same strategy

It can be compiled, anyway, so it's not that slow

I was looking to take cover just in case a battle of the wizards 
was going to happen!  Glad to hear it has been avoided.

I used to be a Dragon-Wizard level 37 in CircleMUD back when Internet 
was mainly text-only, so this could have become nasty indeed. ;-)

Is there comprehensive updated documentation for RED?

Not yet Paul. Though there is for  Red/System.