World: r4wp

From a routine, if str is a red-string! pointer, this is the dispatch 
code you would need to use:

s: GET_BUFFER(str)
switch GET_UNIT(str) [
	Latin-1 [...conversion code...]
	UCS-2 [...conversion code...]
	UCS-4 [...conversion code...]
]

Beginning of internal string buffer is given by:

	string/rs-head str	(returns a byte-ptr!)

Should be GET_UNIT(s) above, sorry for the typo/

Another typo: should be Latin1.

Anyway, you don't need any conversion for Latin1, so you just have 
to do it for the other two formats.

Sticking to Latin1 is not much use these days. Many data such as 
web sites is in Unicode. It would be fine if it worked like R2, as 
a transparent passthrough, but Red eats your Unicode and won't give 
it back from its internal format

How does stdout support deal with that? Is there no conversion to 
the platform format there?

I'd be happy to look at a UCS-2 to UTF-8 conversion function but 
I don't have the time to do it at the moment.

I'm pretty sure that would be enough for Kaj's immediate needs.

Yes

I see there are specialised platform specific print functions only 
for printing the internal format. They look like a base for the general 
purpose conversions, though

I've written a quick function that will take a Red char (UCS4) and 
output the equivalent UTF-8 as bytes stored in a struct!.


It can be used for the base of converting a Red sting to UTF-8. What 
is needed is to extract Red Char! s from the Red String, call the 
function and then appedn the UTF-8 to a c-string!

The function only covers the BMP at the moment.

You can find it at:


https://github.com/PeterWAWood/Red-System-Libs/blob/master/UTF-8/ucs4-utf8.reds

It's so nice to see C written that way, Peter.

Yes, finally a C, that makes sense :-) Well, nothing against C, I 
am glad it is still around and going to stay ....

I've just committed a slightly improved version that retunrs a c-string! 
instead of  a structure.

For me the big issue of turning the function into the utf-8 string 
that Kaj's wants is "How to allocate a c-string! using the Red Memory 
Manager rather than malloc"

Any suggestions appreciated.

It would be best to do the conversions on the fly, that is why I 
want to wait for I/O get done to implement such conversion routines. 


Anyway, for doing it now, you need to allocate a new string, the 
best way to do it is:

    str: as red-string! stack/push*
    str/header: TYPE_STRING
    str/head: 0
    str/node:  alloc-bytes size


The new string! value will be put on stack, so any other call to 
a Red internal native or action might destroy it. Also, keep in mind 
that the GC is not there yet, so intensive I/O might quickly eat 
up all your RAM.

Oh, you meant a c-string!, not a string!, so it's even easier, just 
use: alloc-bytes size

Thanks.

Is there any easy way to free the c-string?

Currently no, the freeing function requires a memory frame pointer 
in addition to the buffer pointer. It is meant for internal use only 
for now.

Anyway, even freeing it won't help much as long as the GC doesn't 
do the cleanup.

Here's how your main loop would look like for retrieving every codepoint 
from a string! value:

	head: string/rs-head str
	tail: string/rs-tail str
		
	s: GET_BUFFER(str)
	unit: GET_UNIT(s)
		
	while [head < tail][
		cp: switch unit [
			Latin1 [as-integer p/value]
			UCS-2  [(as-integer p/2) << 8 + p/1]
			UCS-4  [p4: as int-ptr! p p4/value]
		]
		...emit UTF-8 char...
		head: head + unit
	]

Oops, you should replace 'head by 'p in the above code.

Many thanks.

cp hold your codepoint as a 32-bit integer.

I should be able to turn this into a function for Kaj to include 
in his routine! where he needs UTF-8

I guess that should be enough for his needs.

Fingers crossed :-)

why not to use native OS functions? At least on WIn there is: http://msdn.microsoft.com/en-us/library/windows/desktop/dd374085(v=vs.85).aspx

Kaj is working on Linux and Syllable only. Also that API provides 
UTF-16 to UTF-8 support, but we need also UCS-4 to UTF-8 (UCS-2 being 
a subset of UTF-16).

Some UCS related code for porting is here: http://public.googlecode.com/svn/trunk/UCSUTF.cpp

Endo: I have submitted a report for false positive to AVIRA, I hope 
Red binaries will be whitelisted soon. It seems to be the last AV 
vendor producing false alams, according to virustotal online testing 
tool.

Thank you, I'll check it later.

Kaj - You can find a rough and ready  red-string! to  c-string! function 
at:


https://github.com/PeterWAWood/Red-System-Libs/blob/master/UTF-8/string-c-string.reds


it #includes the UCS4 character to UTF8 convertor which you will 
need in the same directory as the string-c-string func.

The ucs4 -> utf8 char convertor:


https://github.com/PeterWAWood/Red-System-Libs/blob/master/UTF-8/ucs4-utf8.reds

I haven't really tested it as you can see from :


https://github.com/PeterWAWood/Red-System-Libs/blob/master/UTF-8/Tests/string-c-string-test.red

I'm not sure how it will cope with repeaated use as there is no way 
to release allocated c-strings under the Red memory manager.

Hope it helps.

who needs a GC/memory manager these days, just buy more RAM :-)

Peter, maybe you could user ALLOCATE function from Red/Sytem and 
let Kaj's code call FREE on UTF-8 buffers after usage?

I didn't think that it was possible to mix using the Red Memory Manager 
and C memory management in the same program. Is it safe to do so?

Yes, it is.

I have committed the change for the c-string to be allocated with 
Red/System ALLOCATE function.

FYI, Bruno is working on a Zlib binding for Red/System:
https://github.com/be-red/Red/commits/zlib

Much obliged, Peter. I can work that into my I/O routines

Really the thanks should go to Nenad. Without his help, I still be 
trying to work out how to do it.

I've add support for code points above the BMP.