World: r3wp

Ask a question about portability.

What platforms are supported by initial release, and what platforms 
do you plan to support in future?

Answer to above question might be related to the World language being 
open-source or not. Dunno if that one was answered already, or not 
...

Yes, I had some questions about this. I'll add your new question.

Will be there countdown for the QA as well? :)

QA in 10 ...

** Cortex meltdown: unknown OP-code
[KWATZ!]

For everybody who want to see the complete World (like me) before 
it meltdowns = http://themes.googleusercontent.com/image?id=1f-p4d2MS_T8qP_40US1-noQRizaQ_2OtaFi66bAkuKLLkaaBJGuFTmrkvYxWGgzDXvIG

If you do copy-on-write then TO-BINARY some-kwatz-value would be 
able to work on the same data, at least until you modified it. That 
would make the binary-vs-kwatz distinction less important. Nonetheless, 
Cyphre's solution of having an internal flag on the binary! type 
to mark it as kwatz! and a KWATZ? function to test for it is similar 
to the fake datatype trick used in R2 2.7.7 and 2.7.8.

My answers to questions will follow in the next couple of posts. 
I've tried to answer the questions as precise, as I could, and thanks 
for the interest.

Q: Will this language be an open or closed source project?


A: Long answer: The plan is to fully open source it at some point, 
when version 1 is ready. Host depending sources are open in the alpha 
release, and it may make sense to open source more and more along 
the way, like sources for the different datatypes. The IT world is 
constantly changing. My nephew just went to the Devoxx Java conference 
in Belgium. Everybody had portable Mac computers. That was very different 
just a few years ago. Who knows, what devices we'll use in 5 years? 
I like to bring my software with me to new platforms. To be able 
to support new platforms faster, open source is a good thing. But 
some things needs to be fulfilled, before I'll open source it all.

1. This isn't a hobby project. I've invested a lot of time and work 
in this project, and I need to find a way to get something back from 
all that. One option is to have a good manual ready for developers 
to buy, open source it to get a lot of developers interested, and 
make a profit that way. There may be other ways (like someone or 
some company paying me to open source or make projects in World).

2. I need to know more about the consequences of open sourcing it, 
so I can avoid any nasty surprises. Knowing more from other similar 
projects could help. I need to figure out a proper license.

3. Even if I keep the sources simple and clean, I would like to clean 
up even more, before it can be open sourced.

A: Short answer: Currently it's partly open source.

Q: What is the main target?

A: The World prompt is a very powerful tool. Being what is known 
as "network transparent" helps in the era of the Internet. Server 
scripting and client scripting are obvious uses. The ability to easily 
make dialects makes it possible to wrap the language around the problem 
instead of trying to change the problem to fit the language. Interfacing 
with other technologies ... see "Introduction" on http://world-lang.org
in a few days.

Q: Is it closer to R2, R3, RED, Topaz architecture?

A: World run its code within a custom made virtual machine (VM). 
Compiling in World compiles World sources to this VM. This happen 
when source code is run or if explicitly compiled. REBOL is always 
interpreted (AFAIK), so that's different. I'm far from an expert 
in Red and Topaz, but as I understand it, Topaz is compiled/translated 
to JavaScript. So the JavaScript engine can be seen as the VM in 
World. And Red is compiled to native code, so that's different too. 
As I see it, all these languages work differently and can complement 
each other. (Help me here, if I'm wrong in any of this.)

Q: What was the main motivation, aka - why yet another language?

A: After I've learned REBOL, I find it frustrating to program in 
most (all) other languages. The difference is like when humans moved 
from carving words in stones to using a printing press a la Gutenberg. 
Then I found, I couldn't complete projects like my Canvas RPaint 
program on all major platforms (Win32, OS X and Linux), I couldn't 
run my code on new portable devices, and I needed better performance 
with scientific applications. With my background in graphics, music 
and science, I expect the language to develop into those areas.


Q: If (you) don't have any other target, why don't you work on Red? 
Or fork a project from Red?

A: I started initial work on World in late March 2009. The first 
post in the Red group here in AltME is from 27-Feb-2011. At that 
time, World was more than 7'000 lines of C.

Q: Does World compile into bytecodes (a la java) or machine languages?

A: Into bytecodes for the virtual machine. Each VM instruction is 
32 bytes (256 bits) including data and register pointers.

Q: Can you do operators with more or less than 2 arguments?

A: Not yet. I've considered post-fix operators (1 argument), and 
it shouldn't be too hard to implement. To motivate me, I would like 
to figure out some really good examples. With more arguments, I can 
only think of the ternary operator ("THE ternary operator"). I'm 
not sure, World needs that.

Q: Is range! a series! type?

A: No, range! is a component datatype. It has two components just 
like pair!.

Q: What platforms are supported?

A: For now Mac OS X (64 bit), Linux (32 bit) and Windows (Win32). 
The code is very portable. It took me a few hours to port to Linux 
from OS X and just a few days to Windows.

Q: What platforms do you plan to support in the future?

A: It would be cool to see World on all thinkable platforms. I personally 
don't have time to support all. World is not a hobby project, and 
I'm open for business opportunities to support other platforms. The 
host depending code is open source. I mainly think 64-bit.


Q: I'm a little sorry to see the R2-style port model instead of the 
R3 style. Are all ports direct at least?

A: Yes, ports are direct (no buffering). The ports and networking 
are some of the most recent implemented. More work is needed in this 
area. I would like to keep it simple and fast, yet flexible so we're 
all happy.


Q: What in the world is going on with the World Programming Language? 
This looks like something that must have been under wraps for a long 
time. What's getting released?

A: I didn't speak up about this, until I was sure, there were no 
show-stoppers. The open alpha of World/Cortex is being released as 
executables for Mac OS X, Linux and Windows (Win32), as are the platform 
dependent sources and initial documentation. World implement 74 natives 
and more than 40 datatypes. The Cortex extension (cortex.w) implement 
100 or so mezzanine functions and some definitions. The REBOL extension 
(or REBOL dialect in rebol.w) implement close to 50 mezzanine functions 
(not all functionality) and some definitions.

Q: Did you do some speed benchmark? (R3 vs R2 vc World) ?
A: Yes:

(All tests under OS X using R2 v. 2.7.7.2.5 and R3 v. 2.100.111.2.5)

- A mandelbrot routine (heavy calculations using complex! arithmetic) 
is 6-7 times faster in World than code doing the same without complex! 
in R2 and 11-12 times faster than R3. If using same code, it's 2.5 
times faster in World than R2 and 4.2 times faster than R3.
- A simple WHILE loop like:
n: 1000000 while [0 < n: n - 1] []

is 1.8 times faster in World than in R2 and 2.8 times faster than 
in R3.

- I tested networking in two ways. One sending one byte back and 
forth between client and server task 100'000 times using PICK to 
get it, and another sending 1k bytes back and forth 10'000 times 
using COPY/PART to get it from the port. Both were around 3 times 
faster in World than in R2. (I didn't test this in R3.)

- I tested calling "clock" and "tanh" routines in the libc library. 
I called those routines 1'000'000 times in a loop and subtracted 
the time of the same loop without calling. Calling "clock" is 2.4 
times faster in World than in R2. Calling "tanh" (with argument 1.0) 
is 5.9 times faster in World than in R2. (I didn't test this in R3.)


(Some functions are mezzanines in World, which are natives in REBOL, 
so they'll in most cases be slower in World.)

Q: We already know that your physics background influenced the new 
complex numbers datatype. Should we expect further progress of this 
kind (physics/science)?

A: After pointing this language out to a couple of friends from university, 
I was quickly asked to give scientific examples, like making a Lorentz 
attractor. There will come examples like that. Also I have some contacts 
at the Niels Bohr Institute (Copenhagen University), that I would 
like to show the language to and see, if we can create some projects.


Q: How long has this been in the making in general & how much time 
have you spent programming it?

A: I started R&D late March 2009. In March 2010, I had >7'000 lines 
of C. Then I took almost a year break from World and started up again 
Spring 2011. I have used much time on it this year (2011). So I've 
used 1-2 years effectively, I guess.


Q: Is there a way loading and interfacing pure object files, how 
about callbacks?

A: I don't have much experience in this area. Two of the closed alpha 
testers have looked at interfacing with sqlite3, which uses callbacks 
for some stuff. I would say, it doesn't work 100% yet, but it's being 
worked on. I need to see examples of loading and interfacing with 
pure object files to judge, how much of such functionality should 
be in World.


Q: Wouldn't it be useful to treat the kwatz! type same way as binary! 
so you don't need to always convert it? I mean, all functions that 
are able to take binary! arg should be able to use the kwatz! as 
well...or do you think are there any problems regarding that?

A: I've thought of that and came to the decision, that KWATZ! should 
be treated with some care. Are we always sure, we want to treat that 
data as binary? I need more experience using KWATZ! to judge that. 
Conversion could be fast, if the AS function was introduced, which 
should just change the type without copying.


Q: Is there a call-in interface available, meaning I can embed World 
in other programs?

A: Not yet. Internally I do call World functions from C (to parse 
URLs) by pushing arguments on the VM stack and call the execute_vm() 
C function. I imagine an interface much like in Lua (lua_call), but 
a little more work is needed, before we're there. This is an alpha 
release, so things will change/be added, before we move to beta release.


Q: Regarding your thesis I guess you have something like an integrated 
db or a special datatype for permanent storage too.. ?
A: Too early! :)

Q: What subset of REBOL2 will run without change in World?

A: Uh, ah, hard to tell at this point. When I need new functionality 
in World (because I want to run some of my R2 scripts), I consider, 
if it should be part of World/Cortex or if the new functions should 
go into the REBOL expansion/dialect (%rebol.w). I want World/Cortex 
to be small and compact. The idea with %rebol.w is, that much of 
R2 code could run, after this script has run.


Q: Are there datatype and function comparisons between World and 
REBOL2?

A: No, I haven't documented that ... yet. Maybe someone else wanna 
document that!? But there are differences, like the REBOL decimal!, 
which in World is called real!. And then %rebol.w just include the 
line:
decimal!: :real!
, so REBOL scripts using that will run.


Q: How many people were working on this World? :) Is it just a single 
man project?
A: Yes, just me.

Q: Will there be a GUI?

A: I would really need a GUI for my own work at some point. I have 
ideas, but nothing set in stone yet. And I want World to be open, 
so different GUIs should be possible, also the native GUIs in the 
different OS.


Q: Are you rich enough to buy Rebol Technologies and employ Carl? 
:-)
A: [KWATZ!]

When we tried something similar to the LOAD/on-kwatz trick in R3 
(named LOAD/on-error there) it turned out to slow down LOAD so much 
that it made it unusable - something about switching back and forth 
between native and interpreter execution schemes. The alternative 
of having a kwatz! type returned had the effect of changing a load-time 
check for syntax errors in the source code into a runtime check for 
invalid values, but only if you are expecting such things and testing 
for them explicitly. Under normal circumstances a kwatz-accepting 
loader would often result in code that would run slightly wrong but 
not trigger an error to help you figure out why. This is why you 
should make sure that your regular code loader triggers errors by 
default when it encounters unexpected syntax. An option to be more 
forgiving would be helpful when loading dialects that explicitly 
support particular kinds of extended syntax - and thus are expecting 
kwatz! - or for loading user data for analysis.

Glad to hear that C-like code is faster in World than in REBOL :)

Overall the language seems rather Lua-like, with the same advantages 
and disadvantages of that model.

Yeah. I'm also glad, it's faster. That was some of the first tests, 
I did, long before you could call it a language.

One of the main reasons for making R3 was Unicode support. How it 
looks in the World?

I've dealt with unicode in some REBOL projects, simple by making 
functions to deal with it. I would initial do it like that in World, 
but on sight, a unicode! datatype would probably be a benefit.

The lexer is 7 bit, so words can only hold 7-bit ascii characters. 
String and other data is 8-bit.

REBOL can be blazingly fast if you use it for REBOL-like tasks. It 
has a different set of tradeoffs. Unfortunately for its image, most 
language-vs-language comparisons tend to have C-like code since it's 
the common denominator, and that makes languages that aren't optimized 
for that style of code look bad. However, if you have to do C-like 
tasks, a language that isn't optimized for them won't help you very 
much. This is one of the reasons that I like the REBOL spinoff languages. 
For me R3's model leads to the fastest code, but I don't have to 
do much C-like stuff.

I like to keep things simple, and I feel, implementing unicode everywhere 
very fast make things very complex.

Words are probably ok as ascii, but unicode! datatype is a must if 
you don't want to end with binary data instead which is doable like 
in R2, but ugly.

Wow.. another language!


I saw it mentioned few times that it's like Lua.. can someone explain 
it to me since I love some aspects of Lua very much and I don't know 
what you mean? (except I saw somewhere values are pushed to stack 
/ between c - world (which is probably similar to Lua)

I fear, that implementing unicode! datatype will mean, all series 
functions will have to deal with it separately, like LENGTH?, NEXT, 
SKIP, etc. Would it be an idea to make a unicode! datatype just to 
pass data back and forth and be able to view it somehow, but without 
support for all the series function?

@Janko, it's about the fact, that Lua implement a virtual machine, 
that runs the vm asm code, after Lua source has been compiled to 
that. World does it the same way, but in many ways, World is also 
very different from Lua.

Isn't 32bytes per instruction too much?

REBOL on the other side, interpret the REBOL source all the time, 
like every time the same function is called. In World, the function 
is compiled to VM code, the first time, it's called, after that the 
compiled code is just run (unless the source is compiled again).

Oldes, complex numbers hold 2 double = 128 bit, then OP-code, register 
pointer and maybe some more.

Janko: Compiled to a byte-code VM, it looks like a similar embedding 
model. The advantages of bytecode VMs are that they can be tuned 
to be faster for low-level operations that can match the VM author's 
needs, though most are optimized for C-like code, simple math and 
loop evaluation. The embedding model makes it so that you can embed 
in a C-language program (lowest-common-denominator) without affecting 
the language itself much, though the actual interface is slow enough 
that efficient code tries to avoid going back and forth across that 
barrier. Disadvantages are that higher-level operations tend to be 
slower than the lower-level operations that have bytecodes associated 
with them.

thanks for explanations to both of you. so it's more of a backend 
similarity

I wasn't aware of byte-code VM specifics .. so thanks for that too

Thanks for the comprehensive Q&A, John.

@Janko, look at "Countdown: 1" tomorrow.

You're welcome, Andreas.

I also think that 256-bit VM insn size sounds a bit wasteful. That'll 
thrash the data cache easily.

I'm not sure that holding complex values as immediates is worth it, 
even for complex-heavy code.

REBOL code is interpreted, but not its source. The slow part of a 
source interpreter is parsing the source into the intermediate code, 
the AST. REBOL is an AST evaluator. The advantage to that relative 
to a bytecode VM is that you can extend the runtime with more fast 
operations without breaking the bytecode encoding, but the disadvantage 
is that the interpreter overhead is larger so if you want your operations 
to be efficient you have to use larger ones. This is why C-like code 
is slow in REBOL, but high-level code can be fast.


If you want to get the advantages of a bytecode VM with the extensibility 
advantages of REBOL's model you could go with an address-threaded 
interpreter. Address-threaded interpreters have more data going through 
the processor than bytecode interpreters do, but it you need to support 
higher-level operations they are more efficient overall. However, 
if you don't need to support higher-level operations and only need 
to support a tiny number of low-level operations then bytecode can 
be encoded in a much smaller amount of space. If your language is, 
for instance, a spreadsheet formula evaluator then you might even 
be able to have 4-bit bytecodes, with two operations per byte, and 
have an interpreter that fits entirely in the instruction cache of 
a processor. Bytecodes can be much faster then.


Still, Lua's bytecode VM, as efficient as it is, has been running 
into performance limits as well. Fortunately, a bytecode model that 
maps well enough to the native code model (remember what I said earlier 
about C-like bytecode VMs?) can have the bytecodes translated to 
native code at runtime and then execute the native code. For C-like 
code that is usually even faster than address-threading. This is 
why LuaJIT has been doing so well when compared to Lua's bytecode 
VM.


World being Lua-like means that it can improve using methods similar 
to the ones that Lua has been using to improve. That's definitely 
a good thing, since it means that Geomol doesn't have to work from 
scratch :)

As you talk about registers in context of your VM, I assume that 
your VM is register-based, right? (Not stack-based.)

If you want to compare to a source interpreter, try old versions 
of TCL before it switched to bytecode interpretation. That was *slow*, 
not like REBOL at all.

If so, I'd probably try splitting immediate complex-loads into two 
insns. Then reduce insn size to 128-bit (if possible) and check the 
effect on code size.

Just a thought, though. You (John) probably already tried that :)

Looking very much forward to playing with World.

Uh, I'm not 100% on the technical terms. Registers are on a stack, 
and a register pointer is an offset. So you can have lots of registers, 
and the VM can access each of them very fast. Values can also float 
in memory (not on stack), and then we have a real pointer to it. 
See blog tomorrow.

Ah, then it's stack-based with deep access into the stack.

ok

Can you access registers that aren't on the top of the stack? Direct 
addressing, rather than stack operations?

Is that good? :)

Brian, yes.

That's not why I was asking :)