World: r3wp

OK, what is 'sys level precisely good for?

Maybe we should explicitly add 'all feature, which would be just 
the same as not specifying boot-level at all ...

No, it wouldn't, because there are still a few opt-in modules that 
are included but not imported by default. For instance, modules that 
implement conflicting functionality, such as CGI vs. GUI.

This stuff will require very precise documentation and examples, 
explaining e.g. how user can save some cycles for CGI purposes, yet 
how he/she can load his own framework (modules) etc.

The 'sys level is good for really low-level programming, replacing 
the mezzanines with your own, testing host code, all sorts of things. 
And it will enable the few users in the REBOL community who write 
their own module systems to use them instead of the standard one. 
You might be able to build sandboxes at that level too.

The best would be to have complete map of what functions are available 
for what level ...

Which has not yet been decided. For that matter, there is also the 
question of *where* they are available at different levels. It is 
possible that some stuff put into 'lib will be transferred from other 
contexts in the system catalog.

I like the SDK aproach - you get your base, and then you include 
any mezz funcs/packages you want. This all now sounds so cryptic 
...

I mean - without user not precisely understanding new module system, 
all the bindings, he has absolutly no chance to understand, what 
we are talking about ...

It only sounds cryptic because it isn't done yet. A system redesign 
has been planned for a while now, but we had to nail down the semantics 
of the language first and fix a *lot* of bugs.

This should become part of docs asap - http://www.rebol.net/r3blogs/0334.html

The module system, the boot levels, it is all progressive-reveal. 
By default, things are really simple for users and there isn't that 
much to understand. As you learn options you can make the system 
do more tricks, but the default is really simple.

Are you still working on module system?

I am putting the finishing touches on the initial implementation. 
More tweaks will be necessary for the integration (mostly renaming 
variables), but the new system model will simplify things a bit. 
In particular, the predefined 'sys word makes the module-tools mixin 
unnecessary, which simplifies the bootstrapping and use of the system 
immensely.

Why the rewrite, if I may ask? I thought your module code was already 
finished?

Well, there was a new feature required (delayed modules) that when 
added had wide-reaching implications for the rest of the module system, 
and caused a bunch of other features and the improving of others 
as a side-effect. Also, the old module system violated the "hit by 
a bus" principle: I was the only person who understood the code fully, 
so noone else could modify or enhance the code, and if I went away 
the code would become instantly unmaintainable. So the new code is 
easier to use, does more, and can be understood by the average REBOL 
guru. Believe me, it was that last part that has been the trickiest.

BrianH note that if you read my blog posts, I am rooting for 'RESIDENT... 
not 'EXPORTS.


and thanks for your better explanations they shed a little bit more 
light on the whole thing.

Don't handicap the implementors of R3 just because you want to handicap 
yourself.


Don't harm the users of R3 just because you are a lazy implementor.

But it will be easier to discuss this once there is a concrete implementation 
to play with and discuss, not just vapour.

And in any case, even having 'lib defined in 'lib won't stop script 
writers from just ignoring it, continuing to use the LIB name for 
their own purposes, and refer to the LIB context as system/contexts/lib 
if they ever need it.

See, that last message is exactly what I was recommending, Andreas. 
You have just pointed out that there is no harm whatsoever to defining 
the 'lib word in 'lib. Though there is no evidence of the "lazy" 
in your earlier comment.

http://www.rebol.net/r3blogs/0339.html

Module issues.

uh oh, now I fear Carl will scrap BrianH's work, and we are going 
to wait for 3 months for Carl to come-up with his own version :-)

I hope this is not the case, however following is scary, while understandable 
-  ".... a fight between simplicity and complexity, between maintainability 
and chaos, between elegance and ugliness"

no Brian is doing the module work, its just that the changes to how 
the contexts are now layed out provoke deep changes in how things 
are bound.


because that is a big part of the module system's job, it means Brian 
has to update a lot of the code.


also remember that Brian has been splitting up the module code into 
sub-functions, so all of that makes it simpler, and more re-usable.

Earlier prototypes worked well. They were functional, clean, and 
simple. Understandable.

 - that simply means, that Carl does not like something about current 
 system ...

They where coding at opposite ends of the spectrum, now they are 
fighting to merge the two together.  that's how I read it.


IMO its just a question of getting it to work again. with new contexts 
layout and new, better module functionality.

AFAIK, Brian's code is usually accepted.

I like the following part :-) "Some of you may be saying "Carl, we 
don't care." Yes, I know, I've heard that before. But, (if I had 
a Yoda voice, I'd use it here) you will care. You just don't know 
it yet."

The module system has grown to include many advanced features, which 
are all usefull and viable.  Brian has been working to simplify the 
code by breaking it up into smaller pieces.   At some point you can't 
have features without at least a minimal amount of code.


The new contexts layout makes it a more complex task because my guess 
is that basically, its broken everywhere and that is hard to debug.


The original module system wasn't very powerfull in the sense that 
it didn't add much more than special objects... what Brian is doing 
is sooooo much more than that.

So I don't think Carl and Brian are fighting... hehe

I think they are fighting the code  :-)

Henrik - I know, that is just why I am surprised by Carl stating, 
that he liked to early prototypes. This just seems to follow the 
Gab's VID scenario - Carl detached from Gab's VID for quite some 
time, then having difficulcy to understand it, then - scrapping it.

Max - I hope so :-)

I've looked at the module code and it was a large and barely understandable 
by its complexity.  Even by Brian.   He was already working on this 
problem... its just that now that its all broken, they can't ignore 
it anymore  ;-)>

We weren't ignoring it, trust me. I have been working on integrating 
the new features, which resulted in a redesign of the semantics. 
But the code was intimidating even after the rewrite, and there were 
some repeating awkward code patterns that needed native replacement, 
which I couldn't do. The new module system that Carl and I are working 
on is based on my work, even if the code may not resemble my code 
on a surface level.

Main changes so far, relative to my recent work:

- A different code style which Carl thinks will be easier to read 
and maintain (mostly using CASE/all).

- Some of the more awkward repeating code patterns have been split 
out into functions, which in some cases will be made native.

- The API of LOAD has been simplified, though is more flexible. Its 
behavior with no options is the same, but some options have changed.

- Some functions are renamed, and some code has been moved from function 
to function (this was expected).

- The sys and lib contexts take the role previously planned for the 
exports context and module-tools mixin, though the usage is the same.

- Fewer functions will be exported into lib than I was expecting. 
The short names of the lib and sys contexts enable this.

Those first three were tricks that I couldn't do, because I don't 
set the standard APIs and don't write the natives. But the core semantics 
are quite similar to my recent work.

Btw, the reason why I didn't use CASE/all before is because I thought 
it would make things trickier. Apparently it made things easier. 
Who knew? :)

me :-)

Well, yeah, of course you :)

I have been recently starting to use CASE... funny how we can discover 
new code patterns after over a decade of using a tool.

BrianH, could you show us a before & after example of code modified 
to use CASE ?

Here's a low-level function to parse and process script headers, 
which shows how many features are built into the base script model 
in R3:

load-script: funct [
	"Decode a script into [header-obj script-ref body-ref]"

 source [binary! string!] "Source code (string will be UTF-8 encoded)"
	/header "Return the header object only, no script processing"

 ;/check "Calculate checksum and assign it to the header checksum 
 field"
	/original "Use original source for Content header if possible"
] compose [
	data: either string? source [to-binary source] [
		unless find [0 8] tmp: utf? source [ ; Not UTF-8
			cause-error 'script 'no-decode ajoin ["UTF-" abs tmp]
		]
		source
	]

 ; Checksum all the data, even that before the header or outside the 
 block
	;sum: if check [checksum/secure data]  ; saved for later
	
	if tmp: script? data [data: tmp] ; Find the start of the script
	
	; Check for a REBOL header
	set/any [hdr: rst:] transcode/only data
	unless case [
		:hdr = 'rebol [ ; Possible REBOL header
			set/any [hdr rst] transcode/next/error rst
			block? :hdr ; If true, hdr is header spec
		]
		:hdr = [rebol] [ ; Possible script-in-a-block
			set/any [hdr rst] transcode/next/error rst
			if block? :hdr [ ; Is script-in-a-block
				unless header [ ; Don't decode the rest if /header
					data: first transcode/next data
					rst: skip data 2
				]
				true
			] ; If true, hdr is header spec
		]
	] [ ; No REBOL header, use default
		hdr: [] rst: data
	]
	; hdr is the header spec block, rst the position afterwards

 ;assert/type [hdr block! data [binary! block!] rst [binary! block!]]
	;assert [same? head data head rst]
	
	; Make the header object, or fail if we can't

 unless hdr: attempt [construct/with :hdr system/standard/header] 
 [
		cause-error 'syntax 'no-header data
	]
	; hdr is a correct header object! here, or you don't get here

 ;if check [append hdr 'checksum  hdr/checksum: sum]  ; calculated 
 earlier

 ;assert [sum =? select hdr 'checksum]  ; Should hdr/checksum be reserved?
	

 if header [return hdr] ; If /header, no further processing necessary

 ; Note: Some fields may not be final because post-processing is not 
 done.
	
	; Skip any whitespace after the header

 ws: (charset [1 - 32]) ; For whitespace skipping (DEL not included)
	if binary? rst [parse rst [any ws rst:]] ; Skip any whitespace
	
	; Check for compressed data and decompress if necessary
	case [
		; Magic autodetection of compressed binary
		tmp: attempt [decompress rst] [
			data: rst: tmp  ; Use decompressed data (no header source)
			append hdr 'compressed  hdr/compressed: true ; Just in case
		]
		; Else not directly compressed (without encoding)
		(select hdr 'compressed) != true [] ; Not declared, do nothing
		; Else it's declared to be compressed, thus should be
		binary? rst [ ; Regular script, check for encoded binary
			set/any [tmp rst] transcode/next/error rst
			either tmp: attempt [decompress :tmp] [
				data: rst: tmp  ; Use the decoded binary (no header source)
				hdr/compressed: 'script  ; So it saves the same way
				; Anything after the first binary! is ignored
			] [cause-error 'script 'bad-press -3] ; Else failure
		]
		; Else it's a block, check for script-encoded compressed binary
		tmp: attempt [decompress first rst] [

   data: rst: tmp  hdr/compressed: 'script  ; It's binary again now
		]
		; Else declared compressed but not compressed, so fail
		'else [cause-error 'script 'bad-press -3]
	]
	
	; Save the script content in the header if specified
	if :hdr/content = true [
		hdr/content: either original [source] [copy source]
	]
	

 ;assert/type [hdr object! data [binary! block!] rst [binary! block!]]
	;assert [same? head data head rst]

 reduce [hdr data rst]  ; Header object, start of source, start of 
 body
]


Note all the commented assert statements: they are for testing (when 
uncommented) and documentation. Also, I later removed the checksum 
calculation from this code because it was the wrong place to put 
it, at least as far as modules are concerned. However, Carl didn't 
know this because he was working on it while I was offline for a 
few days.

Here is the corresponding function in the code reorg, renamed. The 
friendly empty lines and comments haven't been added yet.

load-header: funct/with [
	"Loads script header object and body binary (not loaded)."

 source [binary! string!] "Source code (a string! will get UTF-8 encoded)"

 no-decompress [logic!] "Skip decompression of body (because we want 
 to look at header mainly)"
][
	; This function decodes the script header from the script body.

 ; It checks the 'checksum, 'compress and 'content fields of the header.

 ; It will set the 'content field to the binary source if 'content 
 is true.

 ; It will set the 'compress field to 'script for compressed embedded 
 scripts.

 ; If body is compressed, it will be decompressed (header required).

 ; Normally, returns the header object and the body text (as binary).

 ; If no-decompress is false and the script is embedded and not compressed
	; then the body text will be a decoded block instead of binary.
	; Errors are returned as words:
	;    no-header
	;    bad-header
	;    bad-checksum
	;    bad-compress
	; Note: set/any and :var used - prevent malicious code errors.
	case/all [
		binary? source [data: assert-utf8 source]
		string? source [data: to binary! source]
		not data: script? data [return reduce [none data]] ; no header

  set/any [key: rest:] transcode/only data none ; get 'rebol keyword

  set/any [hdr: rest:] transcode/next/error data none ; get header 
  block

  not block? :hdr [return 'no-header] ; header block is incomplete

  not attempt [hdr: construct/with :hdr system/standard/header][return 
  'bad-header]

  :hdr/content = true [hdr/content: data] ; as of start of header (??correct 
  position??)
		:key = 'rebol [ ; regular script

   rest: any [find rest non-ws rest] ; skip whitespace after header

   ;rest: any [find rest #[bitset! [not bits #{7FFFFFFF80}]] rest] ; 
   skip whitespace
			case/all [

    all [:hdr/checksum :hdr/checksum != checksum/secure rest] [return 
    'bad-checksum]

    no-decompress [return reduce [hdr rest]] ; decompress not done

    :hdr/compress = 'script [set/any 'rest first transcode/next rest]
			] ; rest is now suspect, use :rest
		]

  :key = [rebol] [ ; embedded script, only 'script compression supported
			case/all [
				:hdr/checksum [return 'bad-checksum] ; checksum not supported

    no-decompress [return reduce [hdr rest]] ; decompress not done

    rest: skip first transcode/next data 2 none ; decode embedded script

    :hdr/compress [hdr/compress: unbind 'script  set/any 'rest first 
    rest]
			] ; rest is now suspect, use :rest
		]

  :hdr/compress [rest: attempt [decompress :rest]] ; :rest type-checked 
  by decompress

  not :rest [return 'bad-compress] ; only happens if above decompress 
  failed
	]

 ;assert/type [hdr object! rest [binary! block!]] ; just for documentation
	reduce [hdr rest]
][
	non-ws: charset [not 1 - 32]
]

Notes:

- The other half of the CASE/all style is a lot of explicit shortcut 
RETURN statements, whenever the normal flow differs.

- Errors are returned as a word from the error catalog, which is 
later passed to CAUSE-ERROR.

- Carl redid the checksum calculation so that scripts can verify 
against a checksum in their header, to detect corruption.

- The checksum in the header probably can't be used for the module 
checksum because the header itself matters for modules.

- Compressed scripts lost a couple minor, unimportant features that 
we are likely better without. Quiz: What features?

- Part, but not all of the reason the code is shorter is because 
the doc comments haven't been added yet. The CASE/all style helps 
though.

- The option of using the original data in the content field is now 
mandatory. If you need to copy it (rare), do so yourself.

I am not yet sure if using FUNCT/with is OK with the new build process 
(haven't heard back), but serialized values are now OK. This is why 
I have some alternate code with a serialized bitset.

The new code is not much less complex than the original, but it is 
more compact and faster too. And it is easier to maintain, because 
rearranging CASE clauses is easier to do without a full reorg than 
nested conditional code.

There are some other micro-optimizations as well in the new code. 
I was writing the original to determine functionality, not trying 
to prematurely optimize.

Thankyou, BrianH. Illuminating.

Is tasking close now? Express your opinion to proposed interpreter 
RESET functionality - http://www.rebol.net/r3blogs/0340.html