World: r3wp

I could screen for it in DO-NEEDS, but that wouldn't help with everywhere 
else the modules list is accessed. It would really be better to not 
have any optional keyword at all, but there's that backwards-compatibility 
thing. We'll have to resolve this before we have a real R3 release.

For now, you can have a module named 'core, but you might run into 
gotchas when trying to use it.

Should we do it here?

Sure, let's show how the sausage is made :)

If you are adding a module to the module list, and there is an existing 
module of that name, then the new module either overrides it, replaces 
it, or doesn't get added (possibly with an error triggered, but so 
far not). The question is which one to do in the particular circumstances. 
The factors are whether it is the same module, for whatever "same" 
means here considering it might be reloaded or still source; whether 
the versions are the same or greater; whether the existing module 
has already been made or is still source, and the same for the module 
to be added.

There's also the option of keeping both. That is, my code may have 
bindings to the first instance.

(Maybe that's the "overrides" option.)

Yes.

So far, my guess is that

- Premade modules can't be delayed since their code blocks and side 
effects have already been executed.

- Delayed modules can't be added unless they have at least the version 
of an existing delayed module, or more than an existing imported 
module.

- Premade modules can't be added unless they have at least the version 
of an existing delayed module, or more than an existing imported 
module.

- If a overriding module is added and the existing module is delayed, 
the existing reference should be replaced, not overriden.

- If a overriding module is added and the existing module is already 
imported, the new module overrides the old but the old reference 
is still there.

- If a delayed module is a mixin, the module is made and returned, 
but the stage-two delayed source is kept in the list.
- I'm missing something.

The questions I have are:

1. What do we do when a module is not added due to a policy issue? 
Currently the add accessor returns none if it is a version issue, 
and triggers an error for a checksum violation.

2. How do we determine (officially) that two modules are to be considered 
the same? Name and version?

2. Can we safely LOAD-EXTENSION more than once with the same extension?

3. Does LOAD-EXTENSION on an embedded extension have any side-effects 
beyond creating an object?

4. ... return the same source each time, or different copies of the 
same source? Testable by SAME?

5. Is is safe to delay the object returned by LOAD-EXTENSION instead 
of the source?

6. Should the checksum of an extension include the extension-specific 
source added?

7. Should the version in the header of a module be set to 0.0.0 if 
not a tuple? Currently it is.

8. If so, should module checksums be done after the version field 
is fixed?

Questions and answers posted here: http://www.rebol.net/wiki/Module_Design_Details

Will be edited as the remaining design decisions are made.

Awesome.  Just read through the messages here and will be reading 
the document you linked shortly.

I haven't added documentation for the individual functions yet, since 
they are not yet done. Well, most are, but there might be changes 
required to implement the rest. That is a guru-level doc, but it 
could be used as source material for user-level docs.

Excellent.  I'm looking for guru-level docs.  


If I'm reading this correctly, I will finally have everything I asked 
Carl for (in regards to Object! enhancements) at the '04 Devcon. 
 I'm going to spend some time stubbing out the functions I will need 
to do the advanced IDE tricks I've been wanting to do.  I'll have 
some questions for you once I get some of that done...

I'll be done before then. Yeah, a lot of the proposed object enhancements 
have been done to modules instead. Objects will likely still get 
enhanced in some ways, but modules can better handle a lot of what 
objects were previously used for.

In theory, it should be possible for delayed network protocol modules 
to be autoloaded on first use.

The modules that implement the protocols could be delay-loaded and 
registered with the general protocol dispatcher. Then the dispatcher 
could import the module the first time it is needed.

Yes. Could be made even simpler, though.

Bundle the modules as what Carl now calls "optionally included". 
Also keep a list of scheme prefix to module name, and just auto-import 
the module from this list when a scheme is used in on of the scheme 
action functions (READ, OPEN, ...).

But well, I guess that in fact is exactly what you also describe 
:)

Yup :)

That kind of seamless autoimport is one of the main advantages to 
the import-to-system model of the R3 module system. Even if you did 
explicit import, such as with mixins, the rest of the dependencies 
would be able to be resolved automatically.

You don't necessarily need to explicitly import regular modules in 
order to use them, but you do need to do so for mixins. Mixins are 
like the modules in Maxim's or Gabriele's module systems, more or 
less. As such, mixins will only be used by the kind of advanced programmers 
that already need to write modular code. Regular programmers won't 
need them.

So if, for example, DECODE-CGI is in an "optionally included" (which 
I guess == "delay-loaded") CGI module, DECODE-CGI will be available 
even if you don't import 'cgi first?

A regular, non-delayed module could include a delayed module that 
does the real work. And there are other tricks that can be done.

Well, I personally see not cluttering the "global" namespace with 
those names as a great advantage.

A lot of those names won't need to be exported at all. Internal use 
words can stay internal.

The wiki link above has some other ideas for advanced tricks, though 
that page is still being written.

And it's not user-level, so be warned: Here there be dragons :)

OK, the third rewrite of the module system has a few simplifications 
as related to the (unreleased) second rewrite, but it is mostly the 
same design. I will be revising the wiki above with the changes.

Now that issues are no long strings but words, maybe we want to revisit 
the decision to introduce EXPORT and HIDDEN keywords. In http://www.rebol.net/r3blogs/0300.html
the main reason not to use #export and #hidden seems to be the nature 
of issues.

I'd actually prefer to use #export and #hidden, as I rather dislike 
the concept of "module keywords".

I think that most of guys did not accept changes to issue dtype ... 
maybe it should be put back to original functionality, no?

That is a separate discussion.

Currently we have the issues-as-words, and it does not look like 
they are going away, but rather their semantics be adapted to address 
the concerns raised.

*VERY* good catch !!!

pekr, most of whined and tried to put the change in our perspective, 
yes  :-)

Pekr be happy i absolutly don't care about any issue ;) aaaaaaaaaahahahaha

so much of a thing used by 3 guys around the world. That impress 
me.

Well, I hope that we are able to make things good enough for you 
without your help. Wish us luck!

hum that was the cas before too no? You will need luck that's a sure 
thing.

It wasn't a request for assistance. If you want to help, it will 
be welcomed. If not, we are doing well now. Either way you are free 
to use what we create, or not. Enjoy!

Over the last day I have been fleshing out the test suite for the 
new module system. Almost all features have corresponding tests now. 
As soon as we have complete coverage I will try to get the tests 
published somewhere official. FYI, alpha 109 fails a lot of the tests, 
but also in the last day I fixed all of the errors that were causing 
test failures. In theory - I need a new build with the fixes to be 
sure, but code tracing says the behavior matches the tests. In any 
case, the test suite will have full feature coverage before alpha 
110 comes out.

I have also started writing some simple charts to explain the details 
of the design and behavior of the module system. In CSV format. These 
charts helped a lot in the fixing of the problems and implementation 
of the tests. As with the tests, I will try to get the charts published 
somewhere official.

With alpha 109 we got some significant usability revisions to the 
design of the module system, relative to alpha 108:

- The return of unnamed modules. They are now changed to private 
modules (mixins) which aren't stored in the system modules list.

- IMPORT now effectively works a lot like the Needs header in user 
scripts. Most users won't be able to tell the difference.

- The return value of IMPORT block is now a block of the modules 
you imported (but not the modules *they* imported).

- The refinements of IMPORT have been renamed and their behavior 
tweaked to be nicer and more useful - the first API change since 
Carl's original.
  - /no-share: The previous /isolate option. Same behavior.

  - /no-lib: Don't export to the runtime library. Private modules don't 
  do this anyways. Also, don't add to the system modules list.

  - /no-user: Don't export to the user context, even as a private module. 
  When importing to a module, /no-user applies.

- The old /only option was split into /no-lib and /no-user, for more 
control. Specify both if you don't want IMPORT to export anything.

Alpha 110 should bring these changes:

- The above will work properly. With a bunch of specs and charts 
that define what "properly" means. With a full test suite to make 
sure.

For the sake of completeness, here are the highlights of the alpha 
108 changes:

- Script headers can have an options block, a simple block of flag 
words. User extensible.

- The standard script header now has a lot fewer words in it. More 
stuff is optional or in the options block.

- Script compression, either binary and base 64 binary! encoded. 
Automatic, transparent.

- Script checksums, both to verify the script and for IMPORT to compare 
with. Applies to decompressed source.

- An optional script length header field (like http's Content-Length). 
This allows binary script embedding.

- Internal support for getting the end of an embedded script, so 
a multi-loader is possible.

- The 'content and 'isolate header fields are changed to option words. 
The content is still saved to a 'content header field.

- The 'content field, if set, is set to the start position of the 
script proper, even if there is stuff before it.

- The whole system/contexts/system concept is gone, as part of the 
system restructuring. Now we have SYS.

- The system/contexts/exports concept is gone too, replaced by a 
not-module-specific runtime library called LIB.

- The old type: 'extension is now the 'extension header option word. 
The only module type is 'module. And it's optional for most code.

- Mixins are now called "private modules", and are flagged by the 
'private option word. And they can have names.

- Private modules can be added to the system modules list (because 
of the names). This lets them be reused without being reloaded.

- Unnamed modules are now prohibited (until alpha 109, where they 
become private modules that reload every time).

- Delayed modules, which can be partially loaded and then not fully 
made until they are imported. Use the 'delay option word.

- A HIDDEN module source keyword, which applies PROTECT/hide to a 
word or words. Acts like the EXPORT keyword.

- Better errors are triggered when the bad things happen. Including 
new error codes.

- DO and MAKE--MODULE intrinsics are now in sys, as DO* and MAKE-MODULE*. 
No more system/intrinsics.

- DO-NEEDS is no longer exported (it's in sys). IMPORT block is a 
public alias for DO-NEEDS anyways.

- MODULE now makes modules that act more like those in script files. 
And has /mixin support too.

- A whole bunch of changes and fixes to native functions to support 
the above stuff.

Shadwolf's "used by 3 guys around the world" comment brings to mind 
one of the more ironic things about the module system:


Most user code for R3 will be written in "scripts", not "modules". 
This will be even more the case once we get more of concurrency working, 
because "script" code works in the user context, which will be task-local. 
We are going out of our way to make it extremely easy to just use 
"scripts" and not have to bother with "modules".


The ironic part is that "scripts" are just another kind of module, 
one of the three including regular and isolated modules. In particular, 
user scripts are a kind of module that we try to make as non-module-like 
as it is possible to be (given that they run in a module system). 
The entire module system structure is built around the challenge 
of making the module system apparently disappear, or at least be 
something that you can be almost completely ignorant of. The module 
system is built for script programmers, to let people do PITS on 
a systerm that they don't even have to know is capable of the most 
advanced PITL.


So the module system we are discussing here will be used by *everyone 
who programs in R3*, whether they know it or not :)

(I am politely assuming that Shadwolf was not referring to the entire 
REBOL community when he said "3 guys".)

can modules be released as encrypted code ?