World: r3wp

Roughly summarising: an extension search path (preferred) + os loader 
search (fallback) + a cross-platform mechanism to load extensions 
via abstract names.


Could be as simple as reusing system/options/module-paths and load-extension, 
allowing the latter to take a word! parameter and also search the 
former. The possible extensions themselves are already in place (system/options/file-types). 
Loading is already done via dlopen, so loading a lib only by name 
(w/o any path component) will use the OS loader's search mechanisms.


And alternative would be to add a system/options/extension-paths 
(block!) option, and a dedicated loading primitive, say, import-extension, 
which searches the extension-paths and tries all possible platform-specific 
extensions (as per system/options/file-types).

good summary.  I'd like brian To add his grain of salt, as he might 
have other insight, but I think this allows us full control along 
with full flexibility.

(and minimal fuss when it doesn't matter)

Many of you are conflating distribution and usage here.

Exactly

yeah, I know.   but modules and extensions are very different things.

Yes, these are also being conflated in these arguments

how so?

As you said, they're different. For example, system/options/module-paths 
is needed for REBOL modules, but it doesn't work for extensions

You want an optional search path for extensions, and that's fine, 
but Andreas and I are pleased with the OS managing them

ok yes.

I'd also be fine with having module-paths used for extensions as 
well (before falling back on the OS search paths). In fact, I would 
prefer re-using module-paths over adding another extension-paths 
option.

While there wouldn't be a point to the OS managing REBOL modules, 
because it doesn't know what they are

But in any case, as you say, _don't_ take away the ability to just 
have extension search managed by OS mechanisms.

Exactly, and in a portable way

If you want to fully integrate R3 into an OS, that is a job for the 
host code. For most OSes you want as much isolation between the OS 
and R3 as you can get. Unless you have a locked-down OS (most don't) 
then what constitutes a "safe" extension would vary widely between 
programs. That is why the module-paths option and others like it 
are just set by default by the startup code, to be changeable by 
the program code as needed.

We do have a way to have extension search managed in a portable way 
by OS-specific means: LOAD-EXTENSION is implemented in the host code.

Yes, and all we need to do is add some simple filename management 
to it.

Which doesn't make any sense at all to replicate for each and every 
otherwise portable extension.

Sigh. I'm not going to spend another day rehashing that it's currently 
not portable, because of the very simple detail Andreas says

Yup. But keep in mind that for most potential R3 users (who like 
most users are running Windows, OSX or some stock Linux) the OS mechanisms 
for managing libraries shared between programs are really poor, and 
they are encouraging program-specific storage. I'm really interested 
in how you will experiment with integrating R3 with a properly managed 
OS, which (I hope) Syllable is.

loader mechanisms on linux are anything but poor (at least compared 
to windows)

Damned by faint praise, but true. The real trick is to make sure 
that LOAD-EXTENSION doesn't try to load non-extension system libraries.

why? it will fail anyway

no need to needlessly overcomplicate things

Even though Syllable has its own dynamic loader in the kernel, we're 
not managing user space libraries vastly differently than Linux, 
which has a user space loader. The difference is in kernel drivers, 
which are standard ELF libraries just like user space libraries, 
which isn't possible in Linux. I consider this similar to the nice 
circumstance that R3 extensions are native OS libraries, which has 
similar integration benefits - if you take advantage of them

The idea of a "safe" extension is tenuous at best as it is. Don't 
add the possibility of having some system-specific libraries polluting 
the file name space. For instance, I would consider one of the minimum 
requirements of a system-installed "safe" extension to be that it 
be guaranteed delayable.

Nobody but you is talking about "safe" extensions.

We are talking about making those pesky & horrendously unsafe native 
extensions work slightly more smoothly across platforms.

If you want safe binary extensions, rewrite REBOL on Genode with 
a GPL license. Anything less won't do

One interesting thing you haven't considered: sys/load-module calls 
lib/load-extension without checking whether it is native. You can 
override lib/load-extension with a REBOL code wrapper, in %rebol.r 
even. Then you can do any safe lookups you like without messing with 
host code, and even translate .rx suffixes to .so if you like.

Interesting indeed. But one of the main points is that condemning 
this functionality into %rebol.r is not sufficient.

This is a great way to experiment with platform integration, and 
what you learn can be adapted to the host code.

Also, I don't think that triggering platform-specific suffix replacement 
by using %filename.rx is a particularly good idea (too much magic). 
But it is one option.

I don't think using a platform-specific suffix for a cross-platform 
extension and then expecting code that uses it to be cross-platform 
is a good idea. YMMV.

Yeah, as should be obivious Kaj and me strongly disagree with this 
sentiment.

The extension file itself is not cross-platform (in most cases, at 
least), therefore using a standard suffix is extremely appropriate.

And "code that uses it" can trivially be cross-platform. We only 
need to solve the filename issue.

Yup. But that's why extension lookup is implemented in a platform-specific 
way, internally, to gloss over those differences.

Fortunately there aren't many platforms that only use platform-specific 
filenames for dynamic libraries. Those can be adjusted for. At least 
the .rx naming convention is better than something like -rxt.so or 
something.

Brian, it really is not about "platforms that only use platform-specific 
filenames".

Really? Because't on Windows and OSX this is not an issue, this is 
expected behavior.

Linux's loader in general couldn't care less about the extension 
of a shared object. That doesn't make it any less prudent to stick 
with the standard .so extension.

prudent to stick with the standard .so extension

 - I'm having a little trouble understanding this, but maybe that 
 comes from too much experience with platforms where it has been demonstrated 
 that there are advantages to distinguishing between different dynamic 
 library types by using different file suffixes. The "standard .so 
 extension" means that these differences get put somewhere else in 
 the filename. But I can see your point with libraries like System.Data.Sqlite 
 that support more than one calling convention in the same file, so 
 that the R3 extension API would just be added to a system dynamic 
 library that is otherwise meant to be called by non-REBOL code - 
 incorporating its own wrapper extension.

Remember, on many Linuxes you just want to load libc, but the actual 
file is something like libc.so.6 or something (I'm paraphrasing here). 
Having LOAD-EXTENSION translate .rx is no different from that. You 
can even have a filename map on your platform if you like.

It is different, as in the Linux case you add the standard extension, 
while in the REBOL case you want to change the extension. The latter 
is less expected, so more confusing

There is no further translation happening on Linux from libc.so to 
libc.so.6 as that is just a symlink in the file system

Based on our discussions I started writing an improved extension 
loading mechanism:
https://github.com/earl/rebol3/blob/master/modules/extload.r3

It's main purpose is to translate a "generic" extension suffix (%.rx 
per default) to platform-specific extension suffixes (yes, potentially 
many).

It hooks into the extension loading process (LOAD-EXTENSION) as suggested 
by Brian, so it works transparently.