World: r3wp

Regression tests!

... and:
>> ? datatype!

Maybe the amount of work will be too big compared with the number 
of bugs and misbehaviour, we might find. REBOL is rather stable, 
as it is. And: 1) we have RAMBO and 2) the power-users already know 
most of the misbehaviour (, if you can call it that).

An example of a peculiar thing:
>> negate 2 ** 31
== -2147483648.0
>> a: negate to-integer 2 ** 31    
** Math Error: Math or number overflow
** Where: to-integer
** Near: to integer! :value
>> a: negate to-integer 2 ** 31 - 1
== -2147483647
>> a: a - 1
== -2147483648
>> type? a
== integer!

Ah, I just found the explanation!
>> a: to-integer negate 2 ** 31
== -2147483648

It's correct behaviour. Silly me.

Personally, I think it would be worth it to make the tests. There 
have been several times that I have been tripped up by a former bug 
getting fixed, sometimes years prior, and a set of unit tests would 
have kept me informed. Still, I would prefer if any misbehavior found 
would be fixed, not just documented and left to rot.

I often thought it would be nice to see the history of a function. 
That is, which version of rebol it first appeared in, if there were 
any changes in any subsequent versions etc.

Plus, the unit tests would serve as documentation of a function's 
behavior, intended or not.

There used to be calls for "the function diffs" when new versions 
of rebol would come out. The script to produce the diffs compared 
argument spec blocks and function body code where available.

Anton, your function history sounds like a great idea, although I 
would add parse behaviors to that list.

Fixes to parse bugs (somewhere in the /View 1.2 betas) were what 
tripped me up :)

Yes, well that's where the unit tests will help you.

Unit tests will have to be rebol version dependant. Eg. A set of 
unit tests developed on Core 2.6 for the PRINT function may all pass 
on Core 2.6, but not on Core 2.5.  Recording the rebol version also 
captures the date and platform where the tests were developed.

Uhh yes, testing a vocabulary huge and multi-platform language as 
REBOL is a big task. But interesting aspects (like the history), 
as you point out.

I'm thinking more like keeping track of a few things:
- Proper behavior, and version when such behavior was achieved
- Changes in expected behavior

So there would be two sets of versions, the versions of REBOL and 
the versions of the tests. Over time, both REBOL will be fixed and 
the tests will be fixed, refined or altered. This could get pretty 
big pretty quickly I suppose - it could use a database to store the 
tests or some such.

Encyclopedia REBOLae
:-)

Don't tell me this is another thing that would have to wait for the 
REBOL/Services infrastructure...

I think we all have to wait for Rebol 3.0 :-)

It would be good to be able to answer a question like this: "Do the 
functions: [print parse encloak] exist and behave the same on Rebol/Core 
2.5 and 2.6, and Rebol/View 1.2.1 and 1.3.1?"

Brian, I think the results can be stored hierarchically, so a filesystem 
storage can be enough, although I'm not sure.

I'm wondering if everything can be automated so that there are no 
stored results (because results are in fact derived), so when the 
user asks a question, that's when the tests are run and the answer 
obtained.

(The results could be stored after this, mayb, so that only popular 
questions have web pages generated with the results in them.)

I've got a text file going with these notes in it, but I think it 
might be better to make a new project in Qtask. Anyone disagree ?

For the resulst of the use of a word, you may need to save the wanted 
output, and the actual output from different platforms (if they differ).

*results*

Absolutely, but if we have a rebol unit test server constantly running, 
able to answer arbitrary questions, why would we need to store results 
?

right, only the wanted result need to be tested, if you have access 
to all platforms to test. Maybe result from one reference platform 
should be saved, to test other platforms up against?

We could set up a server with as many versions of rebol as possible, 
so it can help as many people as possible, but each of us could also 
run the same system with a more limited set of rebol versions (ie. 
just the ones we bother keeping).

*only the wanted result need to be SAVED, if ...*

(I'm not too sharp today.) :-)

Geomol, every one has a different idea, as time progresses, as to 
what is the "reference" platform. Since it will be a collaborative 
effort everyone will be adding input from various sources. Some functions 
are only available on Rebol/Link etc..

Well for one thing, this would preclude platform-specific tests, 
as your server would only be able to be one platform (probably).

Anton, to make it perfect (total), you need acces to all the OSs, 
REBOL is supported on.

That's right, the server will get as many rebol versions from everyone 
as possible; Core, View, SDK whatever..

So the request to a certain server would be something like: run this 
little test and tell me the result (also if it lead to an error!). 
Could be done easily with REBOL.

More like: the request to the rebol unit test server would be: "Tell 
me all about PARSE on these Rebol versions..."

I see difficulty in testing something like a block!, because you 
would also have to test blocks in objects and look for possible side-effects, 
or what?

I'm not sure a hierarchy would work here - there are too many dimensions. 
Platform (Core, View, ...), platform (Windows, Linux, ...), version, 
test version, etc. Plus a test version would have applicable platforms, 
expiry both for bugs in the test and for changes in expectations, 
and cached results. I'm thinking of more of a formal test suite here 
than an arbitrary test server farm.

Test categories too, perhaps.

Blocks are probably too fundamental to rebol to worry about at an 
early stage, but unit tests could be made for any kind of behaviour 
or to test for any bug.

Brian, you're probably right.

Maybe RebDB could be used in this project? I've no experience with 
RebDB though, so I can't say, if it's suited.

I've done a relational database "NicomDB" as an education project 
2 years ago. It would be suited for this, and I've wanted to push 
it forward for some time. Maybe this is the opportunity? NicomDB 
is used on a webserver in a real application.

So it needs a database, and a web interface to be able to make queries 
to that.

The expiry and applicability info would help us distinguish between 
changes in intended behavior, buggy implementations, beta-vs-release 
and such. REBOL changes a lot even if many of those changes are fixes. 
This could act as a compatibility test suite for alternate implementations.

NicomDB could be used, possibly. We'd have to evaluate the options.

Geomol, what' s the application NicomDB is serving ?

I'll write you privately, as we don't want it to be exposed too much. 
(Users are depending on it, and this goups is web-public.)

Ok.  But this is becoming a very large and complex project. Are we 
really prepared to go through with it ?  I think even if we go a 
short way, the ideas and findings may help someone else come along 
to complete it later. So I think I will make that Qtask project.

Yes, do that as a start. We can move it later if needed.

I'll do some research and think about this some more. It may not 
be as difficult as it sounds.