World: r4wp

Guys, was there any discussion about the COMPRESS/DECOMPRESS natives 
in R3? Do we really need these functions to use some 'custom' compression 
format that noone in the world uses and when it is based on zlib 
anyway? Wouldn't be better to change this in R3 and use for example 
ZIP format? What do you think?

I remember Carl saying the custom format is more efficient but I 
tried to compress couple binary or text files to compare with ZIPped 
results and I don't see any significant difference that would be 
worth the current format incompatibility.

Earlier discussion of COMPRESS vs Zlib.....Basically, the two are 
almost identical already, so it should not take much to make something 
that is compatible:
     http://curecode.org/rebol3/ticket.rsp?id=328

I agree that it should be easily compatible with other compression 
formats, but it does not take much to make it compatible (can be 
done via a mezz).

Sunanda, thanks for the link.

Henrik, I was more thinking to rip the old zlib code out of Rebol 
and put there some more actual(or even smaller) implementation (latest 
zlib or miniz etc.)

maybe, historically, also the licence was a problem? Remember, R3 
was not open-sourced back at that time ...

I am fully for new, more standard way. I e.g. did not like .RIP format. 
E.g. Winamp uses normal zip format for their skin modules, etc. I 
would like to just hit enter or double click, getting into the archive/package.

Cyphre: that sounds fine to me.

Pekr, I don't think .RIP was meant for basic temporary compression, 
but a more fully fledged file format to store serialized REBOL data 
efficiently?

Henrik - RIP was crap :-)

I would prefer my app being packaged in default zip anyday ...

For me it was just another way of reinventing the wheel, with almost 
no added value ...

Pekr, I don't think it's the same thing, plain and simple.

Would you not want a single, universal way to store and retrieve 
application state from disk? ZIP does not do that.

why not? Because you can just 'do the .rip? I would not mind do decompress 
%my-app.zip :-)

Well, then I am wrong, ok. Not sure .rip format was anything in common 
with app state serialisation. I thought it was just plain and simple 
packaging mechanism - having a header, and binary container for compressed 
app modules/scripts ...

Zip does not manage bindings or words transparently. It's simply 
not the same thing.

(and if it turns out, that it was not the goal with .rip, then the 
goal should be changed to do this)

Then we talk two different things though. But - don't push anything 
like .rip on me, just to package my app. I want more transparent 
way, a zip archive I can inspect in Total Commander, Windows Explorer, 
etc.

I would not want that a requirement for .rip anyway.

If you want something modern, add support for LZMA compression. And 
please don't mix compress with zip - zip is high level, compress 
should be low-level function (idealy supporting more formats)

Oldes - we don't mix anything. It was zip vs rip discussion. I say 
- let's .rip RIP :-)

regarding zip versus rip - I can imagine runable zip - just add special 
file in it which would be used as a starter - actually that's how 
works for example APK, which is zip with different extension.

I say - let's .rip RIP :-)

 - how can you say this, when we want a way to store application states?

store them in zip, simply some storage format inspectable from other 
tools. I still don't understand, what is so special on rip, it is 
imo just a storage format, nothing special.

Pekr, there is a lot more to storing application states than simply 
zipping data.

It's quite a long time I used rip last time so my memory may be faded 
out, what do you mean with the application states?

Henrik - maybe you meant a Rebin format? I really thoght, that .rip 
is just packaged scripts with header in order to being runnable? 
So I would like to ask about the app states too?

Storing bindings, managing upgrades of data structures, structural 
checks and error handlign, type assertion, etc.

Pekr, ah, you are probably right. I forgot there was another name 
there.

Well, rebin was not imo finished, even for R3, although maybe R3 
was closer to get to that idea. IIRC .rip appeared somewhere around 
IOS times, as a means of package scripts. It was imo a typical way 
of Carl, introduce his own way, instead of going with some "standard", 
e.g. zip.

OK, then sorry, I was wrong. As far as .rip is concerned, then it 
should be some kind of installer, for it to be justified.

I'm ALL for using a standard compression format.  Then with a little 
work, .rip files could basically be SFX files. That would enable 
a super simple packaging method for delivery of executable REBOL 
applications.

+1 for NickA

as far as i know, zip files allow prefix payload, so you can put 
stuff before the actual .zip file starts... just like REBOL allows 
stuff before the header.   I've seen a demo of a single file which 
is  an .exe,  .pdf ,  and .zip all at the same  time!

Old Bug, New Bug, Red Bug, Blue Bug?


I'm using the ARM build from rebolsource.net and getting the following 
unexpected behavior:

>> template: read %template.html


It outputs template as if it was a binary file, although it is plain 
text.

The build date on the package was 24-Jan-2013.

No bug, READ does no longer automatically decode binary to strings. 
Use READ/string to obtain a a Unicode string obtained by decoding 
the binary with UTF-8.

Cyphre, Rebol compression is the same as zip deflate, but it's not 
the compressed part that's more efficient, it's the headers. The 
zip format has some fairly extensive headers, and Rebol compression 
doesn't. We do need compression to support compressed scripts though, 
so we need something. That doesn't mean we can't have something more 
flexible, and if we stick to what we can get from zlib (since we're 
linking it anyway) we can get this flexibility nearly for free.

@Andreas: Good to know!  Thanks!

If the latest zlib is better, we should consider switching to it. 
If we need to make sure that R2-compatible compression is an option, 
make it an option. Decompression should be able to autodetect the 
compression model though.

R3's existing compressed scripts feature is better than RIP already, 
so we can safely let that fall into history.

ZIP support would be great. I've wanted it for a long time, but want 
more than just a "compress this value" function. i.e. it needs to 
work like a port or have an interface that lets us navigate, list, 
etc.

The ZIP format is supposed to be like a directory/file structure. 
I would rather have it supported as a zip:// port scheme, like file:// 
is for files and directories.

It's not a single-stream model like zlib or gzip - those would be 
better supported by COMPRESS and DECOMPRESS.

Agreed.

BrianH, yup I realized how it works now and I even fixed the /gzip 
quirks in DECOMPRESS...so I can decompress zip chunks with CRC32 
checksums now. But there is still one annoying bug, that the current 
zlib code doesn't handle the CRC32 calculation well on bigger files 
than 32kB :-/ Now I'm trying to fix it so the crc32() calls works 
in the "CRC running" mode where the final CRC32 is calcualted from 
smaller chunks of data which a file consists of.

I'll hopefully push this on github If I manage to fix it succesfully.

Do you think that it is feasible to change the current zlib to a 
more recent version or something else that is better? Keep in mind 
that I would want that zip:// scheme to exist eventually, even if 
I have to write it myself.

We would still need to be able to compress and decompress in a R2-compatible 
way, even if it's optional, not the default. For interoperability.

BrianH: I think we should change the /GZIP refinement to /CRC32 so 
it is independent of the format. Then we can create mezanine/scheme 
support code for various higher-level formats (ZIP, GZIP etc.) at 
the REBOL scripting level.

BTW I have finally fixed the current /GZIP refinement problems so 
now I'm able to ZIP/UNZIP uzing rebol script without the 'PNG load 
hack' that is on rebol.org. In fact I modified the rebzip.r to work 
with R3 and the ZIP/UNZIP operations are now "instant" when comparing 
to the R2 version. So I guess this is a good proof we can write fiast 
zip:// scheme with just the current COMPRESS/DECOMPRESS natives.

Regarding the zlib code change/update. I think that is feasible. 
We could try to use the latest version of ZLIB library (hope the 
licensing is ok, haven't checked that) or look at the MINIZ compression 
library (check it on: http://code.google.com/p/miniz/looks interesting 
as it could be more lightweight)