Compression

floppy disk. Or take a Gigabyte worth of data and compress it to under 400bytes. I purchased /View/Pro and will most likely purchase runtime licenses once they are available for /View and begin distribution of NEW compression software depending on the licensing terms available. Imagine with 400 byes of compressed data (representing a gig or more) what this could mean to handheld devices like portable mp3 players or digital cameras. The capablity to store your entire mp3 collection in a portable device. I know your interested but you will have to wait a bit longer. Thanks to the guys in the IRC REBOL channel on EFNET for your help and cooperation (you know who you are).

-- Ryan Cole Programmer Analyst www.iesco-dms.com 707-468-5400 I am enough of an artist to draw freely upon my imagination. Imagination is more important than knowledge. Knowledge is limited. Imagination encircles the world. -Einstein

you, as you may have guessed. If I did not know you, I would be be laughing at you. Bypassing argument of whether or not you are correct, how far have you gotten? What sort of time does it to compress something? Generally speaking, how does it work?

floppy disk. Or take a Gigabyte worth of data and compress it to under 400bytes. I purchased /View/Pro and will most likely purchase runtime licenses once they are available for /View and begin distribution of NEW compression software depending on the licensing terms available. Imagine with 400 byes of compressed data (representing a gig or more) what this could mean to handheld devices like portable mp3 players or digital cameras. The capablity to store your entire mp3 collection in a portable device. I know your interested but you will have to wait a bit longer. Thanks to the guys in the IRC REBOL channel on EFNET for your help and cooperation (you know who you are).

-- Ryan Cole Programmer Analyst www.iesco-dms.com 707-468-5400 I am enough of an artist to draw freely upon my imagination. Imagination is more important than knowledge. Knowledge is limited. Imagination encircles the world. -Einstein

Exactly my point. So far it seems to be an awful lot. I would say the real problem is that 3 bytes can only contain about 1.7 million different combinations, so at maximum, only that number of documents could be compressed using a truly optimum technique. Such a technique would leave a vast number of documents in the world, and times to come, uncompressable or requiring more than 3 bytes. Several, several years ago I had come up with an compression scheme that could be redudantly compressed over and over, so that you could simply just keep on compressing until the document reached a miniscule size. After writing the program I discovered the only problem was that almost everything I compressed ended up larger than it was before I "compressed" it. However it did make an interesting encryption program. :^) --Ryan

It doesn't matter. Although the sequence "3/7" is a valid encoding for the infinite message 0.42857142857142857142857142857142857142857142857... (and therefore highly efficient ;-) I challenge you to find an equally compact encoding for the highly-similar message 0.42857142857142857142857142857142857142857412857... (yes, they are different, if you look closely enough). If both of these messages are in the set of possible messages I need to be able to encode, then the average cost of an "a/b" encoding begins to cost more as the set of possible messages grows. What makes the whole system cost even more is that you also have to take the size of the (de)compression algorithm itself into account. Consider that the absolute best possible compression technique (averaged, again, over the entire set of messages capable of being handled) would be to use a dictionary containing every possible message. If all messages were equally likely, the best possible compression would be to represent each message by its position in the dictionary (in binary, of course). Finally, I don't claim comprehensive knowledge, but everything I've read about "fractal compression" makes it sound like a lossy compression scheme. -jn-

Not exactly equally compact, but how about this: 23 / 7 S 6 23 divided by 7 skipping the first 6 digits.

I know it does not work, but I could see the expression on your face from here. lol, lol, lol, lol. I am suprised you didnt here my laughs. Wew! I will be getting a good chuckle for awhile, thanks! Of course if I knew how to do such things Joel, you would have heared about it on the 6 o'clock news. ;^) Cheers, --Ryan Ryan Cole Programmer Analyst www.iesco-dms.com 707-468-5400 I am enough of an artist to draw freely upon my imagination. Imagination is more important than knowledge. Knowledge is limited. Imagination encircles the world. -Einstein

...

Well, sorry to disappoint on all three counts: 1) The correct answer is 30000000000000000000000000000000000000000189 -------------------------------------------- 70000000000000000000000000000000000000000000 2) Your visual imagination is better than my auditory perception. ;-) 3) I won't be on PBS, as the producers of Nova were not sufficiently impressed with the production of the above answer. :-( The real point of this little exercise, however, may have gotten lost in the humor. Consider the following two messages: Dear sir, we find that we are so overwhelmed with your ability to perform arithmetic with numbers of more than two digits that we would be delighted to accept your gracious invitation, and will plan to visit your residence for an interview at your earliest convenience. To be brief, yes. and Dear sir, madam, or whatever the case may be: Our attorneys will be contacting you to serve a court order requiring that you cease and desist any further attempts to harrass our receptionist or other employees with your constant requests that we devote air time to the fact that you can do arithmetic. To be blunt, no. If those are the only two messages possible, I can compress each of them to a single bit! The same would be true if each message were composed in triplicate, with the second copy in Anglo-Saxon and the third in Mandarin (which, for the sake of convenience, we'll assume to be represented phonetically in ASCII). The number of bits has nothing to do with the "length" of the message, but with the total number of messages that are possible, and with their relative probabilities. Therefore, if someone claims to be able to condense the content of 3600 CDs onto a single floppy disk, I'd respond that it's entirely possible IF THERE ARE ONLY A RELATIVELY SMALL NUMBER OF POSSIBLE CDs. For example, if there are only 4096 possible CDs, one can code each one by a 12-bit value, regardless of how much data may be on each CD. Sending a 3600 CD message would then require only 675 bytes. (Of course, decompressing that 675 bytes would require that the recipient already have access to the content of a full set of such CDs!) OTOH, if a CD can contain 600Mb of arbitrary data, then a collection of 3600 CDs equates to approximately 2.16 terabytes. Any claim to be able to compress THAT collection of data down to 1.44Mb, if any CD is equally likely to appear, simply violates the laws of physics. It ain't gonna happen. I mean no discouragement to Paul in his research into data compression techniques. It's always possible to find special-case algorithms for special-case data, and some of them have interesting applications, such as JPEG compression of images and MP3 compression of music for human consumption. However, improved performance via a special-case approach always has the cost of narrowing the range of its applicability and/or having the effect you mentioned in an earlier post -- actually making data outside its "target zone" grow significantly. What's interesting is that there's actually a kind of conservation law here; the better it gets inside the target zone, the worse it gets outside. You can't win, you can't break even, and you can't get out of the game. -jn-

You've aroused my curiosity... how NEAR to perpetual motion are we talking? Like almost perpetual? Real close to perpetual? Just short of perpetual? I mean, how much closer, would you estimate, to perpetual motion can you attain with the device you mention than, say, your typical NON-perpetual motion device (like a YO-YO, for instance)? Like MUCH MUCH closer to perpetual? And when you say "perpetual motion generator" I have to ask, what does it generate perpetual motion for? How does the perpetual motion come out of the generator?

Your use of quotes around the word "today" confuses me. Usually quoting a word indicates something with questionable definition, like a jargon term, or something that is qualified in its usage. For example, I might say: This near perpetual motion generator should work "forever".

Certainly. By definition the impossible happens all the time! If it weren't impossible then it wouldn't not not be possible! -jeff

** Syntax Error: Missing [ at end-of-block ** Near: (line 2) ��]@����$��ָuv��y�'EQ$�\f�b�#�� w�y���W�,GT print %data.r decompress read/binary %/a/data.r works fine. What is causing the error message? Louis

** Syntax Error: Missing [ at end-of-block ** Near: (line 2) Y�]@����$�'ָuv��yz'EQ$�\f�b�#�� w�yz��W�,GT print %data.r decompress read/binary %/a/data.r works fine. What is causing the error message? Louis

** Syntax Error: Missing [ at end-of-block ** Near: (line 2) ��]@����$��ָuv��y�'EQ$�\f�b�#�� w�y���W�,GT print %data.r decompress read/binary %/a/data.r works fine. What is causing the error message? Louis

command

Ok this is the real problem - you use the same name for the uncompressed version as you do for the compressed version. Logically it works but in practise it is not a good idea. This is risky. You are increasing your risks of losing data. Why? Because devices (computers) sometimes fail. It might be a floppy (quite prone to failure) or your hard disk or a power shortage - whatever. When you are transforming your data so significantly as in the compression/decompression steps you are taking, you should write the transformed data to a new file. While on risks, the Rebol User guide warns about storing data using compression - if a couple of bits are changed by some stray cosmic radiation or whatever - you're more likely to be unable to recover them.

There's always a trade-off. With compression you are exchanging processing power for disk space. You need to choose what is more important to you taking into consideration the amount of information you are dealing with, the patterns of your use, the equipment you will use it on, etc...

Fair enough. So just do the compression using another script when you want to perform a backup. If you are using a different name there will be no confusion. Floppies are small on price and offer similar protection for backing up data. Multiple backups of the same data would be a good idea. Depends on how critical your data is. Oh and don't forget to check that you can read a file from a floppy once you have written to it - always good to test your backups actually are backups and not placebos.

The solution is simply to leave data.r uncompressed and use it as normal. Then as mentioned above compress the information it contains when you need to and save to a different name and possibly a different location. Brett

Your transcript shows me that %data.r -(compress)-> %/a/data.r -(decomp)-> %data.r after which you try to do %db.r which isn't any of the above files. What's in db.r??? -jn- -- ------------------------------------------------------------ Programming languages: compact, powerful, simple ... Pick any two! joel'dot'neely'at'fedex'dot'com

That would be true only if he had actually done change-dir %/a/ prior to the line above. If REBOL still thought the current directory were anywhere else, then the line above uses the same name but in a different directory. Thus it wouldn't be the real problem. Please see my response to Louis for what I believe to be the real source of his difficulties. -jn- ------------------------------------------------------------ Programming languages: compact, powerful, simple ... Pick any two! joel'dot'neely'at'fedex'dot'com

I can do all of

;... normal output occurs

without incident, after which diff shows compd.r and foo.r to be identical. Given a data file, as below $ cat whatever.data 1234 "Ferd Burfel" 127.0.0.1 #901-555-1212 2345 "Joe Doaks" 127.255.255.255 #615.555.1212 3456 "Patrick Henry" 255.255.255.0 #206.555.1212 $ I can do all of the following without problems

== [1234 "Ferd Burfel" 127.0.0.1 #901-555-1212 2345 "Joe Doaks" 127.255.255.255 #615.555.1212 3456 "Patrick Henry" 255.25...

1234 Ferd Burfel 127.0.0.1 901-555-1212 2345 Joe Doaks 127.255.255.255 615.555.1212 3456 Patrick Henry 255.255.255.0 206.555.1212 Therefore, your core strategy could be 1) decompress the data after reading from file 2) modify the memory data as needed 3) compress the data and write back to file Of course, this provides lots of failure modes: a) an error could occur during (2) b) the system could hang/crash during (2) c) a problem during (3) could smash the data beyond repair ...etc... Problems (a) and (b) could occur even if you were using an uncompressed file. The consequence is loss of modifications since last write. Problem (c) could also occur, but you'd have more of a chance of recovering some data by hand if the file were uncompressed. The worst-case consequence is loss of all data. Standard safety techniques include i) backing up between sessions ii) saving the data (whether compressed or not) between modifications within a session iii) writing a "journal file" entry for each modification, within a session, combined with doing (I) Option (i) is coarse-grained safety, with lowest cost. Option (ii) is most costly in time, and even more so if compressing with each save-to-file operation. Option (iii) is often a reasonable compromise, but does require that you have a recovery script that is able to take a stable image file and replay the journal entries (add/change/delete) to bring back to the last checkpointed status. YMMV, but I'd say that reinventing all of the functionality and security of a real database engine from scratch is a very non-trivial task. Good luck! -jn- ------------------------------------------------------------ Programming languages: compact, powerful, simple ... Pick any two! joel'dot'neely'at'fedex'dot'com

add some little bytes at the start which tells if file is compressed. something like: read-db: does[ data: read/binary %/a/data.r parse/all data [copy type to ":" copy data to end] if "compressed:" = type [data: decompress data] ] write-db: does[ write/binary %data.r join "uncompressed:" data ] write-db-compressed: does[ write/binary %data.r join "compressed:" compress data ] write-db-compressed read-db

Well you've got two on the list currently that I can see Joel. (a) "Try using the same file you just decompressed!" or (b) "...reinventing all of the functionality and security of a real database engine from scratch is a very non-trivial task. Good luck!" (a) is the concrete level. (b) is the overview level. My response to Louis was positioned somewhere between these two. I hope that helps you understand my message now. Brett.

That's the one I meant. -jn- ------------------------------------------------------------ Programming languages: compact, powerful, simple ... Pick any two! joel'dot'neely'at'fedex'dot'com