NTFS Stream Fun
[1/23] from: robert::muench::robertmuench::de at: 5-Aug-2002 17:40
Hi, I don't know if someone has posted something about this already.
Anyway, ever heard of NTFS streams? This is in fact a way to store
multiple data-tacks in one file. And MS has a very simple usage pattern
(at least that's what they sometimes get right) that even works with
Rebol!! Have a look:
>> write %test.txt "This is a test."
>> print read %test.txt
This is a test.
>> write %"test.txt:hidden" "This is an other test stored in a stream."
>> print read %test.txt
This is a test.
>> print read %"test.txt:hidden"
This is an other test stored in a stream.
Cool, eh? Have a look at the file "test.txt", you will only see the 15
bytes from the first write. The stream named "hidden" can't be detected
with just normal tools. Of course this only works on NTFS filesystem
drives. But I think it's a real cool thing. Imagine what you could do
with this: You can add your library files into a "library" stream and
have them loaded at runtime. No need to pack scripts etc. Robert
[2/23] from: ptretter:charter at: 5-Aug-2002 17:09
Very cool Robert. Good info to know.
Paul Tretter
[3/23] from: atruter:hih:au at: 6-Aug-2002 10:42
Wow! This could be a good way to organise data files. Instead of having
pcodes-vic.dat, pcodes-nsw.dat, etc have pcodes.dat:vic, pcodes.dat:nsw.
Makes me wonder if this could be useful for DB type apps . . . db-file:key?
Think I'll see if there are any documented limits / performance issues with
this "feature" (sounds similar to resource forks under Mac).
Thanx for sharing this find . . .
Regards,
Ashley
[4/23] from: bga:bug-br at: 5-Aug-2002 22:09
> Wow! This could be a good way to organise data files. Instead of
> having
<<quoted lines omitted: 5>>
> issues with
> this "feature" (sounds similar to resource forks under Mac).
It is not directly related to Rebol, but I would like to point out that
BFS, the native filesystem under BeOS does support metadata
(attributes) and the number of them per file is unlimited. Also the
size of each of them is unlimited. Attributes of integral types (int,
string) can be indexed automatically and transparently by the FS and
you can them search files in a way you would do in a database ("show me
all mpeg audio files that have Rock as the Genre, that has been
composed after 1990 and that I gave a Rating of 10 to it"), all that
without no need for any specific app (using just the system Find tool).
Not to mention queries are live and the query result is updated on the
fly if you add/remove files that match the criteria.
In other words, it completelly smokes NTFS out. :)
-Bruno
--
Fortune Cookie Says:
Have you noticed that all you need to grow healthy, vigorous grass is a
crack in your sidewalk?
[5/23] from: gschwarz:netconnect:au at: 6-Aug-2002 11:27
Ashley,
I noticed your e-mail address. Is hih address going to be around for a bit
longer?
Regards,
Greg
[6/23] from: atruter:hih:au at: 6-Aug-2002 12:09
> I noticed your e-mail address. Is hih address going to be around for a
bit longer?
<off topic>
I finish up end of October, so expect a different address from me shortly
after that. Last folks to use "hih.com.au" will probably be off the old
mail servers by March. Company will then take another decade to liquidate
totally (long tail claims, court actions, creditor payments, etc).
Most folks don't realise that a financial institution (bank, insurer, etc)
rarely just shuts up shop - there are many many financial, accounting and
legal issues of mid to long term duration that need to be resolved.
</off topic>
Regards,
Ashley Truter (contracted to HIH in liquidation)
[7/23] from: robert:muench:robertmuench at: 6-Aug-2002 11:05
> -----Original Message-----
> From: [rebol-bounce--rebol--com] [mailto:[rebol-bounce--rebol--com]]
<<quoted lines omitted: 4>>
> It is not directly related to Rebol, but I would like to
> point out that BFS, the native filesystem under BeOS does support
metadata
> (attributes) and the number of them per file is unlimited.
Hi, yes the BFS is a nice thing. I liked it a lot. NTFS supports this
too, even it's mostly unknown. Ever looked at the "Summary" tab of a
file properties dialog? All the fields you see there is meta-data stored
with the file. This data is stored in an internal known steam.
However, IIRC you can have a lot of streams in a NTFS file. BTW: The
stream feature is AFAIK available since NT 3.1 (1994). What you see in a
directory listing is the so called "default" stream. This stream has the
form of <filename>:$DATA
Further NTFS streams are not normally visible - unless you know one is
there (and unless you know its name), you are out of luck. If you copy a
multi-stream file to a FAT volume, all the streams will disappear except
the unnamed stream. In addition, applications that transfer files and
are not stream-aware only transfer the main unnamed stream, for example
email clients.
This makes it useful for things like license keys etc. Of course you can
have a look at the MFT of a NTFS partition but all these steps need a
lot of knowledge about NTFS. And Microsoft uses streams mostly for
Macintosh support. Whereas the Windows 2000 Content Indexing Server adds
an alternate data stream named "?Q30lsldxJoudresxAssqpcawXc" to image
files on NTFS volumes. This stream contains thumbnails of the images.
An even less common application for NTFS streams is the creation of hard
links (Posix-style file links). These get done by writing, to a special
stream ID (WIN32_STREAM_ID/BACKUP_LINK), the full path of the new link
to create. And streams can not only attach themselves to files, they can
also attach themselves to directories. This is a not so nice feature as
streams can't actually be deleted. The parent they're attached to must
be deleted in order for the stream to be removed. If you attach a stream
to the root directory of a drive, such as "C::MyStream", you can't
delete this one at all. So be careful.
Further streams, as they are essentially still files, can be executed
and do not have their filenames display correctly in Windows NT/2K Task
Manager, the utility commonly used to view running processes. For
example, if the stream "c:\test.txt:mystream" was running, Task Manager
would only show "test.txt". And while we are at it, try to find a
web-site running some older IIS servers and type in
http://www.victim.com/target.asp::$DATA You will see the source-code of
the ASP file. The server won't execute the code.
However, operating systems can access streams on an NTFS share,
independent of their file system. For example, a Windows 95 operating
system with a stream aware client might view a stream on a Windows NT
4.0 NTFS network share.
> In other words, it completelly smokes NTFS out. :)
NTFS is not that bad. I like some features of BFS as well but IMO the
NTFS is really a nice thing. I don't remember the title of a book (if
you know it let me know it too) that tells the story about the NT
development. Very good read! Not very technical, instead it tells about
the people and up & downs while developing NT. There is a section about
the development efforts that went into NTFS etc. Robert
[8/23] from: bga:bug-br at: 6-Aug-2002 13:38
On Wed, 7 Aug 2002, Anton wrote:
> This is the first of streams that I have heard.
> I am sceptical about its long-term value, except to
<<quoted lines omitted: 12>>
> translate between the two namespaces - a waste of time,
> in my view.
If you move a file, the metadata follows it. if you detete a file, the
metadata is automatically deleted. Metadata (or extended attributes) are
used to store information that is relatd to the file but is not part ofthe
file itself. For instance you have a plain text file and you store the
Author's name in a extended-attribute so you can have this information
available without having to edit the contents of the file. Image thumbnail
is another example. BeOS comes with a text editor (StyledEdit) that lets
you write text files using simple text formating (changing fonts, colors
and font size of parts of the text) but after you save it you can open the
file in an standard text editor and although you will not see the special
formatting you did, there wll be no garbage in the file, only the text
itself. All formating information is saved as an attribute.
-Bruno
[9/23] from: anton:lexicon at: 7-Aug-2002 1:59
This is the first of streams that I have heard.
I am sceptical about its long-term value, except to
support applications that use it already (like
Mac forks).
Isn't it just creating a new namespace - therefore
creating more work in creating/accessing/deleting
the streams?
So, you can hide some streams in one file, keeping
them nicely together. But what is the advantage over
a set of files in a directory?
We already know how to create, access, move, delete,
compress, decompress etc. files in a filesystem, why
create another namespace?
Then we just have to write a set of functions that
translate between the two namespaces - a waste of time,
in my view.
But perhaps there is a high-performance situation
where this NTFS feature is really useful. If someone can
point it out, then I might change my mind. :)
Anton.
[10/23] from: kemp:extelligence at: 6-Aug-2002 13:15
Streams are part of Windows/NTFS Structured Storage, and are used to support
a number of specialized file formats. For example, Kodak's Flashpix image
format uses structured storage streams to hold thumbnails, full images, and
various mangifications in-between, with each level of resultion being tiled
into it's own stream. On top of that, it stores standard and custom metadata
in other streams. All of this is related and inseparable, and streams allow
it to go nicely into one file without having to parse through stuff you
don't always need or shouldn't see.
- Kemp
[11/23] from: atruter:hih:au at: 7-Aug-2002 10:31
> what is the advantage over a set of files in a directory?
Data integrity.
Regards,
Ashley
[12/23] from: anton:lexicon at: 7-Aug-2002 14:30
So you're saying that the fact that the streams
are more difficult to modify & and access is what
makes them useful.
A bit like information-hiding and public interfaces
from object orientation.
Seems like a useful concept - I can buy that.
Although, what happens when you want to get rid
of some "meta-data"? Or copy some meta-data to
another file?
It seems much like an archive, containing a bunch
of files, one of which is a "default" file that
has the data you probably want.
If archives were transparently incorporated into
the filesystem, then I can imagine accessing their
contents directly using path notation:
example.zip/readme.txt
but referring to:
example.zip
would be equivalent to:
example.zip/default.binary
File streaming was obviously designed to solve some
specific problems, so I am not criticising it
as such, just the excited reaction as if it is
some wonderful new way of expressing relationships
between clumps of data.
File streams just add some special leaves to the
branches of the hierarchical filesystem tree.
Anton.
[13/23] from: atruter:hih:au at: 7-Aug-2002 15:44
> File streaming was obviously designed to solve some
> specific problems, so I am not criticising it
> as such, just the excited reaction as if it is
> some wonderful new way of expressing relationships
> between clumps of data.
I am probably one of those who have an "excited reaction" and for me this
is a "wonderful new way of expressing relationships between clumps of
data"; not necessarily *conceptually new* but new in terms of what I now
know NTFS is capable of, and the fact that REBOL transparently handles
this. No I haven't used this yet and I don't know what it performs like or
what its limitations are, but reducing a set of files like the following:
1.jpg
1.dat
1.png
1.txt
to:
1.jpg
1.jpg:effect
1.jpg:thumbnail
1.txt:description
means all the data related to 1.jpg could now be held in one file. This
means moving, copying, deleting files could be done on a single file basis.
If I later needed to add a PDF file to this list then I would envision that
adding "1.jpg:pdf" would cause less code changes than "1.pdf" as move,
copy, delete operations already act upon the parent. I would also surmise
that code required to ensure data integrity (eg. if 1.jpg is renamed to
2.jpg then all associated files must be renamed to 2.???) becomes simpler.
These possibilities, and the ability to have db-file.csv:key, allow me to
look at simpler ways of arranging and managing my data. If it works [well],
great; if not, at least it was a learning experience. If others shrug at
all this and think "big deal", or, "been there, done that" then that's fine
by me . . . the beauty of REBOL is that different aspects of it [and indeed
the problem domain] will appeal to different folks at different times. ;)
Enjoy!
Regards,
Ashley
[14/23] from: atruter:hih:au at: 7-Aug-2002 17:26
Sunanda,
> But it also makes for an application that can't move from the platforms
that
> support streams.
Yeah, I should have said, "... in terms of what I now know NTFS is capable
of, and the fact that REBOL transparently handles this [on WindowsNT, 2000
and XP]".
Encapsulating
stream usage like you suggest, and taking advantage of it
on platforms that support it, sounds like a good approach to me IMHO. Even
in a Windows only shop you have to deal with the fact that while servers
probably run with NTFS there will be at least *some* Win95/98/ME clients
out there, or devices that are FAT16/FAT32 based.
Regards,
Ashley
[15/23] from: robert:muench:robertmuench at: 7-Aug-2002 8:56
> -----Original Message-----
> From: [rebol-bounce--rebol--com] [mailto:[rebol-bounce--rebol--com]]
<<quoted lines omitted: 8>>
> the filesystem, then I can imagine accessing their
> contents directly using path notation:
IMO archives solve an other problem: storage space. While streams solve
storage access. An archive always has to be handled as one piece. You
can't just add a file without "re-compressing" the whole data-stream if
you want to keep good compression levels. Streams let you look at one
file from the outer world and on to many data-tracks from your
application.
It's more like a video-tape. You see pictures and hear sound at the same
time and you don't care about it. You move the whole tape. Did you ever
just listen to the sound track or only viewed the movie track? I didn't.
> File streaming was obviously designed to solve some
> specific problems, so I am not criticising it
> as such, just the excited reaction as if it is
> some wonderful new way of expressing relationships
> between clumps of data.
What I find exciting is that we can use this feature from Rebol and that
it's very simple to use. So what's the deal? In return you get some
decreasing in "data clump" handling with different files. It just makes
live easier.
> File streams just add some special leaves to the
> branches of the hierarchical filesystem tree.
Yes, that's true but it's a level only the application controls. Up to
file level the user can do what he wants, on stream level the
application is doing what it wants. As said by others, it can help to
keep data integrety consistent. Robert
[16/23] from: gschwarz:netconnect:au at: 7-Aug-2002 23:30
Robert,
The Book is called, "Show-Stopper!" by G.Pascal Zachary 1994
Sub title "The breakneck race to create Windows NT and the next generation
at Microsoft"
It is a good read.
Regards,
Greg
----- Original Message -----
From: "Robert M. Muench" <[robert--muench--robertmuench--de]>
To: <[rebol-list--rebol--com]>
Sent: Tuesday, August 06, 2002 7:05 PM
Subject: [REBOL] Re: NTFS Stream Fun
I don't remember the title of a book (if you know it let me know it too)
that tells the story about the NT
[17/23] from: robert:muench:robertmuench at: 7-Aug-2002 17:25
> -----Original Message-----
> From: [rebol-bounce--rebol--com] [mailto:[rebol-bounce--rebol--com]]
<<quoted lines omitted: 5>>
> Sub title "The breakneck race to create Windows NT and the
> next generation at Microsoft"
Hi Greg, thanks a lot :-)))))! I really couldn't remember the title,
just to few show-stoppers in my progams ;-). Robert
[18/23] from: joel:neely:fedex at: 7-Aug-2002 14:14
Hello, all,
I've been following this whole discussion with an acute sense of
irony that provokes the following comments. No offense intended
to anyone...
There was a discussion on the list a few months back about the
notion of the source code for REBOL eventually becoming available.
Someone asserted the view that this would be A Bad Thing since it
would tempt/allow people to begin creating incompatible versions
of REBOL, thus destroying the platform-independence that is such
a desirable feature of the language.
My response was that REBOL already offers the means to write
platform-specific scripts that cannot be ported. The thing that
keeps our scripts transparent/free/portable/virtuous is our own
willingness/discipline to avoid using ANYTHING that locks us into
a proprietary environment.
The best way to avoid becoming Nazgul is to decline Sauron's offer
of a ring, no matter how shiny (and powerful) it may be.
-jn-
[19/23] from: al:bri:xtra at: 8-Aug-2002 16:50
> The best way to avoid becoming Nazgul is to decline Sauron's offer of a
ring, no matter how shiny (and powerful) it may be.
Embrace, extend, nazgul-ize...
Andrew Martin
Why yes, my Rebol script works best with MSIE 6... :)
ICQ: 26227169 http://valley.150m.com/
[20/23] from: brian:hawley at: 9-Aug-2002 17:03
At 11:05 AM 8/6/02 +0200, Robert Muench wrote:
>However, IIRC you can have a lot of streams in a NTFS file. BTW: The
>stream feature is AFAIK available since NT 3.1 (1994). What you see in a
>directory listing is the so called "default" stream. This stream has the
>form of <filename>:$DATA
Isn't that supposed to be <filename>::$DATA ?
The three fields are: <filename>:<stream>:<attribute>
The default stream name is "" (without the quotes). The default
attribute is $DATA. All attributes start with $ .
Oh wait, I just remembered that since all attributes start with $,
you can skip the stream name if you are just using the default and
want to specify the attribute. So I guess that <filename>:$DATA
would work after all :)
> > In other words, <BFS> completelly smokes NTFS out. :)
>NTFS is not that bad. I like some features of BFS as well but IMO the
<<quoted lines omitted: 3>>
>the people and up & downs while developing NT. There is a section about
>the development efforts that went into NTFS etc. Robert
The database search capabilities of BFS are where it smokes NTFS.
But not for long: Future versions on Windows are slated to have
the filesystem built on top of the SQL Server engine, rather than
the other way around. When that happens, the smoke will definitely
be blowing in the other direction!
Brian
[21/23] from: bga:bug-br at: 9-Aug-2002 22:34
> > > In other words, <BFS> completelly smokes NTFS out. :)
> >
<<quoted lines omitted: 13>>
> the other way around. When that happens, the smoke will definitely
> be blowing in the other direction!
Actually, no. First of all, the first version of BFS was a real
database and it was dropped exactly because the performance hit on
using a real database as the filesystem is too big (sure system ara
faster today than they were in 1990, but this is no reason to make then
slower).
Second, BeOS has node monitoring capabilities so you always know rigth
on the spot if a file that you need (and you're monitoring it) has
changed and you can take appropriate action, This all without having to
pool for the file (you receive a notification when it changes).
Besides, BFS implements the live query mechanism where you are notified
when files stop satisfying your query or when new files start to
satisfy them. For instance I have a folder called "Best Musics" that
contains all my MP3 files I gave a rating of 10. This folder is *ALWAYS
* up to date even if you add and remove files.
I am not saying NTFS is not good for what it does, but I am saying BFS
is the best "user-oriented" fs *EVER*. Even SGI's XFS does not match it
feature-wise and XFS rocks.
If you have interest in OS development and FSs is general, take a look
at http://www.bug-br.org.br/openbfs. I am the team lead for the OpenBFS
project (that aims to create an open-source version of BFS) that is
part of the bigger OpenBeOS project (http://www.openbeos.org). Look at
the code and you will see why BFS rocks. :) BTW, OpenBFS is, right now,
*BETTER* than BFS in some aspects.
-Bruno
--
Fortune Cookie Says:
I'd love to go out with you, but I have to floss my cat.
[22/23] from: brian:hawley at: 9-Aug-2002 22:02
At 10:34 PM 8/9/02 -0500, Bruno Albuquerque wrote:
> > The database search capabilities of BFS are where it smokes NTFS.
> > But not for long: Future versions on Windows are slated to have
<<quoted lines omitted: 15>>
>contains all my MP3 files I gave a rating of 10. This folder is *ALWAYS
>* up to date even if you add and remove files.
Strangely enough, all that was exactly what I was referring to
when I agreed that BFS smoked NTFS. I just simplified so that
I wouldn't get too far off-topic.
Thanks for the links, though. I hadn't read _all_ of them ;)
As for the future, we'll have to wait for it. I can't remember
where I put that set of Yukon and Whistler links, so I can't
go into much detail, but there is much to look forward to.
We'll have to see what Microsoft can do with the best coders
that obscene amounts of money and evil can buy. Maybe they'll
surprise us and come up with something anywhere close to what
their research papers claim.
Brian Hawley
[23/23] from: greggirwin:mindspring at: 10-Aug-2002 12:11
<< Future versions on Windows are slated to have the filesystem built on top
of the SQL Server engine, rather than the other way around. >>
The more things change, the more you see how far ahead of their time
are/were people like Dick Pick.
--Gregg
Notes
- Quoted lines have been omitted from some messages.
View the message alone to see the lines that have been omitted