World: r3wp

foreach :rec data [print [name last-name address street]]

Use a get-word.

ah ;-)

I need to evaluate our phone provider CSV data for one year. And 
they mix cell phone and normal phone on the invoice, so I need to 
go down and download details. And as it is pread across per month 
files, I want to do simple counter in REBOL. With read/lines ... 
and subsequent foreach line data [ ... parse line "," .....] REBOL 
is really cool to do such data manipulations ...

If you make an object with images, blocks and a lot of data and then 
set the object to none, and the data inside is not referenced elsewhere, 
will everything else inside be properly deallocated?

Perhaps not - 

>> stats
== 67680368

>> a: make object! [

[    b: make block!  1000000

[    c: make block!  1000000

[    d: make block!  1000000

[    e: make block!  1000000

[    f: make block!  1000000

[    ]

>> stats

== 167685587
>> a: none

== none
>> stats

== 167686079

>> recycle

>> stats

== 167685527

interesting, thanks

Which is probably to be expected as the values referenced by the 
fields in the blocks have not been de-referenced. Looks as though 
you'll have to do it yourself:

>> stats
== 167686051

>> a: make object! [                       
[    b: make block!  1000000                 

[    c: make block!  1000000                 

[    d: make block!  1000000                 

[    e: make block!  1000000                 

[    f: make block!  1000000                 

[    ]
>> stats

== 267690882
>> a/b: none

== none
>> a/c: none

== none
>> a/d: none

== none
>> a/e: none

== none

>> a/f: none

== none

>> recycle

>> stats
== 167687087

I'm sure there's a better way but this should work:


>> foreach word remove first a [do join "a/" [word ": none"]]  ;; 
provided your object is called a

this is very interesting. I think there should be an easier way to 
do this.

in R3, 
>>set a none

in R2, that should work
>> set bind first a a none

hmm...
>>set a none
It works for both...

Peter, here's some R3 fun. To do the initial block allocation in 
R3:
    a: make object! 5
    foreach w [b c d e f] [extend a w make block! 1000000]
or maybe this way:
    a: object [b: c: d: e: f:]
    foreach w a [set w make block! 1000000]
To clear your way:
    foreach w a [set w none]
Of course Steeve's method works too, and better :)

henrik: the GC doesn't deallocate on demand... you can have several 
hundred megs of RAM allocated... and sundendly the RAM goes down. 
 and some things do not cause the rebol executables to shrink.


images, in my experience cause memory leaks, where they never get 
de-allocated from RAM under some circumstances.


in theory if you remove all references to a context, its content 
is supposed to deallocate, but when passing anything through view, 
all bets are off.

I've had 400MB pictures stay stuck in RAM, with absolutely no GC 
happening, not even going through view... just loading the image, 
without assigning it... its stuck  :-(

the easiest and FASTEST object clear:  

foreach itm next obj [set in obj itm none]

That next will trip you up in R3 .... but the set can be simplified 
(as per Brian's example above)
   foreach itm obj [set itm none]

that's a nice improvement in R3

oops... forgot the first in there hehehehe

foreach itm next FIRST obj [set in obj itm none]

such "manual garbage collections" should not be necessary in general, 
if there are no references left

well as pointed above, images definitely do not follow the "no references 
left" argument of the GC.

load %some-image.png


not even assigned once... causes a permanent memory use in the last 
version of which I had to use huge images.  I ended up using image 
magic which was able to use the equivalent of 20 GBs of image data 
in a single 75 minute composition.

Previous discussion on subject....Gabriele says memory is never released 
back to the opsys:

   http://www.rebol.org/aga-display-posts.r?offset=-1&post=r3wp152x2970

yes, but that is a different matter

that's quite a problem, but is it possible to allocate space and 
load new image data into that location?

Diffferent, but related......memory may be deallocated within REBOL's 
sphere of influence one the items have no more references; but it 
is not (it seems) dellocated back to the opsys until the REBOL thread 
closes.


That can create several problems ..... eg an application that uses 
a huge peak memory load at startup, and would like to hand it back 
for the next 8 hours of running. It's all in the arena of efficient 
memory management.

There is a difference between loading image using 'load function 
and loading using 'load-image function (which is used by VID). The 
second one stares data in cache.

And I see no problem with the first one:
>> recycle stats
== 4384079
>> x: load %/f/IMG_0783.jpg
== make image! [1024x768 #{
4E707A59767C5D7172565F5A4D4E464E4A3F554D4258504558494254453E
4F40394E3D364A373148332E4A35304F3A35483E34...
>> recycle stats
== 7531902
>> x: none
== none
>> recycle stats
== 4384079
>> load %/f/IMG_0783.jpg
== make image! [1024x768 #{
4E707A59767C5D7172565F5A4D4E464E4A3F554D4258504558494254453E
4F40394E3D364A373148332E4A35304F3A35483E34...
>> recycle stats
== 7531950
>> recycle stats
== 4384127

Maxim, you said that the problem occurs when the image has been used 
in View. What if the face object that was created for the image was 
not properly unset, like we've talked about above?

Maxim, do you know this:
>> view layout [image %/f/IMG_0783.jpg]
>> second second :load-image
== [%/f/IMG_0783.jpg make image! [1024x768 #{
4E707A59767C5D7172565F5A4D4E464E4A3F554D4258504558494254453E
4F40394E3D364A373148332E...

?? load-image

oldes... no I didn't know this.

but I never let view load images on its own, I always control that. 
 I didn't even know about load-image  ;-)

henrik, if a face isn't unset properly, for sure the image will stay 
stuck in ram... since one reference to it exists.

BUT


I've previously realised that using the stylesheet system actually 
runs init on each face when you build the stylesheet.  as the stylesheet 
is linked within the faces, each stylesheet will effectively constitute 
a memory reference on some resources.

oldes, about the load ... its not totally obvious why sometimes the 
images stay stuck in ram... but in this case it was huge png images. 
 and I did every recycle trick I new of... I tried solving this for 
2 hours... but nothing worked... not even allocating other stuff, 
and freeing it, to make sure the GC really did do a cleanup (in case 
is was count based).

now that I think about it... its not loading the images, it was using 
make image which caused the memory leak... if you call 400MB a "leak".

this is for a pretty insane project I have which will use up about 
30-40GB of image data (once loaded pictures take up much more space 
than on disk, obviously), per output picture.

Maybe it's the content of PNG images what couses problems with GC. 
What do you do with so many images?

I use ImageMagick as well. It allows much more than pure Rebol. But 
of course I call it from Rebol:)

hehe I did the same :-)  I have an automated picture catalogue builder...

its part of a huge visual arts project I have which requires thousands 
of photos... all in high quality... the end goal is a 1.75 x 6m wide 
printout. which use many overlaid independent images.

by many, I expect at least 10000.

Interesting:)

output res should be at 600dpi.  the trick is to find a print shop 
which has a large-print printer with enough ram in its module... 
most shops top out at 2-3m

but i need to build the software first, and right now.. that depends 
(again) on R3... I will be using OpenGL to build the output image 
setups, then dumping the info to a monstrous image magic process, 
which in the end might actually process more raw image data than 
my HD actually can contain.  I expect render times in number of days. 
 (4 pictures on a much smaller canvas took more than an hour).

the processing will be spread out on a farm and reintegrated, as 
tiles render-out.

the subjects of the system can change at will, this whole system 
is a process, not a specific one.  in the best of worlds, I'd find 
a philantropist that would allow me to build the full project (at 
a much larger scale), which costs quite a few thousand dollars in 
printing... one day...

but the technology, once built... can be reused for any project  
 at any scale.

what is the source of images?

the current subject is a church next to my house. I already have 
around 3000 pictures of it at various periods... but the real project 
is a 24hour shooting session, with a top-of the line rented digital 
cam.

I'd like to use hdri imagery.  but with image size that can reach 
in exces of 100mb per picture... its a bit prohibitive on the amount 
of pictures you can take before offloading.