World: r3wp

There are lots of desperate people out there.

Due to a very slow Internet connection, I need to make the FTP module 
of my website builder script more efficient so I don't send files 
unnecessarily. What I have in mind is:


1. Delete all the files in the website directory on my harddrive 
to eliminate all unused files.
2. Build the website to the website directory on my harddrive.

3. Download a list of the file names and creation dates from the 
website (all are in one directory).

4. Read the list of file names and creation dates from the directory 
on my harddrive (all are in the one directory mentioned in 2 above).

5. If a file is on the hard drive but not on the server, send it 
to the server.

6. If a file is on the server but not on the harddrive, delete the 
file on the server.

7. If a file on the harddrive is newer than a file on the server, 
send it to the server.


Has anyone already done this? Am I forgetting anything? Any pointers 
on how to do this?

Hi Louis,

Sorry about the CRLF ..... you don't want to be checking the timestamps 
on the server with a slow connection. Just hold the last updated 
value locally and if it changes then transfer the file.  Same for 
deleting ... else you spend all of your time checking on the server 
over a slow connection.    You could check the timestamps or hash 
the local value ... then if  the hash value of the source changes, 
transfer the updated version.     There's some code to do some of 
this in build-sie.r  http://www.rebol.org/cgi-bin/cgiwrap/rebol/view-script.r?script=build-site.r
 but it's a rebol-ish task.

Thanks, Mike. I'm studying the build-site.r code now.

if you are on an unreliable or slow connection, you might experience 
timeouts which will in turn cause network errors. I recently worked 
on a similar system and you have to basically wrap all code that 
access the internet in TRY and do a lot of error trapping and possibly 
some retrying to ensure that uploads and downloads of entire filesets 
are done correctly. The code in build-site.r will not do that, so 
you have to restart the upload if it fails.

http://www.compkarori.com/reb/ftp-dirupload.r

should do most of what you want ...

4 years old though.

No date checking on files .. only size checking.

Thanks, Henrik and Graham.

Graham, if a file upload fails somewhere in the middle, will you 
script restart the upload where the failure occurred so that the 
first half of the file does not have to be downloaded again?

No, it does not do resume.

it is completely automatic... no intervention required.

otherwise, if you want an interactive ftp program which asks you 
.. do you want to resume or whatever, you need another program.

this once force uploads everything.

I used it to upload hundreds of files that other ftp agents croaked 
on.

I found various FTP servers report dates differently. Also the dates 
may not include the timezone, so you would have to assume it is in 
the timezone of the server and get the timezone from the server another 
way. Because there are so many variants of FTP servers you would 
have to do a lot of research to make this reliable, and then you 
wouldn't be 100% sure it would not fall over with some obscure FTP 
server.

Louis -- a couple of pointers about uploading files to a server using 
a slow FTP connection:

(I do it myself with REBOL.org -- most of the development takes place 
on my machine and is uploaded to RO via a 56K modem, so this is based 
on real experience.)

-- If you are uploading a large live file, that file will be available 
and/or "broken" during the course of the upload. Best to upload with 
a temporary file name, and then rename when uploaded.

-- That won't work with CGI scripts under Apache/UNIX as the rename 
won't leave them with the right file permissions to execute. But 
it will work for all other files, including scripts that are DOne 
by your CGIs.

-- We have a checksums file that the uploader uses.  Before uploading 
a file, it checks the file's upload checksum. That way, we only ever 
upload new or changed files.

Thanks, Graham, for the script. And thanks Henrik, Anton, and Sunanda 
for the pointers. I'm hoping to be able to start working on this 
within the next few days.

No problem. Hope it goes well.

This is very good CSS tutorial, which someone may find useful - http://www.thenoodleincident.com/tutorials/box_lesson/boxes.html

And this is another good place from which I found the link above 
- http://www.mandarindesign.com/

Some good resources about  design/css/ia/etc. on last ALA: http://www.alistapart.com/articles/alaprimer2

My favorite CSS examples http://www.csszengarden.comNot so much 
about "how" to do it but "what" can be accomplished. I find it pretty 
amazing.

I had faced with problem for file uploads:

I had file upload script (upload.r posted somwhere in rebol lists/worlds) 
on my web servers, that works well on KF web server.

But on MS IIS script hangs and I am getting timeout error from server.
If it is needed I can post upload.r here!

Actualy problem is in function read-post data - script hangs on read-io.
Why it is problem for MS IIS and how to solve them? 

read-post-data: func [
    {Reads the HTTP entity body}
    /safe "Disables evaluation of content-length header."
    /local len data tmp
] [

    len: load any [ all [safe "65536"] system/options/cgi/content-length 
    "0" ]

    data: make string! len
    tmp: make string! len
    while [ 0 < read-io system/ports/input tmp len ] [
        insert tail data tmp
        clear tmp
    ]

    data
]

is there any solution available for Rebol, which would handle sessions?

My understanding is, that sessions=cookies (or hidden form field, 
url, or combination of those ones) plus storage/invocation mechanism

there is cookies manager from Oldes somewhere, have a look around 
AltME for URL (he still ignores rebol.org, such a bad bad bad boy 
;).

cookies manager? will try to look for one :-)

yes, it's patched HTTP scheme. Does all session management automaticaly 
IIRC.

hmm, it might not be ideal for CGI, to do 20KB script with each invocation 
...

I found it via google ...

well, working with cookies is not all that difficult, is it? My friend 
just asked me - why rebol does not handle sessions, if any other 
language does. I told him to write it himself, but he probably does 
not know how. Isn't session just about getting a cookie, looking 
into your storage space for the cookie identifier (session identifier), 
loading the session data, using them, and storing them once again?

Pekr: his Cookies Daemon is client-side, not server-side, I probably 
understood you bad.

ah, yes, I wanted server-side one ...

Yes, session handling is not hard. REBOL does not have it built in 
because it was not designed to be mainly a CGI language; so you need 
to add that yourself.

how should I design my function, if I would like to have e.g. session: 
copy [] block, and later would like to append whatever rebol value 
into it? e.g. variable names using in script, objects, etc? when 
I do append session var, it stores its value .... what would be the 
best aproach?

the simplest way, which however needs write permissions to the filesystem, 
is to have a unique session id assigned to users; this id could be 
basically a file name (and you need to check for its sanity then); 
then you read from the file at the beginning, and save to it at the 
end.

eg. you could have session: load session-file at the beginning; then 
your script does whatever with session; then you save session-file 
session at the end.

temple.cgi does basically this.

yes, I know, as for files. My strategy is very simple - use cookies 
(I wonder if there is script being able to handle multiple cookies 
btw), then "start a session" = generate unique ID, store it in \sessions\ 
dir ....

I just thought about how to store and later load some rebol values, 
add new values to them .....

temple.cgi uses cookies. there is no problem in handling multiple 
cookies on the server side, and the bug with multiple cookies on 
the client side has long been fixed.

maybe a pair of word name (literal) plus value ... and then some 
little accessor function

i just let the session variable be whatever the user wants. it could 
be an object!, or a block!, or whatever.

in some cases you may just need a logic! value, in others a block 
with words and values may be best...

multiple cookies? how are they separated in http header? there are 
various scripts around, not sure all of them handle multiple cookies 
... I will investigate ...

ok, so you know you want to save variables: name last-name user-object 
some-block-here .... how do you save them, and later invoke them?

you just need to parse system/options/cgi/other-headers iirc. see 
temple.cgi, it shouldn't have problems with that. (but if you have 
session handling you only need one cookie in the end).