AltME groups: search

Btw, I've tried to use file locking method to be able to sync writes 
to the same file from several processes. None of the file locking 
code found on rebol.org works reliably enough. At best, all processes 
can write their content to the file, but the order is messed up (each 
line starts with a timestamp), so not usable in my case. I didn't 
want to try using OS native file locking API, I'm afraid that a frozen 
process could permanently lock the file and block all the other processes, 
so it's too risky (at least using only REBOL, I could have some escape 
strategy to avoid that).

So, the solution I choose was to add a log service in Cheyenne that 
would serialize all log messages passed through TCP connections and 
write them to log file from the main process. It's kind of pulling 
out the big guns for a very simple problem, but that's the most reliable 
solution I could found. I would be interested to know if anyone has 
a better option.

Same idea here Doc. Apps can do individual logging, which can then 
be merged for analysis. But if you need everything to go to a single 
log file, you need a controller for them to go through. I'm also 
still very big on the DTrace model.

New Cheyenne 0.9.19 beta version available for testing at : http://cheyenne-server.org/tmp/cheyenne-r0919.zip


Tested only on Windows  (my Linux image network has currently some 
issues).

ChangeLog (diff-ed from last test version) :


 o RSP: an HTTP redirection in 'on-page-start won't evaluate the page 
 script anymore.
	

 o CGI: mezz function READ-CGI now patched to be compatible with Cheyenne. 
 That's
	  the right way of reading POST data in REBOL CGI scripts.
	  
	o RSP: fixed a bug in session/add when setting a block! value.
	

 o Task-handler: fixed a network error on first packet read (high 
 load + fast hardware).
	

 o Task-handler: TCP keepalive mode activated (test workaround half-closed 
 connections).
	
	o Task-master: 
			o 'no-delay mode removed and replaced by 'keep-alive mode

   o now forks a new process as soon as one dies (not waiting for a 
   new request)

   o fix a long standing bug in queued job module name mismatching (can 
   happen under extreme load)
			o minor code cleanup
	

 o Uniserve: 'no-delay TCP network mode now switched off for all connections. 
 Fixes a

   stability issue on Vista and probably on UNIX with very high load.


 o RSP: fix a bug in 'decode-multipart when there's no file received.
	

 o UniServe: new logger service. Now all info or error logs, and debug 
 messages from

   CGI/RSP scripts are written in %trace.log. Additionnaly, you can 
   now log messages
	  using :
	  
	  		- debug/print msg 	; msg [string!]
	  		- debug/probe value 	; any mold-able value
	  		- ?? word		; works like REBOL's '?? function
	  		- ? msg		; alias for debug/print
	  

 o RSP: in debug mode, page generation time and SQL queries stats 
 now added at bottom
	  of pages.
	  
	o RSP: error in events from %app-init.r now logged.
	

 o RSP: fix a rare RSP freezing issue when an error occurs in %RSP.r 
 (for example, by
	  a user script that breaks RSP sandbox).
	  

 o RSP: sanboxing now protects from Exit/Return/Break calls made outside 
 of a
	  function context.
	  
	o RSP: %misc/rsp-init.r file removed.
	

 o RSP/CGI: New config keyword added in global sections : 'worker-libs. 
 It lists the

   librairies to load when a worker process is started. An optional 
   'on-quit section can

   be added to call cleanup code when the process quits. Examples :
	  
	  		worker-libs [
				%libs/mysql-protocols.r
				...
			]
		or	
			worker-libs [
				%libs/mysql-protocols.r
				...
				on-quit [
					%/libs/free-resources.r
				]
			]
	

 o Task-master: now you can reset all worker processes without stopping 
 Cheyenne. This

   is usefull when you need to force non-RSP/CGI files reload (helper 
   scripts, 3rd party
	  librairies,...). 
	  
	  	Usage:
	
			Windows : tray icon -> Reset Workers
		

   UNIX : kill -s USR1 <pid>	(<pid> is Cheyenne's main process ID)
	 
	 

  o Added a -w command line option to set the worker processes number. 
	 
	   	Usage:
	   
	  		$ cheyenne -w <n>		(n [integer!] : CGI/RSP process number)
	   

    Use -w 0 to help debug CGI/RSP code by resetting worker processes 
    after each request.
	   (it's like calling "Reset workers" after each request).

this one is a little different and a bit more expensive on resources, 
if a request doesn't exist (no file match, no folder match) proxy 
reverse to cheyenne:
	DirectoryIndex default.html index.html
	RewriteEngine on
	RewriteCond /Volumes/data/web%{REQUEST_FILENAME} !-f
	RewriteCond /Volumes/data/web%{REQUEST_FILENAME} !-d
	RewriteRule ^(.*)$ http://cheyenne.com$1 [P]

Just unpacked 19 again and ran it .. noticed this


16/2-16:33:37.109-# Warning in [RSP] : Include RSP failed: max inclusion 
level reached - file: %inc.rsp !

Getting this when trying to load my login.rsp page in my webapp


16/2-16:46:24.353-## Error in [task-handler-49652] : Make object! 
[
    code: 303
    type: 'script
    id: 'expect-arg
    arg1: 'protected-exec
    arg2: 'code
    arg3: [block! function!]

    near: [protected-exec/event request/parsed/file get in session/events]
    where: 'fire-event
] !

It's something like framework... if you check the httpd.cfg, you 
can see something like:
webapp [
	virtual-root "/testapp"
	root-dir %www/testapp/
	auth "/testapp/login.rsp"
	;debug
]

When you that access the server with starting with the virtual-root, 
it's proccessed by Cheyenne using the %www/testapp/app-init.r file 
where are several handlers which are processed by the process. So 
you don't have to for example think how to update session time etc.

Would a simple file containing:

<% now %>

work?

As a design principle, I don't want to have code inside the config 
file. Why would you need that?

So I won;t forget about the app-init.r file :)

in uni-engine.r is used:
launch*: func [cmd [string!]][
	call join form to-local-file system/options/boot [" -s " cmd]
]
There must be -sq not just -s

Deploying the RSP stuff in an "encapped" way would be nice too. So 
just one file including cheyenne and all RSP stuff.

Robert, what your explaining is one of the main goal of Cheyenne. 
:-) Simple, lightweight, all-in-one file web applications deployement. 
Still some work to be able to achieve that, the main missing feature 
is a good and lightweight virtual file system for RSP scripts (so 
you can run them in source form or encapped with Cheyenne without 
changing anything).

I am not shooting down your idea.. I am just trying to say that cheyenne 
with just simgle -- exe + folder + config file -- already provides 
very deployable webapp solution, compared to for example installling 
apache + php + apache + pear.. and django / ruby (with just development 
server) also wasn't anywhere near as simple to install last time 
I tried

I have one question... after working in various other languages + 
mysql/sqlite I am using normal files with rebol structures and LOAD 
for my first projects here. Now I have a little more serrious project 
up so I started thinking if by using just files I can corrupt data 
somehow. I am not that good on low level details, but I imagine that 
I don't have to worry too much. Because cheyenne is single process 
I imagine only single write to file can happen to some file at any 
given time. Am I correct or wrong?

What you describe would mean, than you can only do one CGI request 
at the time. Cheyenne will launch new CGI process at each request, 
hence your file operations could collide. I like SQLite very much, 
but they don't provide server level functionality. They are able 
to work at file-lock level, but dunno how solid it is ...

I was just trying to say, that even with SQLite (which solves some 
file access sharing problems), you are accessing one file from multiple 
processes.

Janko, if you can ensure that only one request accesses one file 
than you are safe. If not, the last writer will win. But no data-corruption 
will happen.

I think that Robert is very brave with his statement :-) I would 
not bet if data can or can't be opened. If file is not locked, and 
you use write on it, and another process too, you can corrupt data, 
no?

I suspect on the system level at the time one write is in action 
file is locked for all other writes

Petr, the filesystem will ensure that this won't happen. The thing 
is, that for the time you write to the file, you get a file-lock. 
But this is immediatly released after you finished. So, if you try 
to write to a file with a lock, you get an error.

A DB handles this by having one file lock for the database file all 
the time, taking several request at the same time and doinga  DB 
locking scheme on-top of the filesystem locking.

Robert - I can understand, but you access one DB file from separate 
processes, hence separate DB engines.

Cheyenne serves static resources from the main process (UniServe 
process), but CGI and RSP are executed by pre-forked worker processes. 
So yes, writing to the same file from RSP script can be an issue 
if you don't have a mean to ensure that write accesses are serialized. 
I had that issue recently for RSP log/debug file, I had to build 
a small logger service in the main process to be able to serialize 
 write access (after stress testing different file locking solutions 
from REBOL, no one seemed reliable enough to me). 


I thought about adding a synchronization service in Cheyenne that 
could be used to (but not only) address the write file sync issue. 
That could work for low sync needs (like writing to a file once every 
few seconds), but for massive sync needs (dozens or hundreds of sync 
req/s), I'm afraid that the TCP port overhead would be too costly...(maybe 
a separate sync server process with persistent TCP connections could 
be good enough even for heavy uses?)

aha, so rsps work on multiple worker processes, interesting.. well 
I think to me it's not a problem as each user has it's own data files 
so it's hard to imagine multiple writes could happen at the same 
time for the same file. And if I would need something reall y heavy 
duty I would make a server serving files and caching them in ram 
for speed etc, which would also take care for sync. or something 
off the shelf

If you have a page with 2 (or more) frames, each pointing to RSP 
scripts that may all write to user's data file...that could be a 
problem. Same issue if your user opens 2 browser windows, or several 
users using the same account...the risks are not very high, but such 
cases can happen.

since processing of requwest seems to take just 1ms I think chances 
per user are slim. What would happen in it collides? would one of 
processes "get file access error" or something worse can happen by 
your exp? (I was asking about the fear of file getting corrup / half 
written before)

You can get file access errors and corrupted data in file (last write 
wins probably). A simple RSP page may be rendered very fast, but 
there's situations where it can take much more time. Imagine a complex 
query on database or using CALL to a third-party command-line tool.


With RSP pages rendering in a few ms, the risk for collision is very 
low, but it's not zero.

Oh, and if you really think that's it about it. Wait: If your files 
is on a Samba network share you have to deal with the Samba way of 
locking, not the OS where the file is stored on. And Samba locking 
can be problematic as well.

thanks for all info.. to me it's important difference if (1) last 
write wins or (2) data get's corrupted meaning you get file that 
doesn't have a rebol data that could be load-ed any more. Well I 
think I have been asking too much and should try experimenting and 
looking what happens. If only (1) can happen with very small probability 
(because I have many separate small files which don't get edited 
most of the time) in case of my current app it seems ok, but I will 
test (also what happens with read and write at the same time.) If 
(2) can happen at all then I will use/make a centralized file/data 
server right now so webapp will access no files directly any more 
and that server will have to take care of locking or serialize all 
reads writes.

I will have a "lab" project today with a title "Try to make a corrupt 
file" , this will be fun

I had a nasty experience just now.   I had spent the last couple 
of days writing my prototype website .. and got all the ajax stuff 
working.  I decided to reboot because the css wasn't showing properly 
in chrome but was working in FF.  Big mistake.  Windows Vista reported 
a problem booting, and started it's recovery process.  At the end 
of it, all my RSP files I had created, or edited, in the last 2 days 
were gone!  Other files ( html ) were untouched.  System restore 
failed to recover these files and using file recovery tools also 
was unable to locate them.

I guess Windows does not recognize RSP files and decides that they 
are potentially malicious ie. not a document file, and so removes 
them :(

Graham: you can set your webapp (or at domain level) in debug mode 
(using the 'debug keyword in config file). If the debug mode could 
be tested, it could allow you to enable/disable the captcha system 
(or anything else) based on the working mode (debug / in production). 
I'll add that to the todo list also.

Robert, xfdf is a deprecated format once used by Adobe to specify 
the contents of fields in a PDF.  So, the above xfdf file will populate 
the fields TextField1, TextField2, syupdfid with $fname, $surname, 
and $syupdfid ... ( well,  I will replace those before I send the 
pdf. )

So, I send this tiny xfdf file to the browser.  the browser loads 
up the acrobat plugin.  Acrobat now reads the pdf on my website as 
specified in the href field, and then fills in the fields in that 
pdf with the values as specified in the xfdf file.

Regarding httpd.cfg file, I'll look on how to avoid writing it down 
on disk when running from a custom encapped binary.

Not sure that's the right place. It's supposed to be a static information 
from the user POV, so the documentation might be a better place.


Btw, after reading again your posts above about that issue, you should 
have provided an encapped Cheyenne with your own httpd.cfg config 
file built-in (instead of using the default one). Moving the binary 
aournd wouldn't been an issue then.

Multiple running instances :


1 ) Cheyenne uses a port translation method for task-master to avoid 
clashing between Cheyenne's instances. This translation is currently 
triggered by the -p command line option specifying on which port(s) 
Cheyenne should listen. So the 'listen keyword in config file doesn't 
trigger the translation. After a quick look, it should be fixable.


2) Logger and RConsole services ports are not translated. This should 
be fixed too.


3) As a consequence of 1) & 2), Cheyenne can't currently run multiple 
instances on the same machine.


I am running two instances because the web app uses the same database 
name
 :


4) I very well aware of this issue and I intend to find a solution 
asap.

OpenSCManager failed : Access is denied
 :


5) I should load that routine! only if trying to run as a Windows 
Service without having admin rights.

Just replace the patched file in your current Cheyenne v0.9.19 sources 
folder.

The login? session special value is used only to control access when 
authentication protection is set using the AUTH keyword in config 
file. You need to set it only when you consider that the user has 
provided enough credentials to let him in (for example once he has 
filled a login form with correct ID / pass values).

All this to avoid a blank Acrobat page appearing.  if you call an 
*.xfdf file, acrobat opens up, and then the browser plugin starts 
up to load the remote PDF.  so, you have this blank PDF on screen 
as well. But if I use localhost to create a web page that emits the 
xfdf, then I only get one browser window with the pdf and no more 
blank Acrobat instance.

you can still just send any changes from a given revision forward 
as a single file (bundle)

there is a thing called an xfdf file ... when acrobat reads such 
a file, it goes to fetch the pdf in the href spec in that xfdf file.

Oh, I think I understand:

Before:
	Acrobat opens xfdf -> pdf -> image
Now:
	Acrobat opens xfdf -> pdf [image]


So before, the client Acrobat opens the xfdf file, which links to 
the pdf on your website, which links to the signature image on your 
website.

Because of the signature image, authentication is required when linking 
to the image, and Acroforms used in the PDF limits this to Basic 
Authentication.


Now that you've figured out how to embed the image in the pdf, the 
client Acrobat opens the xfdf file, which links to the pdf on your 
website, which has the signature image embedded already. Because 
of the signature image, authentication is required, but because the 
image is embedded in the pdf container, authentication is required 
when linking to pdf. (And I presume still limited to BA?)

How do you handle file uploads in Cheyenne RSP? I have a need to 
upload a file regularly. Post is fine - I don't need put.

Two ways.  If it is small enough, it is saved in the session, otherwise 
it is saved to a temp file which is accessed by the session variable.

so if request/posted is binary, then it's what was posted, but if 
it's a string, it points to a temporary file.

The uploaded file is written on disk if it's size exceed 100'000 
bytes. You can change this default threshold value by using the 'post-mem-limit 
 keyword in virtual host configuration options (in %httpd.cfg).


Cheyenne can handle file uploads up to 2GB size with no noticeable 
impact on memory usage.


The temporary file is stored in an %incoming/ folder at Cheyenne's 
root. The file is deleted once the response is sent to the client, 
so in order to keep it, you have to rename it or copy it elsewhere.


You can use the %show.rsp sample script as target of a POST action 
to test it.

If the file exceeds the post-mem-limit value I get an empty response 
error.
Error 324 (net::ERR_EMPTY_RESPONSE): Unknown error.

No file gets written to the %incoming/ directory, though that directory 
gets created.

I'm going to increase the post-mem-limit for now, but it would really 
help if I had the external file so I could have MySQL read it.

I just tested with latest Chrome (1.0.154.53) with %www/post.html 
test script and it seems OK here, no error. I get the file(s) in 
 %incoming/ folder.

Check if your FORM tags are using enctype="multipart/form-data" to 
indicate proper encoding for file uploading.

Here's the test source:
<!doctype html>
<html><body>

<form action="blah.rsp" method="POST" enctype="multipart/form-data">
<input type="file" name="ufile" size="16" />
<input type="submit" name="usubmit" value="upload" />
</form>
</body></html>

And here's the test target:
<%=mold request/content%>


I get the same error with show.rsp, so I went with something simpler.

I'm posting a 150k file.

What's the uploaded file name extension?

I've tried a .r, .csv, .exe, .zip. The only thing that affects things 
is whether the file size is larger than the post-mem-limit.

Weird, I've just tried uploading using Chrome various kind of file 
with size up to 10MB with no issue.

Great. Just tried 458k pdf files and I get the same Error 324, but 
with no logged script error and the temp file gets left there.

Strange. Any %crash.log file produced by Cheyenne?

no %crash.log file.

Cheyenne process script files declared in %httpd.cfg config file. 
In the default config file, you have :
	bind-extern CGI to [.cgi .r]

which means that both .cgi and .r are treated as CGI scripts and 
executed.

If you want to serve .r files as static files, just remove it from 
the BIND-EXTERN command in config file.

Yes, in my experience, as long as the other page is in the web app 
directory. You can't even use an external CSS file in the directory 
until you log in, even if it is referenced from the login page.

Cheyenne's mod-ssi currently only supports FILE and VIRTUAL directives 
(that's probably 90% of SSI usage). I've planned to add all the other 
missing directives, but it's quite low priority.

I think to add on Unix, search paths for config file in the following 
order : local folder, $HOME, /etc/ . Would that be enough to cover 
all use cases?

I used a file monitor and found the reason, it tries to load mcvcrt7.dll 
(visual c 7 runtime dll)  can't find it and gives error with "can't 
open service.dll"

I am porting an app to the latest cheyenne, I tried -vv and -vvv 
and uncommented the debug option in cfg file but I can't make output 
+ error  -- in case of error - show up in the browser. I saw trace.log 
and chey-pid*.log but that doesn't help me debug current case. Is 
there a way to show in browser while developing?

the current problem is that something is undefined which suggests 
that it doesn't load some of my library files. I did this in   on-page-start 
before (during development - I intended to move it to on-application-start 
when it's in full production mode). First thought is that that function 
doesn't execute so I put print "zzzzzz" into it and look the output, 
but I am unable to see the output before error hapened with my current 
knowledge now as browser always shows the "Sorry this page cannot 
be displayed" page and the log files don't hold that string either. 
(I tried adding print "xxxxx" above function that causes error too 
so I know it get's executed and I couldn't  find the string in any 
output or log file

Hi Doc. I am "porting" to 0.9.19 , the main thing that was causing 
causing the confusion since I didn't yet fully get it  was that "do" 
inside one of  another file that was "do-ed" didn't seem to use path 
relative to the that file but to web-app and before that wirtualroot 
property wasn't correct, I added the folder below root path.. now 
it works, and debug makes the whole output visible so I made it work 
better also.

my report is json data rebol2json and then javascript displays it 
all ,... it's all single static html file with data and javascript 
(via my jsgoo library :) )

As a result of an RSP script I want to return a new web-page that's 
on the file-system. I do it like this:

print read %payment/index.html


So far this works. What I need to do is, to insert some dynamic content 
into the read HTML file.

For this I see several options:

1. Somehow load this file as an RSP file and have it processed.

2. Somehow load this file as a SHTML file and have it processed as 
SSI file.

3. Add some marks and inser the stuff on the fly before delivering 
it.

Option 3 has the advantage, that I stay in my running RSP file context. 
On the other hand it's the "handmade" approach.

Q1 - WORDS dialect : this dialect allow defining new config keywords 
that can be used in the %httpd.cfg file for your mod. IN defines 
the config file sections where it can apply. Possible values are 
:
- globals : global config block used for server-wide options
- main : applies to a domain or webapp context
- location / folder : reserved for future use.

Well, maybe you can't. I haven't given any thought to what it takes. 
I was only thinking of the basics like large file transfer and proper 
working async ports and threading. some basics that a good webserver 
can do.

Having the TCP/IP part open-sourced in R3 will be great. It will 
allow to use much faster OS hooks for file transfers, extend the 
port! API to bind only on selected interfaces, etc...I wonder if 
the main event loop will be there also, so we can replace the not-scalable 
Select( ) call by other faster ones or even integrate libevent. That 
would definitely make Cheyenne able to handle a much higher number 
of connections.

How exactly does Cheyenne cache file loads? I have trouble getting 
a specific REBOL script to load.

I think I have found a bug where Cheyenne keeps serving an empty 
file, if I have first had it put in a server directory as a MacOSX 
shortcut and then replaced it with a real file of the same name.

Not sure how shortcut are handled by REBOL for OSX. The cached version 
is reloaded only if the file modification timestamp changed.

If you want mod-remark to take other mod-static caching, just provide 
a 'make-response handler in mod-remark, give it a higher priority 
and make sure to return a TRUE value (that will end the phase shadowing 
mod-static's handler for that phase). You can also just unload mod-static 
by commenting it inside GLOBALS section in config file (you'll have 
then to provide all the adequate handlers for serving static resources).

how do I use rconsole to tell cheyenne to reload a mod that has changed 
on disk... 


or does cheyenne detect mod-file changes and reload them automatically?

with your permission I would like to rebrand ssh-admin to  Cheyenne-admin.

its like a stand-alone cpanel for cheyenne-based systems.


All the current file and system commands options to manage the host 
server too will be complimented with any tools needed to configure 
and control cheyenne and any mods which are installed.

mine is already capable of all the file browsing/upload/download 
to the server  :-)

help! cheyenne has no trace.log file in logs, and I can't get the 
verbosity to work without using command-line args which isn't an 
option in my current setup.

You can set verbosity in the config file - check the comments in 
the file for how.

no comments of verbosity in my config file.... no trace.log in my 
main cheyenne dir  ' : -/

I have no .log file of any kind being created in any of the cheyenne 
dirs.

thanks for the log-dir config... should put ALLthe configs in the 
httpd.conf file and gray them, like apache does it... with comments 
on what each config does...

if you had log option and console option within the default config 
file, (commented out or not) then users choose what they want.

Max: I agree the main issue is not having config options documented. 
About the current logging rules, I've always found that's way handier 
to pass command-line options than having to edit a config file. I'll 
see in the next version how I can improve that. 


Btw, I recommend running Cheyenne as encapped binary on production 
servers, it's simplier to handle (especially on Unix) and more secure 
(you can't corrupt some vital source file).

I know in apache "virtualhost" commnad needs to be uncommented...I 
do not see anything else in the config file that idicates vhosting..

anyway... I forgot that amacleod needs the virtual working from outside 
so the hosts file probablz is not the right file to look at

Mostly nothing. Cheyenne is working as a reverse proxy and just servers 
an RSP file.

doc: might I do a RFE (request for enhancement)?


add a ./conf/  dir to cheyenne and load every file that ends with 
.cfg


this would allow us to distribute a configuration file with a module 
and provide setups per mod... its much more flexible to manage.

we could also have a setup for each vhost in our system, if that 
makes sense for the web admin.

cool.  and I'm reiterating the need to provide a sample file ala 
apache with a paragraph of comments or two which explain all configs... 
just in case you forgot to note it... this for me is big hassle.


for example... the subtleties behind 'BIND  and 'BIND-EXTERN  are 
not obvious to deduce just by the name... 
-what is the difference in how they are cached? 

-is an external handler explicitely needed with 'BIND-EXTERN   (no, 
in fact,  but it enables it)
-how does one use an external handler?

BIND associates one or more file extensions to a Cheyenne internal 
mod.

BIND-EXTERN associates on or more file extensions to a background 
handler (worker process through task-master service).

yes... I already new... but I had to trace the code and lost some 
time wondering why my page wasn't being re-rendered when I first 
used 'BIND  ;-)

I also had to trace the logic to make sure that cheyenne wasn't actually 
expecting an external handler if I used 'BIND-EXTERN...


I ended up loosing more than an hour to figure it out myself... now 
that is just one config... there are MANY... a lot of them I don't 
even know exist.


the above is exactly the kind of information which should be included 
within the httpd.cfg file, even if an example is commented out, but 
provided as an example use.  just like apache does it.

I hate Apache config file. Because I hate having to read tons of 
docs to just "switch on" some app. Cheyenne's config file has never 
been designed to copy the Apache way, nor to be used by average end 
user. It's just a placeholder waiting to be replaced by a builtin 
web GUI allowing a simple, fast and straightforward way to manage 
the server. That has been the plan since the beginning and one of 
the main motivation for building Cheyenne. Unfortunately, I never 
had the required time to complete that goal yet, so I'm stuck with 
that, and that's also why Cheyenne is still at 0.9.x.