Concurrent access to data file
[1/35] from: coussement::c::itc::mil::be at: 19-Feb-2001 16:30
> Hi,
>
> the problem was discussed here some time before.
[sorry, I didn't got that]
> You need some atomic
> operation, which cannot be successfully done by two processes at the
<<quoted lines omitted: 3>>
> file locking suggested by Brett Handley is reliable only with a local
> file, not a file on a network disk.
[Now, I've learn something ! Thanks !]
> I think that you should use TCP port on the server as suggested by
> Brett Handley. There are also several distributed mutual exclusion
> algorithms that don't require a central server, look at this paper for
> more information:
> http://www.cse.iitk.ac.in/~deepak/cs632/mutual-exclusion.ps
[I'm not familiar with post-script files. Which soft do you use to
read it ?]
> HTH,
>
> --
> Michal Kracik
[
Thanks for your answer,
chr==
]
[2/35] from: kracik:mbox:dkm:cz at: 19-Feb-2001 17:40
Hi Ladislav,
sorry, I looked at it and it is atomic in both NFS and SMB. Next time
I'll check before confusing anyone. Still there's a possibility that a
client does not know that it succeeded or failed if the reply is lost
and the connection time-outs. After the connection is re-estabilished,
recovery should consider this situation, but that can happen even with
a local file when power fails at the "right" time.
Regards,
--
Michal Kracik
Ladislav Mecir wrote:
[3/35] from: kracik:mbox:dkm:cz at: 19-Feb-2001 17:43
Hi,
CRS - Psy Sel/SPO, COUSSEMENT Christophe, CPN
wrote:
> > Hi,
> >
<<quoted lines omitted: 8>>
> > file, not a file on a network disk.
> [Now, I've learn something ! Thanks !]
Sorry for confusion, rename operation is atomic even on network
drives. Now I've learnt something from Ladislav ;-) I'm not sure about
file locking though.
> > I think that you should use TCP port on the server as suggested by
> > Brett Handley. There are also several distributed mutual exclusion
<<quoted lines omitted: 14>>
> [rebol-request--rebol--com] with "unsubscribe" in the
> subject, without the quotes.
--
Michal Kracik
[4/35] from: coussement:c:itc:mil:be at: 19-Feb-2001 10:39
Hi REBOLians:
I'm confronted with a little but important problem...
I'm using a block structure for storing data on a server.
40 PC's -soon 80- could potentialy dump their data on the same time into one
single data file on the server.
Because we using Win NT -which I do not trust for reability-, I use a system
of token to insure concurrent access to the file:
<snip>
forever [
either all [(count<= 20) token?] [
count: count + 1
;--- attempt to reserve token
if error? err: try [write token-file now (true)][
disarm err
return false
]
wait 5 ;--- take delay to be sure token is written
;--- try write operations
answers: load local-file
either error? err: try [save remote-file append load
remote-file answers (true)][
;--- release token
delete token-file
disarm err
return false
][
;--- release token
delete token-file
delete local-file
return true
]
][
return false
]
]
token?: func [][
return not exists? rejoin [global-answer-path global-token]
]
</snip>
The token is just a text file in this case, and gives the PC the right to
talk
to the server.
The proc checks for the existence of a token on the server, for a maximum of
20 times. If there's no token on the server, it writes its own one, dump the
data and release the token.
Empirically, I had to set a delay of 5 sec to ensure the token is written,
because of the slow network connection.
Well, my question is not about straight REBOL, but more a matter of
methodology :
The procedure works, but, as you can imagine, it is not very efficient for
so many PC working and dumping together.
How can I achieve a better performence regarding concurrent access in REBOL
on WinNT, which can garanty me no data will be lost ?
Best regard,
chr==
[5/35] from: petr:krenzelok:trz:cz at: 19-Feb-2001 11:29
Hi,
I don't have currently enough free time to read whole your email, but I would
like to suggest you to use semaphore like solution:
sem: open/new %semaphore.sem
->> error? try [delete %semaphore.sem]
== true
You can't delete your semaphore file while OS holds lock on it .... but be
carefull and don't open several instances of 'sem ...
The advantage here is - even if your app crashes, the solution will work,
because OS will free lock it holds upon the file and rebol will succesfully
delete it ... or so I think :-)
Hope this helps,
Cheers,
-pekr-
CRS - Psy Sel/SPO, COUSSEMENT Christophe, CPN
wrote:
[6/35] from: coussement:c:itc:mil:be at: 19-Feb-2001 11:51
Thanks Petr, I will test it at once !
chr==
[7/35] from: brett:codeconscious at: 19-Feb-2001 22:58
Hi Christophe,
Firstly I assume this bit of code is going to be running on each of the 80
Pcs, not as some sort of server process. Is this the case?
(1) If so, I think your proposed method is not going give you any of the
protection you expect.
There could be a "window of opportunity" for another process to get in even
though it should not.
The weakness is that two processes could both succeed in "reserving" the
token.
(2) The other problem is the possibility that a program will fail due to an
error yet leave the token in place so that other processes believe it is
locked.
I suspect that you really need to trust file locking, but the problem is I
haven't been able to work out if there is a way to get Rebol to lock files.
I attempted this:
x: open/write/direct/binary %lock-file.dat
and again in another rebol session which did not return an error so it seems
that Rebol appears to opening the file without exclusive write locking.
With them still open I attempted to open the file with Wordpad.exe. Wordpad
did not open the file - complaining that another application had the file
open - correct. To verify I close the opened ports and tried wordpad.exe
again - this time it opened.
So I believe that it would be good if rebol had file locking abilities.
However I found that if you open a file like I did above, and try to rename
it you will cause an error - indicating the file is in use. This was
confirmed when I belatedly read my site :-\ below
http://www.codeconscious.com/rebol/rebol-net.html#Filelocking
This indicates that a good idea might be for the client to open a tcp/ip
connection with a rebol process on the server. This way if the client
process dies, the connection should (?) timeout - thus dealing with problem
(2). I haven't tried it so ...
Hope it helps.
Brett.
----- Original Message -----
From: "CRS - Psy Sel/SPO, COUSSEMENT Christophe, CPN"
<[COUSSEMENT--C--ITC--mil--be]>
[8/35] from: rgombert:essentiel at: 19-Feb-2001 13:38
If the system does not have to be full realtime, each PC can write data in
his own file and an administartor (or a cron) can check for all of these
files and update the main one... then delete each individual files.
This way, each PC could know if his file have allready be processed, and
concurent acces is no more a problem.
Renaud
[9/35] from: coussement:c:itc:mil:be at: 19-Feb-2001 13:56
> This indicates that a good idea might be for the client to open a tcp/ip
> connection with a rebol process on the server. This way if the client
> process dies, the connection should (?) timeout - thus dealing with
> problem
> (2). I haven't tried it so ...
[good suggestion: it's IMHO the most safe way to do it. I will test
it ASAP.
Thanks a lot !
chr==
]
[10/35] from: coussement:c:itc:mil:be at: 19-Feb-2001 13:52
Thanks for the suggestion, but the system HAS to work in real time: the
results file have to be processed immediatly.
Thanks anyway !
Regards,
chr==
[11/35] from: kracik:mbox:dkm:cz at: 19-Feb-2001 15:36
Hi,
the problem was discussed here some time before. You need some atomic
operation, which cannot be successfully done by two processes at the
same time. One such operation is in most OSes (including Windows NT)
local file rename. Any operation on network drives (NetBios or NFS) is
not atomic, so you cannot rely on it when synchronizing multiple PCs -
file locking suggested by Brett Handley is reliable only with a local
file, not a file on a network disk.
I think that you should use TCP port on the server as suggested by
Brett Handley. There are also several distributed mutual exclusion
algorithms that don't require a central server, look at this paper for
more information:
http://www.cse.iitk.ac.in/~deepak/cs632/mutual-exclusion.ps
HTH,
--
Michal Kracik
CRS - Psy Sel/SPO, COUSSEMENT Christophe, CPN
wrote:
[12/35] from: lmecir:mbox:vol:cz at: 19-Feb-2001 16:12
Hi Michal,
Did I understand correctly, that RENAME is not an atomic operation for
network drives?
Regards
Ladislav
[13/35] from: coussement:c:itc:mil:be at: 20-Feb-2001 9:01
> Hi,
> >
<<quoted lines omitted: 14>>
> drives. Now I've learnt something from Ladislav ;-) I'm not sure about
> file locking though.
[Thanks for the tip. I will try to test such a solution.]
> > > I think that you should use TCP port on the server as suggested by
> > > Brett Handley. There are also several distributed mutual exclusion
> > > algorithms that don't require a central server, look at this paper for
> > > more information:
> > > http://www.cse.iitk.ac.in/~deepak/cs632/mutual-exclusion.ps
[I'm still interrested in your paper, but still don't know how to
read it.
I apologize for the simplicity of my request, but how can I access a
readable content ?]
> >
> > > HTH,
> > >
> > > --
> > > Michal Kracik
[Best Regards,
chr==]
[14/35] from: brett:codeconscious at: 20-Feb-2001 19:33
> > > http://www.cse.iitk.ac.in/~deepak/cs632/mutual-exclusion.ps
> [I'm still interrested in your paper, but still don't know how to
> read it.
> I apologize for the simplicity of my request, but how can I access a
> readable content ?]
Try Ghostscript and GSView
http://www.cs.wisc.edu/~ghost/
They worked for me.
Brett.
[15/35] from: andrew:wxc at: 20-Feb-2001 21:31
> > > > more information:
> > > > http://www.cse.iitk.ac.in/~deepak/cs632/mutual-exclusion.ps
> [I'm still interrested in your paper, but still don't know how to read it.
I apologize for the simplicity of my request, but how can I access a
readable content ?]
Try the GhostScript viewer available from here:
http://www.cs.wisc.edu/~ghost/gsview/
for various computer systems. It should be able to read the file.
Andrew Martin
ICQ: 26227169 http://members.nbci.com/AndrewMartin/
[16/35] from: sharriff:aina:med-iq at: 20-Feb-2001 8:39
Hi Michal!
I think youŽll have to download a Postscipt viewer for it, for windows use
Ghostscript
, its freeware...
Sharriff Aina
CRS - Psy
Sel/SPO, An: [rebol-list--rebol--com]
COUSSEMENT Kopie:
Christophe, CPN Thema: [REBOL] Re: Concurrent access to data file
<[COUSSEMENT--C--I]
TC.mil.be>
Gesendet von:
[rebol-bounce--re]
bol.com
20.02.01 08:01
Bitte antworten
an rebol-list
> Hi,
> >
<<quoted lines omitted: 6>>
> > > same time. One such operation is in most OSes (including Windows NT)
> > > local file rename. Any operation on network drives (NetBios or NFS)
is
> > > not atomic, so you cannot rely on it when synchronizing multiple PCs
-
> > > file locking suggested by Brett Handley is reliable only with a local
> > > file, not a file on a network disk.
<<quoted lines omitted: 3>>
> drives. Now I've learnt something from Ladislav ;-) I'm not sure about
> file locking though.
[Thanks for the tip. I will try to test such a solution.]
> > > I think that you should use TCP port on the server as suggested by
> > > Brett Handley. There are also several distributed mutual exclusion
> > > algorithms that don't require a central server, look at this paper
for
> > > more information:
> > > http://www.cse.iitk.ac.in/~deepak/cs632/mutual-exclusion.ps
[I'm still interrested in your paper, but still don't know how
to
read it.
I apologize for the simplicity of my request, but how can I
access a
readable content ?]
> >
> > > HTH,
> > >
> > > --
> > > Michal Kracik
[Best Regards,
chr==]
[17/35] from: andrew:wxc at: 20-Feb-2001 21:58
> I'm confronted with a little but important problem...
>
> I'm using a block structure for storing data on a server. 40 PC's -soon
80- could potentialy dump their data on the same time into one single data
file on the server.
Perhaps better would be for the 40 (or 80) PCs to each write their own data
into a file on a shared folder on their Hard Disk Drive. That way the PCs
don't have to _compete_ for network access at the same time. Then a program
on the server can periodically read each PC (say, every ten seconds or when
convenient) and accumulate the data files, one at a time.
Andrew Martin
Asbestos-suited Rebol...
ICQ: 26227169 http://members.nbci.com/AndrewMartin/
[18/35] from: timewarp:sirius at: 20-Feb-2001 1:16
i solved this problem with poem access on my site by
creating new read and write functions that "lock" the
file while it's being read from or written to.
the lock is just a temporary file that is removed as
soon as the read or write is finished. the lock exists
only as long as it takes to grab the data out of the
file or write to the file ... so the lock does not
exists the whole time the script is running, just for
about a fraction of a second, if that.
read locks are time stamped so there may be multiples
of them, this is because a read lock only prevents
writing, not reading. a write lock is just a single
file that prevents other processes from writing to or
reading from the file.
the read and write functions recheck for the lock
file's existence about every 100th of a second. the
read function only checks for a write lock while the
write function checks for a read or write lock.
i'd share the functions with you, but i wrote them
in perl. should be very easy to do in rebol.
cheerfulness,
-----EAT
Andrew Martin wrote:
[19/35] from: timewarp:sirius at: 20-Feb-2001 1:18
another solution, one that bo told me about, is to write
a serialization server. then send all the data to the server
script and let it handle writing/reading to/from the one
data file on it's own.
Andrew Martin wrote:
[20/35] from: coussement:c:itc:mil:be at: 20-Feb-2001 10:27
> > I'm confronted with a little but important problem...
> >
<<quoted lines omitted: 9>>
> when
> convenient) and accumulate the data files, one at a time.
[Thanks for answering.
I did consider this solution. It's indeed IMHO a safe one, and
easy-to-implement, but I would like to go for a more "elegant" solution.
If I do not find anything else I will use that.
Best Regards,
chr==
]
[21/35] from: coussement:c:itc:mil:be at: 20-Feb-2001 10:39
> another solution, one that bo told me about, is to write
> a serialization server. then send all the data to the server
> script and let it handle writing/reading to/from the one
> data file on it's own.
[I didn't already consider this one. It seem to be an usefull one in
my case.
I will launch some tests ASAP.
Thanks !
Best Regards,
chr== ]
[22/35] from: andrew:wxc at: 20-Feb-2001 22:56
Probably best to find out if the network between the PCs and the server is a
limiting factor. I was thinking it was an internet or slow LAN network and
big messages. If it's a fast network with small messages, then Erin's
solution is better.
Andrew Martin
Taking a cold shower...
ICQ: 26227169 http://members.nbci.com/AndrewMartin/
[23/35] from: coussement:c:itc:mil:be at: 20-Feb-2001 10:53
> > > > > more information:
> > > > > http://www.cse.iitk.ac.in/~deepak/cs632/mutual-exclusion.ps
<<quoted lines omitted: 5>>
> http://www.cs.wisc.edu/~ghost/gsview/
> for various computer systems. It should be able to read the file.
[Thanks to all for answering !
There seem to be some problem with their ftp server now, but I'll
download and test it as soon as possible.
Best Regards,
chr==
]
[24/35] from: andrew:wxc at: 20-Feb-2001 22:59
> I would like to go for a more "elegant" solution.
Rebol/Express is an elegant solution.
Andrew Martin
ICQ: 26227169 http://members.nbci.com/AndrewMartin/
[25/35] from: coussement:c:itc:mil:be at: 20-Feb-2001 10:37
> i solved this problem with poem access on my site by
> creating new read and write functions that "lock" the
<<quoted lines omitted: 16>>
> i'd share the functions with you, but i wrote them
> in perl. should be very easy to do in rebol.
[Thanks Erin for answering.
I think the solution you propose here is the one I presented when
initiating this thread.
It's a -relative- safe one (see thread for discussion about it) but
will give poor performences when 40 to 80 PC's you want to dump their data
together ...
Thanks anyway ;-)
Best Regards,
chr==
]
[26/35] from: coussement:c:itc:mil:be at: 20-Feb-2001 11:06
> Probably best to find out if the network between the PCs and the server is
> a
> limiting factor. I was thinking it was an internet or slow LAN network and
> big messages. If it's a fast network with small messages, then Erin's
> solution is better.
[It's a slow LAN and about 5kb messages.
Although I will test them all I think your solution is the most
robust one.
Thanks, CU
chr==
]
[27/35] from: coussement:c:itc:mil:be at: 20-Feb-2001 11:08
> > I would like to go for a more "elegant" solution.
>
> Rebol/Express is an elegant solution.
[
Right !
I don't know how much RT will ask for this, but refering to a thread
from some time ago, I think it will be far above the budgets I could get for
the project !
Bad luck for me ;-\
Cheers,
chr===
]
[28/35] from: g:santilli:tiscalinet:it at: 20-Feb-2001 12:06
CRS - Psy Sel/SPO, COUSSEMENT Christophe, CPN
wrote:
> Thanks for the suggestion, but the system HAS to work in real time: the
> results file have to be processed immediatly.
The simplest way might be to serialize access using a server. Your
server process takes care of updating your file; it stays there
listening to a TCP port and accepts one connection at a time only.
Your clients just connect to the server and send it the data.
HTH,
Gabriele.
--
Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer
Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/
[29/35] from: coussement:c:itc:mil:be at: 20-Feb-2001 12:47
> > Thanks for the suggestion, but the system HAS to work in real time: the
> > results file have to be processed immediatly.
>
> The simplest way might be to serialize access using a server. Your
> server process takes care of updating your file; it stays there
> listening to a TCP port and accepts one connection at a time only.
> Your clients just connect to the server and send it the data.
[
Thanks for sharing ;-)
I will try a the serialization server solution ASAP.
Cheers,
chr==
]
[30/35] from: sharriff:aina:med-iq at: 20-Feb-2001 12:58
Excuse my question, but could this slow down the workflow if only one
connection is allowed to the server in the case of 20 or more PCs?
Regards
Sharriff Aina
Gabriele Santilli
<[g--santilli--tisca] An: [rebol-list--rebol--com]
linet.it> Kopie:
Gesendet von: Thema: [REBOL] Re: Concurrent access to data file
[giesse--dsiaq1--ing]
.univaq.it
20.02.01 11:06
Bitte antworten
an rebol-list
CRS - Psy Sel/SPO, COUSSEMENT Christophe, CPN
wrote:
> Thanks for the suggestion, but the system HAS to work in real time: the
> results file have to be processed immediatly.
The simplest way might be to serialize access using a server. Your
server process takes care of updating your file; it stays there
listening to a TCP port and accepts one connection at a time only.
Your clients just connect to the server and send it the data.
HTH,
Gabriele.
--
Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer
Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/
[31/35] from: timewarp:sirius at: 20-Feb-2001 7:04
it seems to me the server can accept more than one connection
at a time, but it would "serialize" the data being output to
the data file. i don't see why it would accept only one
connection at a time.
[Sharriff--Aina--med-iq--de] wrote:
[32/35] from: jeff:rebol at: 20-Feb-2001 8:44
Depends on the platform, but here under linux:
>> x: open tcp://:9922
>> y: open tcp://localhost:9922
<<quoted lines omitted: 3>>
>> c: open tcp://localhost:9922
>> d: open tcp://localhost:9922
-\|/-\|/-|/-\|/-\|/\|/-\|/-\|-\|/-\|/-\/-\|/-\|/-|/-\|/-\|/\|/-\|/-\ . . .
(will eventually time out)
The listen port has a certain fixed sized queue, but if it can
dispatch connections faster than the queue can fill up it can
handle an unlimited amount of connections.
[33/35] from: g:santilli:tiscalinet:it at: 21-Feb-2001 12:41
[Sharriff--Aina--med-iq--de] wrote:
> Excuse my question, but could this slow down the workflow if only one
> connection is allowed to the server in the case of 20 or more PCs?
It depends on the kind of access you are doing. File locking would
cause waiting too.
The efficient way to handle concurrency on a set of data is to use
transactions and a good scheduler. This is not very simple, and it
does not seem to be applicable to the case discussed here, at
least as I understood it.
A simple serialization server should be fine in this case IMHO.
Regards,
Gabriele.
--
Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer
Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/
[34/35] from: sharriff:aina:med-iq at: 21-Feb-2001 12:43
Anybody with a URL where I can RTFM on "Serialisation on Servers" for
dummies? further is there some way to implement "Object serialisation"
in REBOL without doing handstands?
Sharriff Aina
Gabriele Santilli
<[g--santilli--tisca] An: [rebol-list--rebol--com]
linet.it> Kopie:
Gesendet von: Thema: [REBOL] Re: Antwort: Re: Concurrent access to data file
[giesse--dsiaq1--ing]
.univaq.it
21.02.01 11:41
Bitte antworten
an rebol-list
[Sharriff--Aina--med-iq--de] wrote:
> Excuse my question, but could this slow down the workflow if only one
> connection is allowed to the server in the case of 20 or more PCs?
It depends on the kind of access you are doing. File locking would
cause waiting too.
The efficient way to handle concurrency on a set of data is to use
transactions and a good scheduler. This is not very simple, and it
does not seem to be applicable to the case discussed here, at
least as I understood it.
A simple serialization server should be fine in this case IMHO.
Regards,
Gabriele.
--
Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer
Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/
[35/35] from: g:santilli:tiscalinet:it at: 22-Feb-2001 20:18
Hello [Sharriff--Aina--med-iq--de]!
On 21-Feb-01, you wrote:
S> Anybody with a URL where I can RTFM on "Serialisation on
S> Servers" for dummies? further is there some way to implement
S> "Object serialisation"
S> in REBOL without doing handstands?
With "serialization" I intended "turning concurrent accesses into
serial accesses"; that has nothing to do with object
serialization. Sorry for the confusion.
Regards,
Gabriele.
--
Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer
Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/
Notes
- Quoted lines have been omitted from some messages.
View the message alone to see the lines that have been omitted