World: r3wp

also... if you want to index domain and user, you should not use 
varchar without length.

aha, I will try that .. and I can use Integer for user ... because 
now each user get's folder like u<Int> ... and I could maybe put 
all 3 mailboxes into the same table so it would be only 1 insert 
/ update instead of 3 on changes ... I didn't think performance will 
be the problem here (it still seems a little funny) , because it's 
just a simple table without any relations or anything and no big 
amunt of data

I used sqlite here and there for more real DB work and I never seen 
any critical slownes (extept if you do a typical like inserting 100 
rows each in it's own transaction (without begin commit), in fact 
it seemed always very fast to me ... thats why I suspect all this 
wouldn't show up if I had some better VPS. Also because if fluctuates 
so much I suspect disk on computer vps is on is maybe busy doing 
other stuff so at one moment it is idle and it works faster at another 
it waits for >3 seconds

Yes.. it's possible, that the VPS does many other disk IO oparations 
so you have to wait.

but the changes you proposed will help me get it faster anyway and 
I will try them

you can use varchar even for indexes, it should speed up things significantly 
...

Is there a way to reorder columns in SQLITE?

alter table supports just renaming table and adding rows, for all 
else you need to create new table -> insert .. select ... ; data 
and drop old and rename new

In sql there is no need for reordering the column order, as you can 
get any order you desire by using the column names in your select 
statement.

That sounds helpful..thanks sqlab.

Can you do the same for inserting values?

I know I can do it for updating row data but I can not figure out 
the syntax for inserting..

For example:

SQL reduce [{UPDATE books SET bk=?, chap=? WHERE ref_number=?} blk/2 
blk/3 blk/1]

should be like
insert into table (col3, col2, col1) values (?, ?, ?)

works...thanks a lot. 

I had this fear of having to change large parts of my code if I added 
or removed columns...This solves that problem.

Janko: I just tested again two concurrent processes writing into 
one table.

This time I used the approach connecting and closing for each statement, 
as the link you gave suggested..
I still got locking.

I understood the text in link that if you get a lock, you wait for 
a while and try again.. and that by this method even it scales to 
quite big number of concurr processes compared to mysql for example

trying again is already implemented in sqlite.r in the function do-step.
I expanded the wait time with a random duration up to one second.
still not good enough

And now additionaly I connected and disconnected according the link 
.
Again I encountered locks

aha ... hm.. I havent tried it myself yet

how many times do you retry?

a few 10k times

what disconnection and connecting? :)

sorry, that was the number of statements until a lockk occurs

aha .. does it at least signidicantly reduce the occurence of the 
lock or it's the same as no retry?

do-step seems to retry until not busy

yes, it reduces, but they happen still

He tried only 100 times (by looking at the example he gave) .. and 
I don't know what he means by 500 >>> in line with mysql 

and others- at 500+ simultaneous users you start to get about a 10% 
drop 
because of lock overhead. <<< 


Most probably not 500 processes writing all the time ...  without 
delays

my mistake, I forgot to set the number of retries during connect.
I will test again

no difference, sooner than later I get a lock, even with 100 retries 
after less than 1000 statements

hm.. too bad

how are you testing it.. how many writes/sec .. processes?

sqlab, use a proxy process that does the priorization. Send requests 
from client to proxy-process and this one handles 1 connection to 
the database.

IMO you are much flexibler anyway.

You can add things like priority, read-only support etc.

Maybe my test settings are different from your  requirements.

During testing I let two concurrent processes insert a record with 
three fields, one ip address, a timestamp and one text field with 
variable length from 1k to 8k., and this i do with full speed

I get up to 200 inserts with just one process, with two concurrent 
process this slows down to 30 to 60 per second

I'm pretty sure a proxy process can handle 200req/s at the frontend 
and backend. So if 200req/s is the maximum you can get from one process, 
adding more processes just devides this down. But it won't scale 
up.

Robert, that could be a solution.

Unfortunately I observed a crosstalk of events with many ip connections, 

means one process is awakened  without data available, if I send 
data additionaly on the same server via ip

Don't understand the problem. Is it realted to my suggestion or your 
current try?

If I have some processes serving incoming events and sending their 
data to a central storage process, the central process sometimes 
seems to get an event, but not the data with that event.
When the data really arrives, I do not get the event.
Maybe he got the first event meant for an other connection

As I know, that there should be data available, I read again after 
timeout

Event = TCP/IP event?

yes

Hm... normaly there should be event/data

normally there is, but not always. 

Most socket connections are static, they stay connected for hours 
or longer.

Reading thru SQLite changelog found this - http://sqlite.org/rtree.html
... not sure of its practical usability, but some of you might find 
it interesting :-)

I love the sqlite motto  :-)   

Small. Fast. Reliable. Choose any three.

rebol.com motto - Web 3.0 starts here. Smarter, faster, better.

only that fast (at sqlite) is still a little problematic to me

SQLIte is fast for simple to middle local stuff. I have few obstacles 
with it 


1) it stores everything in one file. You can't use simplicity of 
file-system for simple back-up purposes. Attaching DBs (max 10 precompiled 
value) is not an option, as then transactions are not atomic

2) it is not secure - can't be secured easily, because encryption 
is not part of the package

3) serverless (both advantage = no install, but also disadvantage). 
It provides locking. They claim multiple instances of app can access 
one file, but I did not find more info on that. Dunno how granular 
locking you can do. You have to create server front-end yourself 
...

I use it for very simple task just so it takes case for locking of 
messages to bots that can come from multiple processes.. but at my 
small/cheap VPS that I use for bots update/insert/delete takes multiple 
seconds which is very bad.. all rebol writes/deletes which it does 
with normal files return imediately on same VPS and yesterday I tried 
rebDB and it was also much much faster for insert/update/delete (almost 
no noticable delay) for the same amount of data (300 lines) as sqlite.. 
funny thing is that sqlite delays the same at these operations if 
there is 300 or 5000 rows in table