World: r3wp

SVN r17 : big update, lot of code added mostly for the new embedded 
async Mail Transfer Agent (MTA)

FEAT: email async sending sub-system (MTA) added . 

FEAT: added two new functions for email support in RSP/CGI scripts: 
'send-email and 'email-info?
FEAT: added email sending demo form %www/email.rsp.

FEAT: Cheyenne's main process ID is now exported in /tmp/cheyenne.pid 
on start and deleted on quit. (All platforms except Windows).

FIX: fixed broken global words protection in RSP.r.
FIX: HTTP Date headers are now in UTC format.

FIX: "Reset Workers" menu wasn't working in service mode. Fixed now. 
(Thanks Will)

FIX: SVNr4 regression bug on system port for UNIX fixed. (Thanks 
Will)

FIX: multipart file uploading code refactored. Fixes many bugs in 
file uploading.

A few words about the new MTA and email capabilities :


- Emails are written to disk before sending in async mode. Upper 
email size limit is 2GB. Concurrent emails sending supported (limited 
only by machine resources).

- Only the higher priority MX is used (fallback on others MX should 
be added soon).

- The MTA currently supports only 8BITMIME compliant SMTP servers 
(fallback to 7bit mode will be added if it still make sense to support 
it).

- MTA will  try to send an email once. If the server is busy, it 
will report a failure. Deferred retries will be added using the scheduler 
module (if MX fallback is not enough).

- Emails queue saving should be added in next update (the queue will 
survive to server restart, but not server crash).

- Automatic failure reports are sent back to emitter (From: field).
- Reports can be fully customized.

- Still needs a lot of testing for tracking small errors or issues 
that can trigger antispam filters (e.g. gmail is very picky about 
correct emails format).


You can test it using the provided test script : http://localhost/email.rsp

In addition :

- custom charset can also be set (default to ISO-8859-1").
- attached files are supported.

SVN r18: 

DOC: added quick syntax overview for email API in %www/email.rsp

Btw, in order to forge emails to be sent, I've tried to rely on REBOL's 
builtin email support functions (big mistake!). You should know that 
they *are not* RFC compliant, the biggest issues being :


- emails produces by REBOL are using LF as EOL instead of CRLF (RFC 
2822). See http://cr.yp.to/docs/smtplf.html

- headers are not encoded for non ASCII-7bits characters (RFC 2047)


So, I've deeply patched the builtin code at runtime to workaround 
this, but, I should have better rewrote it all from scratch (that's 
what I intend to do when I'll have enough free time).

Write up your patches and we can see about getting them integrated 
into the next version of R2 :)

I didn't though that Carl would still support R2 by providing new 
versions...(surprising but) good news. :-)

We need it for R3 too, so it would be wonderful if they could be 
written simultanoeusly.

nice on the mta ... maybe I should get back to work on my mail server

I'd probably use Amazon for email.

stick the email in a SQS queue ... and remove from the queue when 
the send is successful.

so that way the cheyenne mta can keep trying over 4 days to send 
the mail.

If my cheyenne server goes down .. I can resume from another instance.

Cheyenne can be configured to work that way too, but is it desirable? 
I remember the way email was handled 15 years ago, when having failure 
reports after one or 2 days wasn't bothering anyone, but today, we're 
in a world of fast communication (often realtime). When I send a 
mail from gmail, if there's an issue, I get a report in less than 
a minute or two.

I had an email that took 2 days to get a report back from gmail that 
it could not be delivered.

That was last month too.

I was also thinking of how scaleable it could be made ...

Were you satisfied by that or would have you prefered that gmail 
gives you a failure report 5 minutes max after posting your email?

I'd prefer instant failure report ...

yeah cause we'll try another communication channel right away  :-)

but that's not detemined by gmail ... the recipient server might 
be the cause.

If you use SQS then you could have a farm of Cheyenne servers sending 
the mail :)

Me too, that's why I'm reluctant to implement it in the old fashion 
way (retrying every 8 or 12 hours seems way too long now). I would 
support trying without delays alternative MX servers and if all fail, 
try 1 or 2 minutes after that. That would be by default, but could 
be extended by user if required.

of course it could be your own isp that is the issue ... sometimes 
we have lost interrnational traffic, but local traffic is okay.

Emails sending and MX querying are all fully async. Emails are streamed 
from disk, no memory overhead whatever the email size. You could 
send hundred (maybe even thousands on fastest machines) of email 
per second using Cheyenne if needed.

so failing every email ... would not be good.

Guess we need to field test .. and see how it works

Anyone got a list of 1,000,000 valid email addresses for sale??

Yes, it needs heavy testing, but works quite well so far.

:-)

If you're not using your ISP's smtp server, won't the recipient notice 
your domain and the email from address don't match and flag your 
email as spam??

Unless your smtp server is a trusted or whitelisted server

Well, I used to use that criteria when I had my own mail server ... 
I grey listed all such messages.

So, the question is also, does the mta handle greylisting ?

greylisting is when you temp fail an incoming message so that they 
are forced to wait eg. 5 mins or 30 mins before you accept their 
email.

Yes, some SMTP servers apply such restrictive rules, but not the 
big ones (AFAICT). But the more important is to have a reverse dns 
that matches your servers domain.

spammers don't retry .. or retry immediately

they don't delay for 5 mins and retry .. as it is too expensive

Cheyenne's MTA waits for 5mn before raising a timeout as per RFC.

It's not a timeout, it's a temp failure.

Never encountered that so far while testing with the big ones (Yahoo, 
Gmail, Hotmail, French ISPs)

http://www.compkarori.com/vanilla/display/Smtpd.r

This is my teergrubbing greylisting mail server for Uniserve

I tried sendind from home computers connected through either cable 
or ADSL connections. Only Yahoo!Mail rejected emails sent from ADSL 
network.

I was quite surprised to see that Gmail and others accepted emails 
sent directly from a MTA on home computer...(I thought that I would 
need to make all testing from remote servers).

I also would now drop that restriction ...

and analyse the email instead ...

but in 2005 I rejected all broadband based mtas

Yeah, I studied your smtpd.r implementation. I thought I would need 
to support greylisting, but it seems not to be required to deliver 
emails to the big ones.

Good to know ...

Thanks Dock for this new release 8)