The Mail Proxy in BorderManager 3.x has had a history of issues, many of which have been fixed in the latest proxy patches. (Be sure you check the installation readme file for specific Mail Proxy settings to put in the sys:etc\proxy\proxy.cfg file!).
However, you may still experience issues when using the Mail Proxy. These issues seem mostly to be related to sending outbound mail from an internal SMTP server through the Mail Proxy. Symptoms include mail getting hung up in the outgoing directory or problems sending mail to certain domains.
Here is an easy workaround - the concept is simple. Use Mail Proxy for inbound SMTP mail only! Let outbound mail go out
directly from the SMTP mail server via dynamic NAT and a stateful SMTP filter exception.
1. Configure Mail Proxy for internal SMTP mail server. (Put internal IP address of your SMTP mail server and mail domain in the Mail Proxy configuration. Add access rules to allow mail to be forwarded as follows:
a. Allow any source to send mail to your mail domain
b. Allow only your internal subnet (or SMTP mail server) to send to any mail domain.
c. Block all port 25.
These three rules, in that sequence, help to prevent spam relay off your mail proxy. Be sure to look at the readme files for the latest proxy patches (see tip #1 at this web site for latest patch information) for additional anti-spam relay settings for proxy.cfg.
2. Enable TCP/IP routing in INETCFG.NLM, Protocols, TCP/IP if you haven't done so already.
3. Enable Dynamic (or Static and Dynamic) NAT in INETCFG, Bindings, <your public IP address>, Expert TCP/IP Bind Options, Network Address Translation.
4. Set up a stateful SMTP filter exception in FILTCFG.NLM (smtp-st), source interface=<your private interface>, destination interface=<your public interface>.
5. Set up your internal SMTP mail server with a default gateway pointing back toward the BorderManager private IP address. (If on the same subnet, use the BorderManager private IP address as the default gateway/route).
6. Set up the internal SMTP mail service NOT to use a mail relay host. (If using GWIA, comment out the /MH= line in the sys:system\gwia.cfg file.)
This should allow your SMTP server to send outbound mail directly via dynamic NAT. Your SMTP mail server must be able to perform DNS queries, whether through dynamic NAT, DNS Proxy or an internal DNS server. Be sure you test that the SMTP mail server can resolve a URL (try pinging www.novell.com, and if you get an IP address for the ping, DNS is working).