In addition to the above, I have just released version 1.0.4 of the adapter where if the IPN message is received by your host and is being rejected, you can view the reason in the admin interface. From the homepage:
Finally, if IPN messages arrive to your host but are being rejected, you can see a copy of the last error message and the time it was generated in the admin screen. The error message is retained for 1 hour and can help you debug problems with IPN messages that go through your firewall but are not accepted for some reason. This was implemented in version 1.0.4 of the adapter.
It’s worth looking at this, it will tell you if the problem is due to some firewall or misconfiguration.