Java DNS Cache

Well today has been fun, we have an application that calls out to a third party service via xml over https calls. The third party had scheduled some maintenance and notified us that their DNS entries would be updated so traffic would be redirected to their DR site. Well our webapp running under tomcat in the mid tier that makes the outbound requests never picked up the new DNS entries until we bounced the application. At first I investigated our connection pool Apache's MultiThreadedHttpConnectionManager but that was working fine and even if I emptied the pool the new DNS would not change. Well after some googling I came across this article that proved very helpful. http://www.rgagnon.com/javadetails/java-0445.html it turns out the JVM caches DNS entries and by default keeps them cached forever or at least until the application is restarted.

Solution: We needed to change the java.security file as described in the article. We choose to change networkaddress.cache.ttl=60 and add an external way to block our connection pool for 60 seconds during these regularly scheduled redirects rather than not cache any entries ever which is the case if you set
networkaddress.cache.ttl=0 .

Note: I was not able to change the value dynamically under JDK 1.5 as was suggested in the reference above. I could set the value, verify it was set but it would have no effect.

Testing:

To test this I used various levels of testing first and most basic was looking at how the InetAddress.getName(String name); would would with various settings in the java.security file. Basically, I edited the windows host file located in c:\windows\system32\drivers\etc and changed the ip address of one of our test servers.




import java.net.InetAddress;

public class Foo {
   public static void main(String[] args) throws Exception{
       System.out.println(sun.net.InetAddressCachePolicy.get());
       System.out.println("testDNSLoop ..: "  +
     java.security.Security.getProperty("networkaddress.cache.ttl"));
  String name = "dopey";
  InetAddress localAddr = InetAddress.getByName(name);
  System.out.println("reset timeout cache value to .: "  + 
      java.security.Security.getProperty("networkaddress.cache.ttl"));
  System.out.println("start loop...");
  while(true){
   localAddr = InetAddress.getByName(name);
   String host = localAddr.getHostAddress();
   System.out.println("IP: " + host + " ... sleeping 5 secs.");
   Thread.sleep(5000);
  }
   }
}

The next level of testing was confirming that the connection pool actually
followed the same behavior. For this I just put a simple test.html file on 2 of our test servers and ran a class similar to the above watching the output.

Hope this helps someone.