SocketTimeoutException when writing data to hadoop cluster

I have an application that uses apache camel to transfer data from source to hdfs. When I run this application locally (mac) it is able to successfully transfer the data. But when I deploy the same to a different machine (RHEL) I am getting SocketTimeoutException.

When I looked for the file to be created from an edge node, I saw that the file was created but data was not written

-rw-r--r--   3 service_id Hadoop_lookupdata          0 2019-07-23 08:13 /data/send/SCHEMA/TABLE/2019/07/23/part-m-00000.opened

Stacktrace:

2019-07-23 08:13:03,744 [Camel (camel-1) thread #2 - timer://Timer] INFO  org.apache.camel.component.hdfs2.HdfsProducer -
            Connecting to hdfs file-system abc:8020/hdfs://abc:8020/data/send/SCHEMA/TABLE/2019/07/23/part-m-00000 (may take a while if connection is not available)
org.apache.camel.FailedToCreateProducerException: Failed to create Producer for endpoint: hdfs2://abc:8020/data/send/SCHEMA/TABLE/2019/07/23/part-m-00000?append=true. Reason: org.apache.camel.RuntimeCamelException: java.net.SocketTimeoutException: connect timed out
    at org.apache.camel.impl.ProducerCache.doGetProducer(ProducerCache.java:584)
    at org.apache.camel.impl.ProducerCache.acquireProducer(ProducerCache.java:168)
    at org.apache.camel.processor.RecipientListProcessor.createProcessorExchangePairs(RecipientListProcessor.java:202)
    at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:241)
    at org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:172)
    at org.apache.camel.processor.RecipientList.process(RecipientList.java:132)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
    at org.apache.camel.processor.idempotent.IdempotentConsumer.process(IdempotentConsumer.java:169)
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
    at org.apache.camel.component.direct.DirectBlockingProducer.process(DirectBlockingProducer.java:53)
    at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
    at org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:117)
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
    at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715)
    at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:638)
    at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248)
    at org.apache.camel.processor.Splitter.process(Splitter.java:122)
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
    at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715)
    at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:638)
    at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248)
    at org.apache.camel.processor.Splitter.process(Splitter.java:122)
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
    at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715)
    at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:638)
    at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248)
    at org.apache.camel.processor.Splitter.process(Splitter.java:122)
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
    at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:197)
    at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:79)
    at java.util.TimerThread.mainLoop(Timer.java:555)
    at java.util.TimerThread.run(Timer.java:505)
Caused by: org.apache.camel.RuntimeCamelException: java.net.SocketTimeoutException: connect timed out
    at org.apache.camel.component.hdfs2.HdfsFileType$1.createOutputStream(HdfsFileType.java:118)
    at org.apache.camel.component.hdfs2.HdfsOutputStream.createOutputStream(HdfsOutputStream.java:72)
    at org.apache.camel.component.hdfs2.HdfsProducer.setupHdfs(HdfsProducer.java:143)
    at org.apache.camel.component.hdfs2.HdfsProducer.doStart(HdfsProducer.java:104)
    at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61)
    at org.apache.camel.util.ServiceHelper.startService(ServiceHelper.java:75)
    at org.apache.camel.impl.DefaultCamelContext.deferStartService(DefaultCamelContext.java:1560)
    at org.apache.camel.impl.DefaultCamelContext.doAddService(DefaultCamelContext.java:1477)
    at org.apache.camel.impl.DefaultCamelContext.addService(DefaultCamelContext.java:1447)
    at org.apache.camel.impl.ProducerCache.doGetProducer(ProducerCache.java:578)
    ... 59 more
Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
    at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
    at sun.net.www.http.HttpClient.New(HttpClient.java:339)
    at sun.net.www.http.HttpClient.New(HttpClient.java:357)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
    at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:190)
    at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:131)
    at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
    at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322)
    at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:483)
    at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
    at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:478)
    at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776)
    at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
    at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1381)
    at org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1483)
    at org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1468)
    at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:451)
    at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
    at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
    at org.apache.camel.component.hdfs2.HdfsFileType$1.createOutputStream(HdfsFileType.java:103)
    ... 68 more

LATEST OBSERVATION:

When I ran the application locally (Working case). Saw the below log:

2019-08-02 16:38:40,466 [Camel (camel-1) thread #2 - timer://Timer] DEBUG org.apache.hadoop.security.authentication.client.KerberosAuthenticator -
            **JDK performed authentication on our behalf**.
2019-08-02 16:38:41,339 [Camel (camel-1) thread #2 - timer://Timer] INFO  org.apache.camel.component.hdfs2.HdfsProducer -
            Connected to hdfs file-system

When I ran the application remotely (Not working case). Saw the below log:

2019-08-02 10:22:15,620 [IPC Client (34343341) connection to host.com/1.2.3.4:8020 from user@abc.COM] DEBUG org.apache.hadoop.ipc.Client -
    IPC Client (34343341) connection to host.com/1.2.3.4:8020 from user@abc.COM: closed

Leave a Reply

avatar
  Subscribe  
Notify of