All Forums Teradata Studio
charlesbernard 10 posts Joined 12/12
13 Mar 2014
Hadoop profile creation fails

Hello there,
When creating a new Hadoop profile from TD Studio 14.10.01 nothing happens and I get an Unhandled event loop exception logged in the error log view:

java.lang.NullPointerException

at com.teradata.datatools.hadoop.integrator.oozieservice. OozieHadoopService.checkHDFSRoot(OozieHadoopService.ja va:406)

at com.teradata.datatools.hadoop.integrator.oozieservice. OozieHadoopService.checkHDFSConfiguration(OozieHadoopS ervice.java:386)

at com.teradata.datatools.hadoop.integrator.oozieservice. OozieHadoopService.checkHadoopConfigured(OozieHadoopSe rvice.java:321)

at com.teradata.datatools.hadoop.integrator.oozieservice. OozieHadoopService.ping(OozieHadoopService.java:295)

at com.teradata.datatools.hadoop.integrator.HadoopService s.pingHadoopIntegrator(HadoopServices.java:228)

at com.teradata.datatools.hadoop.dialogs.HadoopConnection Dialog$4.run(HadoopConnectionDialog.java:527)

at org.eclipse.swt.custom.BusyIndicator.showWhile(BusyInd icator.java:70)

at com.teradata.datatools.hadoop.dialogs.HadoopConnection Dialog.okPressed(HadoopConnectionDialog.java:506)

at org.eclipse.jface.dialogs.Dialog.buttonPressed(Dialog. java:472)

at org.eclipse.jface.dialogs.Dialog$2.widgetSelected(Dial og.java:624)

at org.eclipse.swt.widgets.TypedListener.handleEvent(Type dListener.java:240)

at org.eclipse.swt.widgets.EventTable.sendEvent(EventTabl e.java:84)

at org.eclipse.swt.widgets.Display.sendEvent(Display.java :4128)

at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java: 1457)

at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java: 1480)

at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java: 1465)

at org.eclipse.swt.widgets.Widget.notifyListeners(Widget. java:1270)

at org.eclipse.swt.widgets.Display.runDeferredEvents(Disp lay.java:3974)

at org.eclipse.swt.widgets.Display.readAndDispatch(Displa y.java:3613)

at org.eclipse.jface.window.Window.runEventLoop(Window.ja va:825)

at org.eclipse.jface.window.Window.open(Window.java:801)

at com.teradata.datatools.hadoop.view.HadoopContainer.han dleHadoopAdd(HadoopContainer.java:338)

at com.teradata.datatools.hadoop.view.HadoopContainer$5.w idgetSelected(HadoopContainer.java:287)

at org.eclipse.swt.widgets.TypedListener.handleEvent(Type dListener.java:240)

at org.eclipse.swt.widgets.EventTable.sendEvent(EventTabl e.java:84)

at org.eclipse.swt.widgets.Display.sendEvent(Display.java :4128)

at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java: 1457)

at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java: 1480)

at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java: 1465)

at org.eclipse.swt.widgets.Widget.notifyListeners(Widget. java:1270)

at org.eclipse.swt.widgets.Display.runDeferredEvents(Disp lay.java:3974)

at org.eclipse.swt.widgets.Display.readAndDispatch(Displa y.java:3613)

at org.eclipse.ui.internal.Workbench.runEventLoop(Workben ch.java:2701)

at org.eclipse.ui.internal.Workbench.runUI(Workbench.java :2665)

at org.eclipse.ui.internal.Workbench.access$4(Workbench.j ava:2499)

at org.eclipse.ui.internal.Workbench$7.run(Workbench.java :679)

at org.eclipse.core.databinding.observable.Realm.runWithD efault(Realm.java:332)

at org.eclipse.ui.internal.Workbench.createAndRunWorkbenc h(Workbench.java:668)

at org.eclipse.ui.PlatformUI.createAndRunWorkbench(Platfo rmUI.java:149)

at com.teradata.studio.StudioApplication.start(StudioAppl ication.java:33)

at org.eclipse.equinox.internal.app.EclipseAppHandle.run( EclipseAppHandle.java:196)

at org.eclipse.core.runtime.internal.adaptor.EclipseAppLa uncher.runApplication(EclipseAppLauncher.java:110)

at org.eclipse.core.runtime.internal.adaptor.EclipseAppLa uncher.start(EclipseAppLauncher.java:79)

at org.eclipse.core.runtime.adaptor.EclipseStarter.run(Ec lipseStarter.java:344)

at org.eclipse.core.runtime.adaptor.EclipseStarter.run(Ec lipseStarter.java:179)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMeth odAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Delega tingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.eclipse.equinox.launcher.Main.invokeFramework(Main .java:622)

at org.eclipse.equinox.launcher.Main.basicRun(Main.java: 577)

at org.eclipse.equinox.launcher.Main.run(Main.java:1410)

 

Thanks for helping.

 

Regards,

 

Charles

 

charlesbernard 10 posts Joined 12/12
13 Mar 2014

I forgot to mention this issue prevents me from exporting/importing data from/to Hadoop obviously using Studio

fgrimmer 553 posts Joined 04/09
13 Mar 2014

Did you download and install the Teradata Connector for Hadoop and Configure Oozie script on your Hadoop System mentioned on the download page ( http://downloads.teradata.com/download/tools/teradata-studio)?

Chuckbert 78 posts Joined 07/09
17 Mar 2014

The Exception in the first post is the result of a bug in the code that is checking to see that the Hadoop system has been configured to run the import/export jobs. It is indeed a result of the fact that the files that the Configure Oozie script sets up are not present. That script needs to be run before the connection profile can be made.

charlesbernard 10 posts Joined 12/12
20 Mar 2014

Yes indeed. Thanks for pointing this out as I missed this bash script.

Although I overcome this issue I get the following error:

The Oozie service cannot be reached, check the host name and port

I'm pretty sure I have given the right host and port though. I could telnet this hostname on this port.

By the way, for those using CentOS 6.4 you need to modify line 90 of the configureOozie.sh script to:
jarcmd=`ls -1 /usr/jdk64/jdk*/bin/jar 2> /dev/null | tail -1`
 
Thanks,
Charles

Chuckbert 78 posts Joined 07/09
20 Mar 2014

Is it possible that your Oozie service is not configured to allow your system or user to use the service?
What version of Hadoop are you using? In HDP 1.3.2, the Oozie configuration file oozie-site.xml has properties that allow hosts and users of groups to access the service. The following lines in the oozie-site.xml open the service to everyone:

    <!-- Hue Configuration -->

    <property>

      <name>oozie.service.ProxyUserService.proxyuser.hue.hosts</name>

    <value>*</value>

    </property>

    <property>

      <name>oozie.service.ProxyUserService.proxyuser.hue.groups</name>

      <value>*</value>

    </property>

The value elements can be set to be more restrictive. Could this be your issue?

charlesbernard 10 posts Joined 12/12
21 Mar 2014

Thanks for helping.
I haven't installed Hue and I don't really understand what Hue's got to do with tdch if it's indeed the tool used underneath?
It wouldn't make sense to add these properties if I haven't installed Hue, right?
I haven't experienced any issue running data transfer using tdch from command line.
 

charlesbernard 10 posts Joined 12/12
27 Mar 2014

Ok Chuckbert.
I overcome this issue by settings those properties below in core-site.xml to * as you suggested:
hadoop.proxyuser.oozie.groups
hadoop.proxyuser.oozie.hosts
However it seems that there is another issue if you haven't installed HDFS and Templeton (WebHCat) on the same node since you can only reference the first hostname in the hadoop profile wizard. After that the system assumes that templeton hostname is the same and you are stuck with this error:

Exception occurred when accessing the Hadoop database.

Unable to connect to http://xxxx.xxx.xxx:50111/templeton/v1/ddl/database?user.name=xxx

 

Can someone at Teradata confirm or pointing out where I'm wrong?

 

Thanks,

 

Charles

 

charlesbernard 10 posts Joined 12/12
27 Mar 2014

I confirm that you need to install WebHCatalog on the namenode in order to create an Hadoop profile.
 

Chuckbert 78 posts Joined 07/09
27 Mar 2014

The Oozie Configure Script does indeed set the WebHCatalog and WebHDFS hosts to be the same as the Name Node. A workaround if either is on a host other than the Name Node is to modify the file in HDFS named "/teradata/oozieworkflows/services.json". You can get it out of HDFS and modify the value for the "WebHCatalog" or "WebHDFS" attribute to be the name or IP address of the host of the service. You then can store the file back in its original location (with the same permissions it had).

charlesbernard 10 posts Joined 12/12
28 Mar 2014

Thanks for this workaround. It's indeed easier to modify this conf file than installing WebHCatalog on another node.

umerbaig 5 posts Joined 12/12
07 Apr 2014

I able to connect hadoop but getting this error:
Unable to connect to http://xxxx.xxx.xxx:50111/templeton/v1/ddl/database?user.name=xxx

Chuckbert 78 posts Joined 07/09
07 Apr 2014

@umerbaig, Do you have the same situation as discussed earlier, that is, where the WebHCatalog service is not running on the Name Node? If so, things might not work even with the workaround I mentioned (editing the HDFS file "/teradata/oozieworkflows/services.json").
The problem is, there is code that checks if the WebHCatalog service is available while connecting that uses the URL you've shown that uses the Name Node host instead of the WebHCatalog host specified in the services.json file.
If you put the URL that is in the error message into a browser's address bar and submit it, do you get a good response? Or is the host name portion of the URL using the name node instead of the WebHCat server?
This problem has been corrected in the version that is about to be released.

umerbaig 5 posts Joined 12/12
08 Apr 2014

I have successfully connected hadoop to teradata through teradata studio. I have configured configureOozie.sh file. WebHCatalog and WebHDFS had been assigned same IP address . I have changed them accordingly to host in which the services running.
thanks chukbert

umerbaig 5 posts Joined 12/12
14 Apr 2014

I am at beginner level in Hadoop. I want to know best configuration of hadoop components on multiple nodes i-e in my case there are 4 nodes.
Also want to know the multiple instance of some components ie zookeper and other components in hadoop. 
If I switch off one data node will my data be saved ? 
If I switched off namenode will I transfer to Secondary namenode?

fgrimmer 553 posts Joined 04/09
14 Apr 2014

@umerbaig, Please post your Hadoop questions to the Extensibility Forum. Thanks.

umerbaig 5 posts Joined 12/12
16 Apr 2014

*** Error 7827 Java SQL Exception SQLSTATE 39001: Invalid SQL state ([TD-
SQLH]: Error serializing hcatalog object: Server IPC version 9 cannot communi
cate with client version 4).
I have Hadoop 2.2.0.2.0.6.0-101 and trying to connect with sqlh teradata.

fgrimmer 553 posts Joined 04/09
17 Apr 2014

@#umerbaig, Again, please post your Hadoop question to the Extensibility Forum that handles questions on Hadoop. thanks.

You must sign in to leave a comment.