All Forums Teradata Studio
h2oknut 5 posts Joined 05/10
09 Aug 2015
Hadoop Databases not found

Hi
 Running hdp vm 2.1, TD Studio  15.10.00 (on MAC) (no problems with 15.10.3 TD Studio express on MAC)
connect to HDP VM, and cannot pull up Hadoop databases on HDP... exact same configuration as my TD Studio express which works
Suggestions?
thanks

fgrimmer 553 posts Joined 04/09
10 Aug 2015

Norm, There was a problem fixed in Studio 15.10.00.01 regarding Hive Hadoop support. Can you upgrade Studio to our latest version, 15.10.00.03?

h2oknut 5 posts Joined 05/10
10 Aug 2015

i think i'm there already, here's the zip file i installed: 
TeradataStudio__mac_x86.15.10.00.03
the about help.. 15.10.3.2015..studio base feature , feature
 
thanks

fgrimmer 553 posts Joined 04/09
10 Aug 2015

Norm, Yes, that is the latest version. When it comes to Hadoop, the main difference between Studio and Studio Express is that Studio Express only supports running HiveQL statements. The connection profile management and Data Source Explorer display of Databases and Tables is the same code. Is there some difference in how the HDP VM is configured in the connection profile? Is there any error messages in the Studio error logs? The error logs are in your workspace directory, <Workspace>/.metadata/.log and /.metadata/Teradata/teradata.log.

h2oknut 5 posts Joined 05/10
10 Aug 2015

attached a snibbet of the log..  i can send the whole log if needed.. thanks
MESSAGE Could not connect to HDP 2.1 VM.

!SUBENTRY 1 org.eclipse.datatools.connectivity 2 -1 2015-08-10 18:38:36.487

!MESSAGE Error creating SQL Model Connection connection to HDP 2.1 VM. (Error: Hadoop not configured to run jobs by this application.)

!STACK 0

java.lang.Exception: Hadoop not configured to run jobs by this application.

at com.teradata.datatools.hadoop.hive.connectivity.HiveConnection.openTdchConnection(HiveConnection.java:403)

at com.teradata.datatools.hadoop.hive.connectivity.HiveConnection.createConnection(HiveConnection.j

fgrimmer 553 posts Joined 04/09
10 Aug 2015

Norm, In Studio, if you are going to perform data transfer between Hadoop and Teradata, so choose the TDCH (Teradata Data Transfers) option, you need to install and configure TDCH, Teradata Connector for Hadoop, on your Hadoop system. The error message you attached appears to indicate that TDCH has not been configured. 

h2oknut 5 posts Joined 05/10
10 Aug 2015

agreed.. i was trying to just connect.. BUT, i went back to the configuration and unticked TDCH, and was able to bring up the list of databases... even though i hadn't tried to use TDCH...
so solved..thanks for the quick response

fgrimmer 553 posts Joined 04/09
10 Aug 2015

OK, glad it is working. We try to avoid the errors later on by confirming the connection configuration information.

wills 6 posts Joined 10/07
12 Aug 2015

I am facing similar issue. I have installed TDCH on HDP VM 2.1 From HDP i can transfer to/from HDFS to Teradata and vice versa but when it comes to TD studio Data Transfer is not working. When I click TDCH and enter login details, i am able to ping successfully but No databases are listed in the drop down. As I unclick the TDCH I am able to query HIVE tables. Is there any other setting than installing TDCH on HDFS to make Data Transfer work.
TDCH installed is teradata-connector-1.4.0-hadoop2.x.noarch.rpm
TD Studio is TeradataStudio__win64_x86.15.10.00.03.zip
HDP is Hortonworks Sandbox 2.2
In the studio log file I see 

!SUBENTRY 1 org.eclipse.datatools.connectivity 2 -1 2015-08-12 15:06:24.739

!MESSAGE Error creating Hadoop Hortonworks Connection Factory connection to Local_Hadoop. (Error: Hadoop not configured to run jobs by this application.)

!STACK 0
 

wills 6 posts Joined 10/07
12 Aug 2015

Just to add i uninstalled tdch 1.4.0 and installed teradata-connector-1.3.4-hdp2.2.noarch.rpm
still the same issue.
 

fgrimmer 553 posts Joined 04/09
12 Aug 2015

William, You need to run the configureOozie script on your Hadoop system. The instructions are in our Readme on the Studio Download page.

wills 6 posts Joined 10/07
12 Aug 2015

I executed configureOozie script on my Hadoop system as root. Still the issue persists

Then it shows me

 

[root@sandbox ~]# hadoop fs -cat /teradata/tdch/services.json

{

        "Distribution":"HDP",

        "DistributionVersion":"2.1",

        "TeradataConnectorForHadoopVersion":"1.3.4",

        "WebHCatalog":<ip of hadoop vm>,

        "WebHCatalogPort":50111,

        "WebHDFS":<ip of hadoop vm>,

        "WebHDFSPort":50070,

        "JobTracker":<ip of hadoop vm>,

        "JobTrackerPort":8050,

        "NameNode":<ip of hadoop vm>,

        "NameNodePort":8020,

        "Oozie":<ip of hadoop vm>,

        "OoziePort":11000,

        "HiveClientMetastorePort":9083

}

 

Connection properties that I used in Database connections of TD Studio are...

Check TDCH and JDBC

Tab TDCH

WebHDFS Host Name: <ip of hadoop vm>

WebHDFS Port Number: 50070

WebHDFS User Name: guest   (tried root/hive as well)

 

Tab JDBC

JDBC Host: <ip of hadoop vm>

JDBC Port Number: 10000

JDBC Database: default

JDBC User Name: hive (tried root/hive as well)

JDBC password: left blank

HTTP Transport Mode is left unchecked

 

Tab WebHCat

WebHCat Host Name: <ip of hadoop vm>

WebHCat Port Number: 50111

WebHCat User Name: hue (tried root/hive as well)

wills 6 posts Joined 10/07
12 Aug 2015

The error shows as below. Not sure what I am missing :(
 

java.lang.Exception: Hadoop not configured to run jobs by this application.

at com.teradata.datatools.hadoop.hive.connectivity.HiveConnection.openTdchConnection(HiveConnection.java:403)

at com.teradata.datatools.hadoop.hive.connectivity.HiveConnection.createConnection(HiveConnection.java:203)

at org.eclipse.datatools.connectivity.DriverConnectionBase.internalCreateConnection(DriverConnectionBase.java:105)

at org.eclipse.datatools.connectivity.DriverConnectionBase.open(DriverConnectionBase.java:54)

at com.teradata.datatools.hadoop.hive.connectivity.HiveConnection.open(HiveConnection.java:144)

at com.teradata.datatools.hadoop.hive.connectivity.HivePingFactory.createConnection(HivePingFactory.java:23)

at org.eclipse.datatools.connectivity.internal.ConnectionFactoryProvider.createConnection(ConnectionFactoryProvider.java:83)

at org.eclipse.datatools.connectivity.internal.ConnectionProfile.createConnection(ConnectionProfile.java:359)

at org.eclipse.datatools.connectivity.ui.PingJob.createTestConnection(PingJob.java:76)

at org.eclipse.datatools.connectivity.ui.PingJob.run(PingJob.java:59)

at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)

harristd 6 posts Joined 07/12
13 Aug 2015

Hey Wills, to address the first problem of databases not showing up in the DSE, this is probaby due the limited memory of the VM.  When we have this issue with our test systems, increasing the memory is the common fix to get WebHCatalog (what we query to get DSE information) to respond in a timely manner.
 
As for the Exception you experienced later while pinging the job, I am still looking into this.  This could come from multiple setup issues on the Hadoop side, and tomorrow I will post some instructions to go through and fine the problem for this exception.

harristd 6 posts Joined 07/12
13 Aug 2015

The Exception's most likely cause is that it cannot resolve the 'sandbox.hortonworks.com' returned from the VM (if you are using the sandbox).
 
Try adding the IP and sandbox.hortonworks.com to the Window's hosts file.

wills 6 posts Joined 10/07
13 Aug 2015

Harris, thanks for the insite. I just did not share my IP. Configuration has IPs itself. It will be helpful if you can post the instructions to resolve the exception...
My services.json looks like below
[root@sandbox tpch]# hadoop fs -cat /teradata/tdch/services.json

{

        "Distribution":"HDP",

        "DistributionVersion":"2.1",

        "TeradataConnectorForHadoopVersion":"1.3.4",

        "WebHCatalog":"153.65.226.207",

        "WebHCatalogPort":50111,

        "WebHDFS":"153.65.226.207",

        "WebHDFSPort":50070,

        "JobTracker":"153.65.226.207",

        "JobTrackerPort":8050,

        "NameNode":"153.65.226.207",

        "NameNodePort":8020,

        "Oozie":"153.65.226.207",

        "OoziePort":11000,

        "HiveClientMetastorePort":9083

}

 

harristd 6 posts Joined 07/12
14 Aug 2015

Even though the configuration has the IPs, when we read a file through WebHDFS (to read the configuration file), it almost always returns an internal IP address or 'sandbox.hortonworks.com'.  So without the internal IPs mapped to the external IPs, or the 'sandbox.hortonworks.com' mapped to its IP in the hosts file, the program will be unable to continue.  This is because WebHDFS likes to redirect you to read the file, and they like to send the internal IPs/sandbox.hortonworks.com back.
 
 

wills 6 posts Joined 10/07
17 Aug 2015

Thanks a lot Harris. When I added entry in hosts file, it worked. Thanks for your help. I dont know, I missed the simple step.

harristd 6 posts Joined 07/12
17 Aug 2015

No problem.  I am going to improve the error messaging in our next release for this exact issue, it's probably the most common issue we have with connecting.  I even ran into it during setting up a test system for HDP.

You must sign in to leave a comment.