Wednesday, January 13, 2016

Hadoop Interview Questions

1) How to write the output of PIG script into a JSON file?

2) What is the data injection tool you used? how to increase the limit?
  and how and what exactly the parallel tasks run while doing the data injection?

Tuesday, January 12, 2016

print only the number's from a give String

Question:
   your function should take a string and print only the numbers contained in the string.
i/p : String inputString = "Sekhar124hekkkoer9239939siiipere23+++33!!!!!!";
o/p: [124, 9239939, 23, 33]

Code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
public class OnlyIntegers {

    public static void main(String[] args) {

        String inputString = "Sekhar124hekkkoer9239939siiipere23+++33!!!!!!";

        List numbers = new ArrayList();

        char[] temp = null;

        char[] in = inputString.toCharArray();
        int tempIndex = 0;
        for (char c : in) {
            if (temp == null) {
                temp = new char[10];
            }
            int ascii = (int) c;
            if (ascii > 47 && ascii < 58) {
                temp[tempIndex++] = c;
            } else {
                if ((new String(temp).substring(0,tempIndex).length()) > 0) {
                    numbers.add((new String(temp).substring(0,tempIndex)));
                    temp = null;
                    tempIndex=0;
                }
            }
        }
        System.out.println(numbers);
    }
}

Saturday, January 9, 2016

Difference between Text and String DataTypes in Hadoop

Here I am going to discuss some of the differences between Text and String class in Hadoop.

Text class lies in the package: import org.apache.hadoop.io.*;

Difference 1:
 
Text is not immutable :   String is immutable

Text t = new Text("hadoop");
t.set("BigData")
print "t" --> prints "BigData"


Difference 2 :
Text stores the string in a byte buffer with UTF-8 unicode encoding

Example : Text t = new Tex("hadoop");

will get converted into byte[] array, and then places in to ByteBuffer.

so the string "hadoop" will get stored like this [UTF-CODE(h),UTF-code(a)........ UTF-code(p)]

so this this is the byte[] array representation for string "hadoop"
   [104,97,100,111,111,112]

Why ?

Text uses standard UTF-8 which makes it potentially easier to inter-operate with other tools that
understand UTF-8.


Difference 3 :
CharAt(int index) in string returns the char at specified index.

charAt(int index) in Text returns the Unicode point in the above case it i 100.

Difference 4 : Due to lack of Rich API for manipulating strings in Text many cases we use to
convert it to String.

Difference 5 : Iterating over Text characters is tedious process when compared to string.
Example of iterating over charactes in Text;

Text t = new Text("hadoop");
      ByteBuffer bf = ByteBuffer.wrap(t.getBytes(),0,t.getLength());
 int cp;
 while(bf.hasRemaining()){
  cp = Text.bytesToCodePoint(bf);  
 System.out.print((char) cp);
 }

Similarity 1 :  find in Text equals to indexOf in String.

Text t = new Text("hadoop");
String s = new String("hadoop");

 System.out.println(" >>> "+t.find("o"));
 System.out.println(" >>> "+s.indexOf("o"));


Friday, January 1, 2016

HIVE Tutorial

There are 2 ways we can load data into  the HIVE Tables.
In this tutorial we are loading data into the HIVE managed Table.

This is my sample data with filename 'manageTableSampleData.txt"

10101|Sataya|Hyderabad|23232333
10102|ankur|Chennai  |3452454555
10102|Balaji|Tirupathi|2323121221
10103|chennapa|Chennai  |6767667677
10104|Danunjai|Cochin   |7676666666
10105|Estella|Adilabad |2323333333
10106|Feranandez|Bangaloe |7878777777
10107|Govind|Surat    |2323232323
10108|Hari|Bangalore|2323333333
10109|Kumar|Rajamundry|9029292992
10110|Narendra|Vijayawada|3343233333
10111|Narayana|Vizag     |9029292999
10112|Paramesh|Cochin    |8928282888
10113|Pavan|Kabul     |0202020200
10114|Nag|Kerala    |9292929299
10115|Adithya|Hyderabad|9290202002
10116|vedic|Bombay|8978399338
10117|Vinay|Hyderabad|9393919191
10118|Vamsi|Hyderabad|9292922222
10119|Santosh|Trichy   |9282828288
10120|Sahith|Delhi    |9303030202
10121|Sankar|Bombay   |8938339933
10122|Ramesh|Chennai  |4949494999
10123|Harish|Bangalore|9003399339

First way to load the data into Hive Managed Table from 'localpath'




Second way to load the data into Hive Managed Table from 'hdfs path'

There are 2 steps required in this setup
step 1: First copy the file from LFS(Local File System) into HDFS.
Command :


Step 2 : Now issue the below command to load the data into the HIVE managed Table from the

HDFS input path in to the Hive Managed Table.








Note:
Differences while loading the data from the Local input path and from the HDFS path

  1. There is keyword local  does not exist in the command while loading the data from HDFS path.
  2. The input data loaded from local path still exists after loaded into the Hive managed table also,Where as the input data loaded from the hdfs path into the HIVE Managed table does not exist.

In simple words:

It is CUT - COPY - PASTE Operation in loading from HDFS path,
where as loading from local path  is COPY-PASTE.

OVERWRITE keyword

Overwrite keyword  in the Load Data statement tells Hive to Delete any existing files in
 the directory for the table.

Files in the Name node before the Overwrite command in load data statement.

Command to overwrite the table data.

.

Files in Warehouse Directory after overwrite in load statement.


How to view the log files in HIVE?

The log file located at the directory /tmp/{$USER}/hive.log


How to alter the table column data type in HIVE?

Below the table managedtable4 contains the mobile column data type as int, I would like to change from int to Bigint?









Command to change the column data type.














                             CREATE TABLE AS SELECT (CTAS)
A table can be created by select all fields/some fields/or based on condition from another table.

Note: Only Managed tables can be created with this feature.

Below is the example of creating a table ctas1 from the managedtable5



Thursday, November 26, 2015

HOW TO DEBUG DATANODE NOT RUNNING ISSUES IN HADOOP

Hi,
    In Recent past I am came across below issue.
Problem Point:
   I tried to copy a file from my local file system into hadoop file system(HFDS) with this command.

  bin/hadoop dfs -put /home/user/sample.txt /user/username/.
But  then I got this error:

15/11/26 08:36:57 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/username/Sample.txt could only be replicated to 0 nodes, instead of 1
bla bla ....

I understood that hdfs could not replicate the file.
Check the cluster health with the command  jps
shows me the following output:

user@localhost:~$ jps
5798 SecondaryNameNode
7648 Jps
6050 TaskTracker
5896 JobTracker
5474 NameNode

Observe the above output the Datanode is not running.

Now I visited the hadoop health status url : http://localhost:50070/dfshealth.jsp

Click on the URL "Namenodelogs"

there I checked for the recent data-logs node logs with the name:
hadoop-user-datanode-localhost.log 25542 bytes 
At the semi end of the log file search for the string "STARTUP_MSG: Starting DataNode STARTUP_MSG"

follow this paragraph observed this error: 
"ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode namespaceID = 429859175; datanode namespaceID = 26135836"

then on in local file system changed the current directory to 
> cd /app/hadoop/tmp/dfs/data
> ls 
total 28
drwxr-xr-x 6 sekhar sekhar 4096 Nov 26 08:34 ./
drwxrwxr-x 5 sekhar sekhar 4096 Nov 21 18:57 ../
drwxrwxr-x 2 sekhar sekhar 4096 Nov 23 23:14 blocksBeingWritten/
drwxrwxr-x 2 sekhar sekhar 4096 Nov 23 23:14 current/
drwxrwxr-x 2 sekhar sekhar 4096 Nov 21 18:57 detach/
-rw-rw-r-- 1 sekhar sekhar  157 Nov 21 18:57 storage
drwxrwxr-x 2 sekhar sekhar 4096 Nov 26 07:40 tmp/

> cd current
> ll
sekhar@localhost:/app/hadoop/tmp/dfs/data/current$ ll
total 24
drwxrwxr-x 2 sekhar sekhar 4096 Nov 23 23:14 ./
drwxr-xr-x 6 sekhar sekhar 4096 Nov 26 08:34 ../
-rw-rw-r-- 1 sekhar sekhar    4 Nov 23 23:14 blk_-2585174188513577469
-rw-rw-r-- 1 sekhar sekhar   11 Nov 23 23:14 blk_-2585174188513577469_1002.meta
-rw-rw-r-- 1 sekhar sekhar  193 Nov 23 23:19 dncp_block_verification.log.curr
-rw-rw-r-- 1 sekhar sekhar  154 Nov 26 07:40 VERSION

> vi VERSION

change the namespaceId to namespaceId of the namenode
#Thu Nov 26 08:50:16 IST 2015
namespaceID=26135836 
storageID=DS-1674635271-127.0.0.1-50010-1448112482162
cTime=0
storageType=DATA_NODE
layoutVersion=-41

namespaceID should be changed to namespaceID of the namenodeID simply get this value from log
here in this case it is 429859175.

All set that'it. Just start the cluster now:
> sekhar@localhost:~$ jps
15278 JobTracker
14828 NameNode
15556 Jps
15172 SecondaryNameNode
15001 DataNode
15440 TaskTracker

WOW. Datanode is back.



Sunday, August 10, 2014

Print ODD numbers using ODD Thread and Print EVEN Numbers using Even Thread

Hi,
       we need to print the odd elements using the odd thread and print the even elements using the even thread:

Here i am placing the java code and output. please post a comment if you see any mistake here.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
package com.learning;

public class PrintODDandEvenNumbersUsingThreads {
 
 
 public volatile static  int counter = 0; 
 
 public static void main(String[] args){
  PrintODDandEvenNumbersUsingThreads obj = new PrintODDandEvenNumbersUsingThreads();
   Thread evenThread =  new EvenThreadClass(obj);
   Thread oddThread  =  new OddThreadClass(obj);
   evenThread.start();
   oddThread.start();
 }
 
  static class EvenThreadClass extends Thread{
  PrintODDandEvenNumbersUsingThreads obj = null;
  EvenThreadClass(PrintODDandEvenNumbersUsingThreads obj){
   this.obj=obj;   
  }
 
  public void run(){   
   while(!Thread.currentThread().isInterrupted()){
    synchronized(obj){
     try{
      if(counter%2==0){
       System.out.println("EvenThread-->"+counter);
       counter++;
       Thread.sleep(100);
       obj.notify();
      }else{
       obj.wait();
      }
     }catch(InterruptedException ie){
      ie.printStackTrace();
     }catch(Exception e){
      e.printStackTrace();
     }
   }   
  }
 } 
 }
   static class OddThreadClass extends Thread{
   PrintODDandEvenNumbersUsingThreads obj = null;
   OddThreadClass(PrintODDandEvenNumbersUsingThreads obj){
    this.obj=obj;
   }
  public void run(){   
   while(!Thread.currentThread().isInterrupted()){
    synchronized(obj){
     try{
      if(counter%2!=0){
       System.out.println("OddThread<-- span="">+counter);
       counter++;
       Thread.sleep(100);
       obj.notify();
      }else{
       obj.wait();
      }
     }catch(InterruptedException ie){
      ie.printStackTrace();
     }catch(Exception e){
      e.printStackTrace();
     }
   }
  }
 }
}

}

O/P:
EvenThread-->0
OddThread<--1 p="">EvenThread-->2
OddThread<--3 p="">EvenThread-->4
OddThread<--5 p="">EvenThread-->6
OddThread<--7 p="">EvenThread-->8
OddThread<--9 p="">EvenThread-->10
OddThread<--11 p="">

Monday, October 17, 2011

printing spiral matrix in java


public class SpiralMatrix {

    /**
     * @param args
     */
    public static void main(String[] args) {
        // TODO Auto-generated method stub
        
        int[][] a = new int[][]{ {1,2,3,4},
                                 {5,6,7,8},
                                 {9,10,11,12},
                                 {13,14,15,16}};
        
        printSpiralMatrix(a,4,4);
        
        
        String s = "sekhar";
        s.substring(s.length());
        s.substring(3, 3);
        
        try{
            String myString = "Indian";
            //myString.substring(2,10);
        }catch(Exception e){
            e.printStackTrace();
            
        }
        }

    private static void printSpiralMatrix(int[][] a, int m, int n) {
        
        // m = total number of rows = 4
        // n = total number of columns
        int k = 0; // Row starts
        int l = 0; // column starts
        
        while(k<m && l<n){
            
          //print first row....
            for(int i = l;i<n;i++){
                System.out.println(a[k][i]);
            }
            k++;
            
            for(int i=k;i<m;i++){
                System.out.println(a[i][n-1]);
            }
            
            n--;
            for(int i= n-1; i>=l;--i){
                System.out.println(a[m-1][i]);
            }
            m--;
            
            for(int i=m-1;i>=k;--i){
                System.out.println(a[i][l]);
            }
            l++;
            
            
        }
        
    }

    }

AWS certification question

AWS AWS Hi! this is for questions related to AWS questions. EC2 instances EC2 storage types cold HDD : 1. Defines performance in terms...