Question No. 1

Problem Scenario 31 : You have given following two files

1. Content.txt: Contain a huge text file containing space separated words.

2. Remove.txt: Ignore/filter all the words given in this file (Comma Separated).

Write a Spark program which reads the Content.txt file and load as an RDD, remove all the words from a broadcast variables(which is loaded as an RDD of words from Remove.txt). And count the occurrence of the each word and save it as a text file in HDFS.

Content.txt

Hello this is ABCTech.com

This is TechABY.com

Apache Spark Training

This is Spark Learning Session

Spark is faster than MapReduce

Remove.txt

Hello, is, this, the

ASolution :
Step 1 : Create all three files in hdfs in directory called spark2 (We will do using Hue). However, you can first create in local filesystem and then upload it to hdfs
Step 2 : Load the Content.txt file
val content = sc.textFile('spark2/Content.txt') //Load the text file
Step 3 : Load the Remove.txt file
val remove = sc.textFile('spark2/Remove.txt') //Load the text file
Step 4 : Create an RDD from remove, However, there is a possibility each word could have trailing spaces, remove those whitespaces as well.We have used two functions here flatMap, map and trim.
val removeRDD= remove.flatMap(x=> x.splitf',') ).map(word=>word.trim)//Create an array of words
Step 5 : Broadcast the variable, which you want to ignore
val bRemove = sc.broadcast(removeRDD.collect().toList) // It should be array of Strings
Step 6 : Split the content RDD, so we can have Array of String. val words = content.flatMap(line => line.split(' '))
Step 7 : Filter the RDD, so it can have only content which are not present in 'Broadcast Variable'. val filtered = words.filter{case (word) => !bRemove.value.contains(word)}
Step 8 : Create a PairRDD, so we can have (word,1) tuple or PairRDD. val pairRDD = filtered.map(word => (word,1))
Step 9 : Nowdo the word count on PairRDD. val wordCount = pairRDD.reduceByKey(_ + _)
Step 10 : Save the output as a Text file.
wordCount.saveAsTextFile('spark2/result.txt')

BSolution :
Step 1 : Create all three files in hdfs in directory called spark2 (We will do using Hue). However, you can first create in local filesystem and then upload it to hdfs
Step 2 : Load the Content.txt file
val content = sc.textFile('spark2/Content.txt') //Load the text file
Step 3 : Load the Remove.txt file
val remove = sc.textFile('spark2/Remove.txt') //Load the text file
Step 4 : Create an RDD from remove, However, there is a possibility each word could have trailing spaces, remove those whitespaces as well.We have used two functions here flatMap, map and trim.
val removeRDD= remove.flatMap(x=> x.splitf',') ).map(word=>word.trim)//Create an array of words
Step 5 : Filter the RDD, so it can have only content which are not present in 'Broadcast Variable'. val filtered = words.filter{case (word) => !bRemove.value.contains(word)}
Step 6 : Create a PairRDD, so we can have (word,1) tuple or PairRDD. val pairRDD = filtered.map(word => (word,1))
Step 7 : Nowdo the word count on PairRDD. val wordCount = pairRDD.reduceByKey(_ + _)
Step 8 : Save the output as a Text file.
wordCount.saveAsTextFile('spark2/result.txt')

Show Answer

Correct Answer: A

Question No. 2

Problem Scenario 41 : You have been given below code snippet.

val aul = sc.parallelize(List (("a" , Array(1,2)), ("b" , Array(1,2))))

val au2 = sc.parallelize(List (("a" , Array(3)), ("b" , Array(2))))

Apply the Spark method, which will generate below output.

Array[(String, Array[lnt])] = Array((a,Array(1, 2)), (b,Array(1, 2)), (a(Array(3)), (b,Array(2)))

ASolution:
au1.union(au2)

BSolution:
au1.union(au3)

Show Answer

Correct Answer: A

Question No. 3

Problem Scenario 62 : You have been given below code snippet.

val a = sc.parallelize(List("dogM, "tiger", "lion", "cat", "panther", "eagle"), 2)

val b = a.map(x => (x.length, x))

operation1

Write a correct code snippet for operationl which will produce desired output, shown below. Array[(lnt, String)] = Array((3,xdogx), (5,xtigerx), (4,xlionx), (3,xcatx), (7,xpantherx), (5,xeaglex))

ASolution :
b.mapValuesf'x' + _ + 'x').collect
mapValues [Pair] : Takes the values of a RDD that consists of two-component tuples, and applies the provided function to transform each value. Tlien,.it.forms newtwo-componendtuples using the key and the transformed value and stores them in a new RDD.

BSolution :
b.mapValuesf'x' + _ + 'x').collect
mapValues [Pair] : Takes the values of a RDD that consists of two-component tuples, and applies the provided function to transform each value.

Show Answer

Correct Answer: A

Question No. 4

Problem Scenario 36 : You have been given a file named spark8/data.csv (type,name).

data.csv

1,Lokesh

2,Bhupesh

2,Amit

2,Ratan

2,Dinesh

1,Pavan

1,Tejas

2,Sheela

1,Kumar

1,Venkat

1. Load this file from hdfs and save it back as (id, (all names of same type)) in results directory. However, make sure while saving it should be

ASolution :
Step 1 : Create file in hdfs (We will do using Hue). However, you can first create in local filesystem and then upload it to hdfs.
Step 2 : Load data.csv file from hdfs and create PairRDDs
val name = sc.textFile('spark8/data.csv')
val namePairRDD = name.map(x=> (x.split(',')(0),x.split(',')(1)))
Step 3 : Now combine the rdd by key.
val combinedOutput = namePairRDD.combineByKey(List(_), (x:List[String], y:String) => y :: x, (x:List[String], y:List[String]) => x ::: y)
Step 4 : Save the output as a Text file and output must be written in a single file. :ombinedOutput.repartition(1).saveAsTextFile('spark8/result.txt')

BSolution :
Step 1 : Create file in hdfs (We will do using Hue). However, you can first create in local filesystem and then upload it to hdfs.
Step 2 : Load data.csv file from hdfs and create PairRDDs
val name = sc.textFile('spark8/data.csv')
val namePairRDD = name.map(x=> (x.split(',')(0),x.split(',')(1)))
Step 3 : Now swap namePairRDD RDD.
val swapped = namePairRDD.map(item => item.swap)
Step 4 : Save the output as a Text file and output must be written in a single file. :ombinedOutput.repartition(1).saveAsTextFile('spark8/result.txt')

CSolution :
Step 1 : Create file in hdfs (We will do using Hue). However, you can first create in local filesystem and then upload it to hdfs.
Step 2 : Load data.csv file from hdfs and create PairRDDs
val name = sc.textFile('spark8/data.csv')
val namePairRDD = name.map(x=> (x.split(',')(0),x.split(',')(1)))
Step 3 : Now swap namePairRDD RDD.
val swapped = namePairRDD.map(item => item.swap)
Step 4 : Now combine the rdd by key.
val combinedOutput = namePairRDD.combineByKey(List(_), (x:List[String], y:String) => y :: x, (x:List[String], y:List[String]) => x ::: y)
Step 5 : Save the output as a Text file and output must be written in a single file. :ombinedOutput.repartition(1).saveAsTextFile('spark8/result.txt')

Show Answer

Correct Answer: C

Question No. 5

Problem Scenario 12 : You have been given following mysql database details as well as other info.

user=retail_dba

password=cloudera

database=retail_db

jdbc URL = jdbc:mysql://quickstart:3306/retail_db

Please accomplish following.

1. Create a table in retailedb with following definition.

CREATE table departments_new (department_id int(11), department_name varchar(45), created_date T1MESTAMP DEFAULT NOW());

2. Now isert records from departments table to departments_new

3. Now import data from departments_new table to hdfs.

4. Insert following 5 records in departmentsnew table. Insert into departments_new values(110, "Civil" , null); Insert into departments_new values(111, "Mechanical" , null); Insert into departments_new values(112, "Automobile" , null); Insert into departments_new values(113, "Pharma" , null);

Insert into departments_new values(114, "Social Engineering" , null);

5. Now do the incremental import based on created_date column.

ASolution :
Step 1 : Login to musql db
mysql --user=retail_dba -password=cloudera
show databases;
use retail db; show tables;
Step 2 : Create a table as given in problem statement.
CREATE table departments_new (department_id int(11), department_name varchar(45), createddate T1MESTAMP DEFAULT NOW());
show tables;
Step 3 : isert records from departments table to departments_new insert into departments_new select a.', null from departments a;
Step 4 : Import data from departments new table to hdfs.
sqoop import \
-connect jdbc:mysql://quickstart:330G/retail_db \
~username=retail_dba \
-password=cloudera \
-table departments_new\
--target-dir /user/cloudera/departments_new \
--split-by departments
Stpe 5 : Check the imported data.
hdfs dfs -cat /user/cloudera/departmentsnew/part'
Step 6 : Insert following 5 records in departmentsnew table.
Insert into departments_new values(110, 'Civil' , null);
Insert into departments_new values(111, 'Mechanical' , null);
Insert into departments_new values(112, 'Automobile' , null);
Insert into departments_new values(113, 'Pharma' , null);
Insert into departments_new values(114, 'Social Engineering' , null);
commit;
Stpe 7 : Import incremetal data based on created_date column.
sqoop import \
-connect jdbc:mysql://quickstart:330G/retaiI_db \
-username=retail_dba \
-password=cloudera \
--table departments_new\
-target-dir /user/cloudera/departments_new \
-append \
-check-column created_date \
-incremental lastmodified \
-split-by departments \
-last-value '2016-01-30 12:07:37.0'
Step 8 : Check the imported value.
hdfs dfs -cat /user/cloudera/departmentsnew/part'

BSolution :
Step 1 : Login to musql db
mysql --user=retail_dba -password=cloudera
show databases;
use retail db; show tables;
Step 2 : Create a table as given in problem statement.
CREATE table departments_new (department_id int(11), department_name varchar(45), createddate T1MESTAMP DEFAULT NOW());
show tables;
Step 3 : isert records from departments table to departments_new insert into departments_new select a.', null from departments a;
Step 4 : Import data from departments new table to hdfs.
sqoop import \
-connect jdbc:mysql://quickstart:330G/retail_db \
~username=retail_dba \
-password=cloudera \
-table departments_new\
--target-dir /user/cloudera/departments_new \
--split-by departments
Stpe 5 : Check the imported data.
hdfs dfs -cat /user/cloudera/departmentsnew/part'
Step 6 : Insert following 5 records in departmentsnew table.
Insert into departments_new values(110, 'Civil' , null);
Insert into departments_new values(114, 'Social Engineering' , null);
commit;
Stpe 7 : Import incremetal data based on created_date column.
sqoop import \
-connect jdbc:mysql://quickstart:330G/retaiI_db \
-username=retail_dba \
-password=cloudera \
--table departments_new\
-target-dir /user/cloudera/departments_new \
-append \
-check-column created_date \
-incremental lastmodified \
-split-by departments \
-last-value '2016-01-30 12:07:37.0'
Step 8 : Check the imported value.
hdfs dfs -cat /user/cloudera/departmentsnew/part'

Show Answer

Correct Answer: A

Free Cloudera CCA175 Exam Actual Questions

The questions for CCA175 were last updated On Nov 20, 2024