PIG- a tool for hadoop Analysis
What Is Pig?
Pig provides an engine for executing data flows in parallel
on Hadoop. It includes a language, Pig Latin, for expressing these data flows.
Pig Latin includes operators for many of the traditional data operations (join,
sort, filter, etc.), as well as the ability for users to develop their own functions
for reading, processing, and writing data.
Pig
runs on Hadoop. It makes use of both the Hadoop Distributed File System, HDFS,
and Hadoop’s processing system, MapReduce. HDFS is a distributed filesystem
that stores files across all of the nodes in a Hadoop cluster. It handles
breaking the files into large blocks and distributing them across different
machines, including making multiple copies of each block so that if any one
machine fails no data is lost. By default, Pig reads input files from HDFS, uses
HDFS to store intermediate data between MapReduce jobs, and writes its output
to HDFS. Pig Latin is a dataflow language. This means it allows users to
describe how data from one or more inputs should be read, processed, and then
stored to one or more outputs in parallel. These data flows can be simple
linear flows like the word count example given previously. They can also be
complex workflows that include points where multiple inputs are joined, and
where data is split into multiple streams to be processed by different
operators. To be mathematically precise, a Pig
Latin
script describes a directed acyclic graph (DAG), where the edges are data flows
and the nodes are operators that process the data.
Pig
is designed for the Hadoop data-processing environment, where schemas are
sometimes unknown or inconsistent. Data may not be properly constrained, and it
is rarely normalized. As a result of these differences, Pig does not require
data to be loaded into tables first. It can operate on data as soon as it is
copied into HDFS.
SQL is the English
of data processing. It has the nice feature that everyone and every tool knows
it, which means the barrier to adoption is very low. Our goal is to make Pig
Latin the native language of parallel data-processing systems such as Hadoop.
It may take some learning, but it will allow users to utilize the power of
Hadoop much more fully.
Difference from SQL
SQL
is a query language. Its focus is to allow users to form queries. It allows
users to describe what question they want answered, but not how they want it
answered. In Pig Latin, on the other hand, the user describes exactly how to
process the input data. Another major difference is that SQL is oriented around
answering one question. When users want to do several data operations together,
they must either write separate queries, storing the intermediate data into
temporary tables, or write it in one query using subqueries inside that query
to do the earlier steps of the processing. However, many SQL users find
subqueries confusing and
difficult
to form properly. Also, using subqueries creates an inside-out design where the
first step in the data pipeline is the innermost query. Pig, however, is
designed with a long series of data operations in mind, so there is no need to
write the data pipeline in an inverted set of subqueries or to worry about
storing data in temporary tables
GRUNT Shell
Grunt
is Pig’s interactive shell. It enables users to enter Pig Latin interactively
and provides a shell for users to interact with HDFS. To enter Grunt, invoke
Pig with no script or command to run.
Typing: pig -x local
will
result in the prompt:
grunt>
Comments
Post a Comment