【翻译】Apache Hadoop MapReduce

原文：http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html

综述

Hadoop MapReduce是一个便于开发并行处理海量数据（TB级）的应用的软件框架，该框架在由普通pc机组成的大规模集群（上千台节点）上实现了可靠性及容错。

一个MapReduce任务（job）通常会将输入数据集分片，这一工作是由map任务完全并行的完成的。框架整理map的运行结果，作为reduce任务的输入。通常数据的输入输出都是在文件系统上完成的。MapReduce框架负责调度、监控及重做失败任务的工作。

通常来讲计算节点和存储节点是一样的，也就是说，MapReduce框架及HDFS运行在同一个节点集合。这种配置使得框架可以在数据已就绪的节点集群内高效的调度任务，这样在集群内获得了非常大的带宽。

MapReduce框架包含一个资源管理器（ResourceManager ），每个节点上的NodeManager及每个应用上的MRAppMaster。

应用至少要指定输入输出位置，并通过适当的接口实现及抽象类来提供map及reduce的功能。

Hadoop的job-client提交前述任务（这个任务可以是jar，也可以是其他可执行的文件），并配置到资源管理器。资源管理器将软件及配置分发给从机，调度并监控任务，向job-client提供状态及诊断信息。

尽管Hadoop框架是用Java实现的，MapReduce应用不限定使用Java编写。

Hadoop Streaming使得用户可以创造及运行任意的可执行程序作为mapper或者reducer。

Hadoop Pipes是兼容SWIG的C++ API，用于实现MapReduce应用。

输入输出

MapReduce框架运行在键值对（<key, value>）上，也就是说，MapReduce框架将任务的输入视为一个键值对的集合，产生新的键值对集合作为任务输出。key及value的类需要能够被框架序列化，因此必须实现Hadoop的writable接口（org.apache.hadoop.io ）。此外，key类需要实现WritableComparable接口（org.apache.hadoop.io）来促进框架的排序。

一个MapReduce任务的输入输出类型示例：

(input) <k1, v1> -> map -> <k2, v2> -> combine -> <k2, v2> -> reduce -> <k3, v3> (output)

接下来玩例子：

MapReduce工作方式的小栗子：词频统计（wordcount）

Java 源代码

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

  public static class TokenizerMapper
       extends Mapper<Object, Text, Text, IntWritable>{

    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(Object key, Text value, Context context
                    ) throws IOException, InterruptedException {
      StringTokenizer itr = new StringTokenizer(value.toString());
      while (itr.hasMoreTokens()) {
        word.set(itr.nextToken());
        context.write(word, one);
      }
    }
  }

  public static class IntSumReducer
       extends Reducer<Text,IntWritable,Text,IntWritable> {
    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values,
                       Context context
                       ) throws IOException, InterruptedException {
      int sum = 0;
      for (IntWritable val : values) {
        sum += val.get();
      }
      result.set(sum);
      context.write(key, result);
    }
  }

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    Job job = Job.getInstance(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

import java.io.IOException;

import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

public static class TokenizerMapper

extends Mapper<Object, Text, Text, IntWritable>{

private final static IntWritable one = new IntWritable(1);

private Text word = new Text();

public void map(Object key, Text value, Context context

) throws IOException, InterruptedException {

StringTokenizer itr = new StringTokenizer(value.toString());

while (itr.hasMoreTokens()) {

word.set(itr.nextToken());

context.write(word, one);

}

public static class IntSumReducer

extends Reducer<Text,IntWritable,Text,IntWritable> {

private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values,

Context context

) throws IOException, InterruptedException {

int sum = 0;

for (IntWritable val : values) {

sum += val.get();

}

result.set(sum);

context.write(key, result);

}

public static void main(String[] args) throws Exception {

Configuration conf = new Configuration();

Job job = Job.getInstance(conf, "word count");

job.setJarByClass(WordCount.class);

job.setMapperClass(TokenizerMapper.class);

job.setCombinerClass(IntSumReducer.class);

job.setReducerClass(IntSumReducer.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(IntWritable.class);

FileInputFormat.addInputPath(job, new Path(args[0]));

FileOutputFormat.setOutputPath(job, new Path(args[1]));

System.exit(job.waitForCompletion(true) ? 0 : 1);

}

复制上述代码到文件：

注意！文件夹及后面的jar需要让hdfs用户有7的权限！！！否则后面执行出错。

[root@test1 ~]# mkdir -p /tmp/Class4_1/
[root@test1 ~]# vim /tmp/Class4_1/WordCount.java

1 2	[root@test1 ~]# mkdir -p /tmp/Class4_1/ [root@test1 ~]# vim /tmp/Class4_1/WordCount.java

使用

假设环境变量设置如下（主要添加了后两条，不加会有classNotFound的错误）

export JAVA_HOME=/usr/java/default
export PATH=$JAVA_HOME/bin:$PATH
export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar

export HADOOP_HOME=/usr/lib/hadoop
export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar:$HADOOP_HOME

export JAVA_HOME=/usr/java/default

export PATH=$JAVA_HOME/bin:$PATH

export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar

export HADOOP_HOME=/usr/lib/hadoop

export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar:$HADOOP_HOME

编译前述WordCount.java文件并生成jar。

cdh hadoop默认lib目录：

/var/lib/

[root@test1 ~]# hadoop com.sun.tools.javac.Main /tmp/Class4_1/WordCount.java
[root@test1 ~]# cd /tmp/Class4_1
[root@test1 Class4_1]# ls
WordCount.class WordCount.java
WordCount$IntSumReducer.class WordCount$TokenizerMapper.class
[root@test1 Class4_1]# jar cf wc.jar WordCount*.class
[root@test1 Class4_1]# ls
wc.jar WordCount$IntSumReducer.class WordCount$TokenizerMapper.class
WordCount.class WordCount.java

[root@test1 ~]# hadoop com.sun.tools.javac.Main /tmp/Class4_1/WordCount.java

[root@test1 ~]# cd /tmp/Class4_1

[root@test1 Class4_1]# ls

WordCount.class WordCount.java

WordCount$IntSumReducer.class WordCount$TokenizerMapper.class

[root@test1 Class4_1]# jar cf wc.jar WordCount*.class

[root@test1 Class4_1]# ls

wc.jar WordCount$IntSumReducer.class WordCount$TokenizerMapper.class

WordCount.class WordCount.java

假设输入输出目录如下

/user/class_example/4_1/wordcount/input

/user/class_example/4_1/wordcount/output

[root@test1 ~]# hadoop fs -mkdir /user/class_example/
mkdir: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
[root@test1 ~]# su hdfs
bash-4.1$ hadoop fs -mkdir /user/class_example/
bash-4.1$ hadoop fs -mkdir /user/class_example/4_1/
bash-4.1$ hadoop fs -mkdir /user/class_example/4_1/wordcount/
bash-4.1$ hadoop fs -mkdir /user/class_example/4_1/wordcount/input/
bash-4.1$ hadoop fs -mkdir /user/class_example/4_1/wordcount/output/

[root@test1 ~]# hadoop fs -mkdir /user/class_example/

mkdir: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x

[root@test1 ~]# su hdfs

bash-4.1$ hadoop fs -mkdir /user/class_example/

bash-4.1$ hadoop fs -mkdir /user/class_example/4_1/

bash-4.1$ hadoop fs -mkdir /user/class_example/4_1/wordcount/

bash-4.1$ hadoop fs -mkdir /user/class_example/4_1/wordcount/input/

bash-4.1$ hadoop fs -mkdir /user/class_example/4_1/wordcount/output/

在本地生成输入文件

[root@test1 ~]# vim /class_example/4_1/wordcount/input/file01
Hello World Bye World
[root@test1 ~]# vim /class_example/4_1/wordcount/input/file02
Hello Hadoop Goodbye Hadoop

[root@test1 ~]# vim /class_example/4_1/wordcount/input/file01

Hello World Bye World

[root@test1 ~]# vim /class_example/4_1/wordcount/input/file02

Hello Hadoop Goodbye Hadoop

导入到hdfs

bash-4.1$ hdfs dfs -put /class_example/4_1/wordcount/input/file01 /user/class_example/4_1/wordcount/input/file01 
bash-4.1$ hdfs dfs -put /class_example/4_1/wordcount/input/file02 /user/class_example/4_1/wordcount/input/file02

1 2	bash-4.1$ hdfs dfs -put /class_example/4_1/wordcount/input/file01 /user/class_example/4_1/wordcount/input/file01 bash-4.1$ hdfs dfs -put /class_example/4_1/wordcount/input/file02 /user/class_example/4_1/wordcount/input/file02

（在jar的目录）跑一把前面生成的MapReduce程序的jar

bash-4.1$ hadoop jar wc.jar WordCount /user/class_example/4_1/wordcount/input/ /user/class_example/4_1/wordcount/output/

15/01/15 19:45:07 INFO client.RMProxy: Connecting to ResourceManager at hdp01/172.19.17.231:8032
15/01/15 19:45:08 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/01/15 19:45:08 INFO input.FileInputFormat: Total input paths to process : 2
15/01/15 19:45:09 INFO mapreduce.JobSubmitter: number of splits:2
15/01/15 19:45:09 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1419153136605_0005
15/01/15 19:45:09 INFO impl.YarnClientImpl: Submitted application application_1419153136605_0005
15/01/15 19:45:09 INFO mapreduce.Job: The url to track the job: http://hdp01:8088/proxy/application_1419153136605_0005/
15/01/15 19:45:09 INFO mapreduce.Job: Running job: job_1419153136605_0005
15/01/15 19:45:23 INFO mapreduce.Job: Job job_1419153136605_0005 running in uber mode : false
15/01/15 19:45:23 INFO mapreduce.Job: map 0% reduce 0%
15/01/15 19:45:31 INFO mapreduce.Job: map 100% reduce 0%
15/01/15 19:45:42 INFO mapreduce.Job: map 100% reduce 15%
15/01/15 19:45:48 INFO mapreduce.Job: map 100% reduce 28%
15/01/15 19:45:49 INFO mapreduce.Job: map 100% reduce 31%
15/01/15 19:45:54 INFO mapreduce.Job: map 100% reduce 38%
15/01/15 19:45:55 INFO mapreduce.Job: map 100% reduce 43%
15/01/15 19:45:56 INFO mapreduce.Job: map 100% reduce 46%
15/01/15 19:46:01 INFO mapreduce.Job: map 100% reduce 53%
15/01/15 19:46:02 INFO mapreduce.Job: map 100% reduce 61%
15/01/15 19:46:07 INFO mapreduce.Job: map 100% reduce 64%
15/01/15 19:46:08 INFO mapreduce.Job: map 100% reduce 72%
15/01/15 19:46:09 INFO mapreduce.Job: map 100% reduce 75%
15/01/15 19:46:10 INFO mapreduce.Job: map 100% reduce 76%
15/01/15 19:46:13 INFO mapreduce.Job: map 100% reduce 78%
15/01/15 19:46:14 INFO mapreduce.Job: map 100% reduce 79%
15/01/15 19:46:15 INFO mapreduce.Job: map 100% reduce 88%
15/01/15 19:46:16 INFO mapreduce.Job: map 100% reduce 90%
15/01/15 19:46:17 INFO mapreduce.Job: map 100% reduce 92%
15/01/15 19:46:20 INFO mapreduce.Job: map 100% reduce 93%
15/01/15 19:46:21 INFO mapreduce.Job: map 100% reduce 96%
15/01/15 19:46:22 INFO mapreduce.Job: map 100% reduce 100%
15/01/15 19:46:24 INFO mapreduce.Job: Job job_1419153136605_0005 completed successfully
15/01/15 19:46:25 INFO mapreduce.Job: Counters: 49
 File System Counters
 FILE: Number of bytes read=1513
 FILE: Number of bytes written=7837484
 FILE: Number of read operations=0
 FILE: Number of large read operations=0
 FILE: Number of write operations=0
 HDFS: Number of bytes read=306
 HDFS: Number of bytes written=41
 HDFS: Number of read operations=222
 HDFS: Number of large read operations=0
 HDFS: Number of write operations=144
 Job Counters 
 Launched map tasks=2
 Launched reduce tasks=72
 Data-local map tasks=2
 Total time spent by all maps in occupied slots (ms)=10984
 Total time spent by all reduces in occupied slots (ms)=388216
 Total time spent by all map tasks (ms)=10984
 Total time spent by all reduce tasks (ms)=388216
 Total vcore-seconds taken by all map tasks=10984
 Total vcore-seconds taken by all reduce tasks=388216
 Total megabyte-seconds taken by all map tasks=11247616
 Total megabyte-seconds taken by all reduce tasks=397533184
 Map-Reduce Framework
 Map input records=2
 Map output records=8
 Map output bytes=82
 Map output materialized bytes=2377
 Input split bytes=256
 Combine input records=8
 Combine output records=6
 Reduce input groups=5
 Reduce shuffle bytes=2377
 Reduce input records=6
 Reduce output records=5
 Spilled Records=12
 Shuffled Maps =144
 Failed Shuffles=0
 Merged Map outputs=144
 GC time elapsed (ms)=7568
 CPU time spent (ms)=113830
 Physical memory (bytes) snapshot=24281419776
 Virtual memory (bytes) snapshot=123326947328
 Total committed heap usage (bytes)=58622738432
 Shuffle Errors
 BAD_ID=0
 CONNECTION=0
 IO_ERROR=0
 WRONG_LENGTH=0
 WRONG_MAP=0
 WRONG_REDUCE=0
 File Input Format Counters 
 Bytes Read=50
 File Output Format Counters 
 Bytes Written=41

bash-4.1$ hadoop jar wc.jar WordCount /user/class_example/4_1/wordcount/input/ /user/class_example/4_1/wordcount/output/

15/01/15 19:45:07 INFO client.RMProxy: Connecting to ResourceManager at hdp01/172.19.17.231:8032

15/01/15 19:45:08 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.

15/01/15 19:45:08 INFO input.FileInputFormat: Total input paths to process : 2

15/01/15 19:45:09 INFO mapreduce.JobSubmitter: number of splits:2

15/01/15 19:45:09 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1419153136605_0005

15/01/15 19:45:09 INFO impl.YarnClientImpl: Submitted application application_1419153136605_0005

15/01/15 19:45:09 INFO mapreduce.Job: The url to track the job: http://hdp01:8088/proxy/application_1419153136605_0005/

15/01/15 19:45:09 INFO mapreduce.Job: Running job: job_1419153136605_0005

15/01/15 19:45:23 INFO mapreduce.Job: Job job_1419153136605_0005 running in uber mode : false

15/01/15 19:45:23 INFO mapreduce.Job: map 0% reduce 0%

15/01/15 19:45:31 INFO mapreduce.Job: map 100% reduce 0%

15/01/15 19:45:42 INFO mapreduce.Job: map 100% reduce 15%

15/01/15 19:45:48 INFO mapreduce.Job: map 100% reduce 28%

15/01/15 19:45:49 INFO mapreduce.Job: map 100% reduce 31%

15/01/15 19:45:54 INFO mapreduce.Job: map 100% reduce 38%

15/01/15 19:45:55 INFO mapreduce.Job: map 100% reduce 43%

15/01/15 19:45:56 INFO mapreduce.Job: map 100% reduce 46%

15/01/15 19:46:01 INFO mapreduce.Job: map 100% reduce 53%

15/01/15 19:46:02 INFO mapreduce.Job: map 100% reduce 61%

15/01/15 19:46:07 INFO mapreduce.Job: map 100% reduce 64%

15/01/15 19:46:08 INFO mapreduce.Job: map 100% reduce 72%

15/01/15 19:46:09 INFO mapreduce.Job: map 100% reduce 75%

15/01/15 19:46:10 INFO mapreduce.Job: map 100% reduce 76%

15/01/15 19:46:13 INFO mapreduce.Job: map 100% reduce 78%

15/01/15 19:46:14 INFO mapreduce.Job: map 100% reduce 79%

15/01/15 19:46:15 INFO mapreduce.Job: map 100% reduce 88%

15/01/15 19:46:16 INFO mapreduce.Job: map 100% reduce 90%

15/01/15 19:46:17 INFO mapreduce.Job: map 100% reduce 92%

15/01/15 19:46:20 INFO mapreduce.Job: map 100% reduce 93%

15/01/15 19:46:21 INFO mapreduce.Job: map 100% reduce 96%

15/01/15 19:46:22 INFO mapreduce.Job: map 100% reduce 100%

15/01/15 19:46:24 INFO mapreduce.Job: Job job_1419153136605_0005 completed successfully

15/01/15 19:46:25 INFO mapreduce.Job: Counters: 49

File System Counters

FILE: Number of bytes read=1513

FILE: Number of bytes written=7837484

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=306

HDFS: Number of bytes written=41

HDFS: Number of read operations=222

HDFS: Number of large read operations=0

HDFS: Number of write operations=144

Job Counters

Launched map tasks=2

Launched reduce tasks=72

Data-local map tasks=2

Total time spent by all maps in occupied slots (ms)=10984

Total time spent by all reduces in occupied slots (ms)=388216

Total time spent by all map tasks (ms)=10984

Total time spent by all reduce tasks (ms)=388216

Total vcore-seconds taken by all map tasks=10984

Total vcore-seconds taken by all reduce tasks=388216

Total megabyte-seconds taken by all map tasks=11247616

Total megabyte-seconds taken by all reduce tasks=397533184

Map-Reduce Framework

Map input records=2

Map output records=8

Map output bytes=82

Map output materialized bytes=2377

Input split bytes=256

Combine input records=8

Combine output records=6

Reduce input groups=5

Reduce shuffle bytes=2377

Reduce input records=6

Reduce output records=5

Spilled Records=12

Shuffled Maps =144

Failed Shuffles=0

Merged Map outputs=144

GC time elapsed (ms)=7568

CPU time spent (ms)=113830

Physical memory (bytes) snapshot=24281419776

Virtual memory (bytes) snapshot=123326947328

Total committed heap usage (bytes)=58622738432

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=50

File Output Format Counters

Bytes Written=41

参观一下输出：

bash-4.1$ hdfs dfs -ls /user/class_example/4_1/wordcount/output
Found 73 items
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/_SUCCESS
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00000
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00001
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00002
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00003
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00004
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00005
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00006
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00007
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00008
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00009
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00010
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00011
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00012
-rw-r--r--   3 hdfs supergroup          6 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00013
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00014
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00015
-rw-r--r--   3 hdfs supergroup         10 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00016
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00017
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00018
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00019
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00020
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00021
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00022
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00023
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00024
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00025
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00026
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00027
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00028
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00029
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00030
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00031
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00032
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00033
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00034
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00035
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00036
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00037
-rw-r--r--   3 hdfs supergroup          9 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00038
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00039
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00040
-rw-r--r--   3 hdfs supergroup          8 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00041
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00042
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00043
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00044
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00045
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00046
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00047
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00048
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00049
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00050
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00051
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00052
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00053
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00054
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00055
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00056
-rw-r--r--   3 hdfs supergroup          8 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00057
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00058
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00059
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00060
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00061
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00062
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00063
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00064
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00065
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00066
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00067
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00068
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00069
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00070
-rw-r--r--   3 hdfs supergroup          0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00071

bash-4.1$ hdfs dfs -ls /user/class_example/4_1/wordcount/output

Found 73 items

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/_SUCCESS

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00000

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00001

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00002

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00003

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00004

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00005

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00006

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00007

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00008

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00009

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00010

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00011

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00012

-rw-r--r-- 3 hdfs supergroup 6 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00013

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00014

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00015

-rw-r--r-- 3 hdfs supergroup 10 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00016

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00017

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00018

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00019

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00020

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00021

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00022

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00023

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00024

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00025

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00026

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00027

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00028

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00029

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00030

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00031

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00032

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00033

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00034

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00035

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00036

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:45 /user/class_example/4_1/wordcount/output/part-r-00037

-rw-r--r-- 3 hdfs supergroup 9 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00038

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00039

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00040

-rw-r--r-- 3 hdfs supergroup 8 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00041

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00042

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00043

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00044

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00045

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00046

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00047

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00048

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00049

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00050

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00051

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00052

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00053

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00054

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00055

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00056

-rw-r--r-- 3 hdfs supergroup 8 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00057

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00058

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00059

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00060

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00061

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00062

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00063

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00064

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00065

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00066

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00067

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00068

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00069

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00070

-rw-r--r-- 3 hdfs supergroup 0 2015-01-15 19:46 /user/class_example/4_1/wordcount/output/part-r-00071

【翻译】Apache Hadoop MapReduce

综述

输入输出

接下来玩例子：

Java 源代码

使用

Leave a Comment 取消回复