hdfs-s3backup/debugrecords at dev · melodyyn/hdfs-s3backup · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110

============================
Big test data:   2835 GB

time hadoop jar /home/shawn/s3distcp-TRUNK-SNAPSHOT.jar  -Dmapred.reduce.tasks=200 -Dmapred.map
.tasks=1
--src /hbasebackups/hbasemetann001e/link_pin_index_v1/2014-08-29-06-49-48
--dest s3n://pinterest-namenode-backup/test/fromhdfs/bighbase-with-checksum/

downloaded from s3 to hdfs:   /user/shawn/froms3/bighbase2/


---------
another test dir at s3:  1 GB
    s3n://pinterest-namenode-backup/test/tos3dir/


155 GB:

/hbasebackups/hbaseusermetann001e/board_interests/2014-08-29-09-20-47

s3n://pinterest-namenode-backup/test/fromhdfs/midhbase-with-checksum/

copy back from s3:
/user/shawn/froms3/midhbase-with-checksum/

======================
debug record:

problem 1. when download large dirs (2781GB), one 20 MB file sees missing data in last chunk.
  changed "max inflight part" to 1 and the problem is gone.

last of portion miss
todo:
(1) multipart download using 1 inflight parts

(2). multipart down, via temp local files, 1 inflight parts

(3). same as 2, with 10 inflight parts
  =>  problem fixed.


2.
two files seem to fail checksum check:
2014-08-31 04:09:51,583 INFO
com.amazon.external.elasticmapreduce.s3distcp.CompareFilesRunnable: compare checksum mismatch:
/hbasebackups/hbasegraphnn001e/node_edge_messages/2014-08-29-06-23-24/.archive/node_edge_messages/2de3072034a5b5428df562a946a2aed2/d/c4010ab1c6a1471a817e6c24cddddbb2
s3n://pinterest-namenode-backup/test/fromhdfs/midhbase-with-checksum/.archive/node_edge_messages/2de3072034a5b5428df562a946a2aed2/d/c4010ab1c6a1471a817e6c24cddddbb2

=> Fixed.
Using the newly developed tool (compare-file-checksums),  these two files are identical.


--------
how to run:
time hadoop jar ./s3get-0.1.jar -Ds3copy.chunkSizeMB=16  -Dmapred.reduce.tasks=200
-Dmapred.map.tasks=1 -Ds3copy.checksum=true
--src s3n://pinterest-namenode-backup/test/fromhdfs/bighbase-with-checksum/
--dest /user/shawn/froms3/bighbase2


===========================

package com.pinterest.hdfsbackup.s3copy;

import com.pinterest.hdfsbackup.utils.FilePair;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapred.Mapper;

import java.io.IOException;

/**
 * Created by shawn on 8/26/14.
 */
public class S3GetMapper
    implements Mapper<LongWritable, FilePair, Text, FilePair> {
  private static final Log log = LogFactory.getLog(S3GetMapper.class);
  private static long count;
  protected JobConf conf;

  public void map(LongWritable key, FilePair filePair,
                  OutputCollector<Text, FilePair> collector,
                  Reporter reporter) throws IOException {
    log.info(String.format("input: %d  [%s]", key.get(), filePair.toString()));
    count++;
    collector.collect(new Text(key.toString()), filePair);
  }

  @Override
  public void close() throws IOException {
    log.info("has processed " + count + " filepairs");
  }

  @Override
  public void configure(JobConf entries) {
    this.conf = entries;
  }
}