Windows上にhadoop2.2.0を配置する方法

2014年7月10日

1.core-site.xmlを編集
<?xml version="1.0″ encoding="UTF-8″?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!–
Licensed under the Apache License, Version 2.0 (the “License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

<!– Put site-specific property overrides in this file. –>

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>

2.hdfs-site.xmlを編集
<?xml version="1.0″ encoding="UTF-8″?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!–
Licensed under the Apache License, Version 2.0 (the “License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

<!– Put site-specific property overrides in this file. –>

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/hadoop/data/dfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/hadoop/data/dfs/datanode</value>
</property>
</configuration>

3.yarn-site.xmlを修正
<?xml version="1.0″?>
<!–
Licensed under the Apache License, Version 2.0 (the “License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an “AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
–>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

4.mapred-site.xmlを修正
<?xml version="1.0″?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!–
Licensed under the Apache License, Version 2.0 (the “License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

<!– Put site-specific property overrides in this file. –>

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

C:\Users\abhijitg>cd c:\hadoop\bin

c:\hadoop\bin>hdfs namenode -format
14/11/03 18:07:47 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ABHIJITG/x.x.x.x
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.2.0
STARTUP_MSG: classpath = <classpath jars here>
STARTUP_MSG: build = Unknown -r Unknown; compiled by ABHIJITG on 2014-11-01T14:42Z
STARTUP_MSG: java = 1.7.0_03
************************************************************/
Formatting using clusterid: CID-1af0bd9f-efee-4d4e-9f03-a0032c22e5eb
14/11/03 18:07:48 INFO namenode.HostFileManager: read includes:
HostSet(
)
14/11/03 18:07:48 INFO namenode.HostFileManager: read excludes:
HostSet(
)
14/11/03 18:07:48 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
14/11/03 18:07:48 INFO util.GSet: Computing capacity for map BlocksMap
14/11/03 18:07:48 INFO util.GSet: VM type = 64-bit
14/11/03 18:07:48 INFO util.GSet: 2.0% max memory = 888.9 MB
14/11/03 18:07:48 INFO util.GSet: capacity = 2^21 = 2097152 entries
14/11/03 18:07:48 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
14/11/03 18:07:48 INFO blockmanagement.BlockManager: defaultReplication = 1
14/11/03 18:07:48 INFO blockmanagement.BlockManager: maxReplication = 512
14/11/03 18:07:48 INFO blockmanagement.BlockManager: minReplication = 1
14/11/03 18:07:48 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
14/11/03 18:07:48 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false
14/11/03 18:07:48 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
14/11/03 18:07:48 INFO blockmanagement.BlockManager: encryptDataTransfer = false
14/11/03 18:07:48 INFO namenode.FSNamesystem: fsOwner = ABHIJITG (auth:SIMPLE)
14/11/03 18:07:48 INFO namenode.FSNamesystem: supergroup = supergroup
14/11/03 18:07:48 INFO namenode.FSNamesystem: isPermissionEnabled = true
14/11/03 18:07:48 INFO namenode.FSNamesystem: HA Enabled: false
14/11/03 18:07:48 INFO namenode.FSNamesystem: Append Enabled: true
14/11/03 18:07:49 INFO util.GSet: Computing capacity for map INodeMap
14/11/03 18:07:49 INFO util.GSet: VM type = 64-bit
14/11/03 18:07:49 INFO util.GSet: 1.0% max memory = 888.9 MB
14/11/03 18:07:49 INFO util.GSet: capacity = 2^20 = 1048576 entries
14/11/03 18:07:49 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/11/03 18:07:49 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
14/11/03 18:07:49 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
14/11/03 18:07:49 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
14/11/03 18:07:49 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
14/11/03 18:07:49 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time
is 600000 millis
14/11/03 18:07:49 INFO util.GSet: Computing capacity for map Namenode Retry Cache
14/11/03 18:07:49 INFO util.GSet: VM type = 64-bit
14/11/03 18:07:49 INFO util.GSet: 0.029999999329447746% max memory = 888.9 MB
14/11/03 18:07:49 INFO util.GSet: capacity = 2^15 = 32768 entries
14/11/03 18:07:49 INFO common.Storage: Storage directory \hadoop\data\dfs\namenode has been successfully formatted.
14/11/03 18:07:49 INFO namenode.FSImage: Saving image file \hadoop\data\dfs\namenode\current\fsimage.ckpt_00000000000000
00000 using no compression
14/11/03 18:07:49 INFO namenode.FSImage: Image file \hadoop\data\dfs\namenode\current\fsimage.ckpt_0000000000000000000 o
f size 200 bytes saved in 0 seconds.
14/11/03 18:07:49 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
14/11/03 18:07:49 INFO util.ExitUtil: Exiting with status 0
14/11/03 18:07:49 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ABHIJITG/x.x.x.x
************************************************************/

Source

Posted by arkgame