香蕉与打火机 | 香蕉与打火机

About 香蕉与打火机

李小蕉！

【翻译】CDH 的Cloudera Manager免费与收费版的对比表 = =

翻译：http://www.cloudera.com/content/cloudera/en/products-and-services/cloudera-enterprise/cloudera-manager/cloudera-manager-features.html

CDH 特性		免费版	付费版
Deployment, Configuration & Management	系统管理
Automated Deployment & Hadoop Readiness Checks	自动化部署及快速检查	•	•
Install the complete CDH stack in minutes and ensure optimal settings	安装完整的CDH及优化配置
Service Management	服务管理	•	•
Configure and manage all CDH services, including Impala and Search, from a central interface	提供统一的界面管理与配置全部的CDH服务，包括cloudera impala及cloudera search
Security Management	安全	•	•
Configure and manage security across the cluster – including Kerberos authentication and role-based (administrator and read-only) administration	跨群集的安全管理与配置（包括Kerberos认证及基于角色的管理）
Resource Management	资源管理	•	•
Allocate cluster resources by workload or by user/group/application to eliminate contention and ensure Quality-of-Service (QoS)	根据工作量分配资源，或根据/user/group/application文件消除争用，保证QoS
High Availability	HA	•	•
Easily configure and manage High Availability for various services like HDFS, MapReduce, Oozie, YARN, HBase	为多种服务配置HA：HDFS,MapReduce,Oozie,YARN,Hbase
Client Configuration Management	管理客户端配置	•	•
Centrally configure all client access to the cluster	集中配置连接到群集的客户端
Node Templating	节点模板	•	•
Easily deploy and expand heterogeneous clusters by creating templates for node roles	通过为节点角色创造模板，来部署和扩展不同的群集
Comprehensive Workflows	全面的工作流	•	•
Perform end-to-end tasks such as start/stop/restart clusters, services and roles, add/delete hosts, decommission nodes etc.	执行端到端的任务，如群集、服务、角色级别的启停，增删主机，解除节点等。
Multi-Cluster Management	多群集管理	•	•
Manage multiple CDH clusters from a single instance of Cloudera Manager	一个Manager管理多个CDH群集

Monitor
Service, Host & Activity Monitoring	服务、主机、活动的监控	•	•
Get a consolidated, real-time view of the state of all services, hosts and activities running in the cluster	对服务、主机、活动的统一的实时的监控
Events & Alerts	事件和警报	•	•
Create, aggregate and receive alerts on relevant Hadoop events pertaining to system health, log messages, user actions and activities Set thresholds and create custom alerts for metrics collected by CM	创建、合计、接收Hadoop相关的系统健康、日志信息、用户动作和活动的警报。设置阈值并创建用户警报。
Diagnose
Global Time Control	全程控制	•	•
Correlate all views along a configurable timeline to simplify diagnosis	通过可配置的时间线串联所有视图，简化诊断。
Proactive Health Checks	健康预检	•	•
Monitor dozens of service performance metrics and get alerts you when you approach critical thresholds	监控服务性能，当达到阈值时向用户报警。
Heatmaps	热度图	•	•
Visualize health status and metrics across the cluster to quickly identify problem nodes and take action	图形化展示群集的健康状态，便于发现故障节点并修复。
Customizable Charts	可定制的图表	•	•
Report and visualize on key time-series metrics about services, roles and hosts	按照时间顺序提供服务、角色和主机的形象报告。
Intelligent Log Management	智能日志管理	•	•
Gather, view and search Hadoop logs collected from across the cluster	可以收集、观察和查询从群集中获得的Hadoop日志。
Integrate
Comprehensive API	广泛的API	•	•
Easily integrate Cloudera Manager with your existing enterprise-wide management and monitoring tools	可以简单的将CM与现有的企业范围的管理和监控工具集成起来。
3rd Party Application Management	对第三方应用的管理	•	•
Deploy, manage and monitor services for 3rd party applications running on the cluster (e.g. data integration tools, math/machine learning applications, non-CDH services etc.)	部署、管理和监控运行在群集上的第三方应用服务。
Advanced Management Features (Enabled by Subscription)
Operational Report & Quota Management	操作报告和配额管理		•
Visualize current and historical disk usage; set user and group-based quotas; and track MapReduce, Impala, YARN and HBase usage	1.当前及历史磁盘用量展示 2.基于用户和组的配额设置 3.跟踪MapReduce、Impala、YARN和Hbase的用量
Configuration History & Rollbacks	记录配置历史及回滚		•
Maintain a trail of all actions and a complete record of configuration changes, including the ability to roll back to previous states	保留所有活动及配置变化的痕迹档案，包含回滚到之前状态的能力。
Rolling Updates	滚动升级		•
Stage service updates and restarts to portions of the cluster sequentially to minimize downtime when upgrading or updating your cluster	分阶段升级和重启群集各部分，最小化宕机时间。
AD Kerberos Integration	AD与Kerberos的集成		•
Integrate directly with Active Directory to get started easily with Kerberos	直接与AD集成，可以方便的与Kerberos一起工作
Kerberos Wizard	Kerberos向导		•
Easily configure Kerberos and trigger automated workflows to secure clusters	方便配置Kerberos，可以自动触发工作流来保证群集安全。
Hadoop SSL Related Configs	Hadoop SSL相关配置		•
Simplify configs and eliminates need for safety valves	简化配置并减少安全阀的需求
LDAP/SAML Integration	LDAP/SAML的集成		•
Integrate user credentials with Active Directory and enable single sign-on (SSO) capabilities	集成了基于AD的用户验证，并提供了SSO能力。
SNMP Support	对SNMP的支持		•
Send Hadoop-specific events and alerts to global monitoring tools as SNMP traps	以SNMP 异常报告的方式向全局监控工具发送Hadoop特定的事件和告警。（参见文末注1）
Scheduled Diagnostics	cloudera技术诊断的支持		•
Take a snapshot of the cluster state and automatically send it to Cloudera support to assist with optimization and issue resolution	优化和解决问题时，收集群集状态快照并自动发送至cloudera支持。
Automated Backup & Disaster Recovery	自动化备份和灾难恢复		•
Centrally configure and manage snapshotting and replication workflows for HDFS, Hive and HBase	集中配置和管理快照，复制HDFS、Hive、HBase工作流。

注：

1.snmp traps：SNMP是指简单网络管理协议，trap是它规定的一种通信方式，用于被管理的设备主动向充当管理者的设备报告自己的异常信息。

香蕉与打火机 2014年10月26日小云云 0 Read more >

linux 查看系统连接数、端口占用

一、查看网络连接数：

netstat -an | wc -l

1	netstat -an \| wc -l

查看文件并发数：没！找！到！

回头再找找。

wc(上厕所)就是计数用的。

二、修改

# max user processes 
ulimit -u 1024

1 2	# max user processes ulimit -u 1024

三、查看进程

ps -aux | grep hive

1	ps -aux \| grep hive

香蕉与打火机 2014年10月23日 Linux 0 Read more >

mysql MHA solutions

参考：https://code.google.com/p/mysql-master-ha/wiki/Overview

一、官网场景—— 一主多从

1.主要使用场景：提供master容灾，在master宕机之后，以最短的时间自动将slave提升为master。

工作方式：在master dead之后，自动选择一个slave（可以配置一个优先slave）接管master业务。并配置其他slave切换replication到新的master。

2.除此之外还有：在线master切换（0.5s-2s的不可写时间）。

二、难点——当然也就是MHA试图解决的问题

以下是手工打造时的难点，而MHA进行了解决。

1.选最新的slave接管master。

最新不好检查及选择。

2.其他不是最新的slave的一致性问题。

其他slave切换到新master，会丢失与原master的transaction，造成binary log event不一致。

手工检查binlog是一件很复杂且不易操作的事情。

三、官网wiki提供的其他各种solution

参考：https://code.google.com/p/mysql-master-ha/wiki/Other_HA_Solutions#Master,_one_candidate_master,_and_multiple_slaves

待续…

香蕉与打火机 2014年10月23日 database, mysql 0 Read more >

mysql MHA configuration

一、编写MHA配置文件

1.全局配置文件

编辑：/etc/masterha_default.cnf

[server default]
user=root
password=rootpass
ssh_user=root
master_binlog_dir= /var/lib/mysql
remote_workdir=/data/log/masterha
secondary_check_script= masterha_secondary_check -s remote_host1 -s remote_host2
ping_interval=3
master_ip_failover_script=/script/masterha/master_ip_failover
shutdown_script= /script/masterha/power_manager
report_script= /script/masterha/send_master_failover_mail

[server default]

user=root

password=rootpass

ssh_user=root

master_binlog_dir= /var/lib/mysql

remote_workdir=/data/log/masterha

secondary_check_script= masterha_secondary_check -s remote_host1 -s remote_host2

ping_interval=3

master_ip_failover_script=/script/masterha/master_ip_failover

shutdown_script= /script/masterha/power_manager

report_script= /script/masterha/send_master_failover_mail

2.应用配置文件

每个slaver的配置文件单独编写。而且需要为多个app单独编写配置文件，并在配置文件中使用不同的物理路径。

app1

[server default]
manager_workdir=/var/log/masterha/app1
manager_log=/var/log/masterha/app1/app1.log

[server1]
hostname=host1
candidate_master=1

[server2]
hostname=host2
candidate_master=1

[server3]
hostname=host3

[server4]
hostname=host4
no_master=1

[server default]

manager_workdir=/var/log/masterha/app1

manager_log=/var/log/masterha/app1/app1.log

[server1]

hostname=host1

candidate_master=1

[server2]

hostname=host2

candidate_master=1

[server3]

hostname=host3

[server4]

hostname=host4

no_master=1

app2

manager_host$ cat /etc/app2.cnf

[server default]
manager_workdir=/var/log/masterha/app2
manager_log=/var/log/masterha/app2/app2.log

[server1]
hostname=host11
candidate_master=1

[server2]
hostname=host12
candidate_master=1

[server3]
hostname=host13

[server4]
hostname=host14
no_master=1

manager_host$ cat /etc/app2.cnf

[server default]

manager_workdir=/var/log/masterha/app2

manager_log=/var/log/masterha/app2/app2.log

[server1]

hostname=host11

candidate_master=1

[server2]

hostname=host12

candidate_master=1

[server3]

hostname=host13

[server4]

hostname=host14

no_master=1

二、其他配置

参考：https://code.google.com/p/mysql-master-ha/wiki/Requirements

运行masterha_manager来检查配置是否有缺漏。

1.ssh 免密码登陆

ssh-copy-id testdb1 #testdb1没有加到host文件的话就直接ip
#本机自己也要这么干，否则检查失败

1 2	ssh-copy-id testdb1 #testdb1没有加到host文件的话就直接ip #本机自己也要这么干，否则检查失败

检查：

# masterha_check_ssh --conf=/etc/masterha/app1.cnf

#输出：

Wed Oct 22 14:03:55 2014 - [info] Reading default configuratoins from /etc/masterha_default.cnf..
Wed Oct 22 14:03:55 2014 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Wed Oct 22 14:03:55 2014 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Wed Oct 22 14:03:55 2014 - [info] Starting SSH connection tests..
Wed Oct 22 14:03:56 2014 - [debug] 
Wed Oct 22 14:03:55 2014 - [debug] Connecting via SSH from root@testdb1(172.19.17.210:22) to root@testdb2(172.19.17.211:22)..
Wed Oct 22 14:03:55 2014 - [debug] ok.
Wed Oct 22 14:03:56 2014 - [debug] 
Wed Oct 22 14:03:56 2014 - [debug] Connecting via SSH from root@testdb2(172.19.17.211:22) to root@testdb1(172.19.17.210:22)..
Wed Oct 22 14:03:56 2014 - [debug] ok.
Wed Oct 22 14:03:56 2014 - [info] All SSH connection tests passed successfully.

# masterha_check_ssh --conf=/etc/masterha/app1.cnf

#输出：

Wed Oct 22 14:03:55 2014 - [info] Reading default configuratoins from /etc/masterha_default.cnf..

Wed Oct 22 14:03:55 2014 - [info] Reading application default configurations from /etc/masterha/app1.cnf..

Wed Oct 22 14:03:55 2014 - [info] Reading server configurations from /etc/masterha/app1.cnf..

Wed Oct 22 14:03:55 2014 - [info] Starting SSH connection tests..

Wed Oct 22 14:03:56 2014 - [debug]

Wed Oct 22 14:03:55 2014 - [debug] Connecting via SSH from root@testdb1(172.19.17.210:22) to root@testdb2(172.19.17.211:22)..

Wed Oct 22 14:03:55 2014 - [debug] ok.

Wed Oct 22 14:03:56 2014 - [debug]

Wed Oct 22 14:03:56 2014 - [debug] Connecting via SSH from root@testdb2(172.19.17.211:22) to root@testdb1(172.19.17.210:22)..

Wed Oct 22 14:03:56 2014 - [debug] ok.

Wed Oct 22 14:03:56 2014 - [info] All SSH connection tests passed successfully.

注： masterha_check_ssh 不加配置文件则会报错：

[root@testdb1 ~]# masterha_check_ssh 
--conf=<server_config_file> must be set.

1 2	[root@testdb1 ~]# masterha_check_ssh --conf=<server_config_file> must be set.

2.关闭自动清除relay log^[1]

关闭原因：By default, relay logs on slave servers are automatically removed if SQL threads have finished executing them. But such relay logs might still be needed for recovering other slaves.

^{什么是relay log： relay-log日志记录的是从服务器I/O线程将主服务器的二进制日志读取过来记录到从服务器本地文件，然后SQL线程会读取relay-log日志的内容并应用到从服务器。}

配置方法：

SET GLOBAL relay_log_purge=1

1	SET GLOBAL relay_log_purge=1

关闭之后要手工进行relay log文件的清理，使用Linux自己的cron。周期可以根据磁盘及文件保留期情况自行安排。

vim /etc/cron.d/purge_relay_logs

# purge relay logs at 5am
  0 5 * * * app /usr/bin/purge_relay_logs --user=root --password=PASSWORD --disable_relay_log_purge >> /var/log/masterha/purge_relay_logs.log 2>&1

vim /etc/cron.d/purge_relay_logs

# purge relay logs at 5am

0 5 * * * app /usr/bin/purge_relay_logs --user=root --password=PASSWORD --disable_relay_log_purge >> /var/log/masterha/purge_relay_logs.log 2>&1

三、测试使用

1.测试master 不存在

关闭master

service mysql stop

1	service mysql stop

可以查看日志：

tail -f /masterha/app1/manager.log

1	tail -f /masterha/app1/manager.log

登陆slave机

[root@testdb2 ~]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3983248
Server version: 5.6.19-67.0-log Percona Server (GPL), Release 67.0, Revision 618

Copyright (c) 2009-2014 Percona LLC and/or its affiliates
Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> insert into test (id,value) values (4,44);
Query OK, 1 row affected (0.00 sec)

mysql> select * from test;
+----+-------+
| id | value |
+----+-------+
|  1 |    12 |
|  2 |    22 |
|  3 |    33 |
|  4 |    44 |
+----+-------+
4 rows in set (0.00 sec)

[root@testdb2 ~]# mysql -uroot -p

Enter password:

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 3983248

Server version: 5.6.19-67.0-log Percona Server (GPL), Release 67.0, Revision 618

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use test

Reading table information for completion of table and column names

You can turn off this feature to get a quicker startup with -A

Database changed

mysql> insert into test (id,value) values (4,44);

Query OK, 1 row affected (0.00 sec)

mysql> select * from test;

+----+-------+

| id | value |

+----+-------+

| 1 | 12 |

| 2 | 22 |

| 3 | 33 |

| 4 | 44 |

+----+-------+

4 rows in set (0.00 sec)

可以代替master的功能接收数据。

2.重新启动原master

[root@testdb1 ~]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.6.19-67.0-log Percona Server (GPL), Release 67.0, Revision 618

Copyright (c) 2009-2014 Percona LLC and/or its affiliates
Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from test;
+----+-------+
| id | value |
+----+-------+
|  1 |    12 |
|  2 |    22 |
|  3 |    33 |
+----+-------+
3 rows in set (0.00 sec)

mysql> exit

[root@testdb1 ~]# mysql -uroot -p

Enter password:

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 4

Server version: 5.6.19-67.0-log Percona Server (GPL), Release 67.0, Revision 618

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use test

Reading table information for completion of table and column names

You can turn off this feature to get a quicker startup with -A

Database changed

mysql> select * from test;

+----+-------+

| id | value |

+----+-------+

| 1 | 12 |

| 2 | 22 |

| 3 | 33 |

+----+-------+

3 rows in set (0.00 sec)

mysql> exit

【注1】原master起来以后有个问题，就是其全局可写。如果这时有数据插进来，会造成新老master数据不一致。因此应该在重启原master后首先设置global read_only=1。然后将原master切为slave，从新master上同步所有数据。

【注2】同步数据时，原master会等到新master有一次数据操作时，统一将所有数据变化同步过来。这个时刻不是在配完slave后就完成，需等新master。

切换主从的blog：http://bananalighter.com/change-slaver2master/

参考：http://ylw6006.blog.51cto.com/470441/890360/

香蕉与打火机 2014年10月22日 database, mysql 0 Read more >

mysql 配置 master slaver replication

配置master用户及权限

grant replication slave,replication client on *.* to rep@'172.19.17.%' identified by 'rep';

grant all on test.* to rep@'172.19.17.%';

commit;

grant replication slave,replication client on *.* to rep@'172.19.17.%' identified by 'rep';

grant all on test.* to rep@'172.19.17.%';

commit;

配置slaver用户及权限

grant replication slave,replication client on *.* to rep@'172.19.17.%' identified by 'rep';

grant all on test.* to rep@'172.19.17.%';

commit;

change master to master_host='172.19.17.210', master_user='rep',master_password='rep',master_log_file='mysql-bin.000001',master_log_pos=0;

start slave;