Mysql-binlog 介绍

介绍
mysql-binlog是MySQL数据库的二进制日志,以事件形式记录了所有的 DDL 和 DML 语句(除了数据查询语句),还记录语句所执行的消耗时间。MySQL的二进制日志是事务安全型的。binlog 的主要目的是复制和恢复。

binlog有多种格式

1、 Statement:每一条会修改数据的sql都会记录在binlog中
2、 Row:不记录sql语句上下文相关信息,仅保存哪条记录被修改
3、 Mixed:是以上两种level的混合使用,一般的语句修改使用statment格式保存binlog,如一些函数,statement无法完成主从复制的操作,则采用row格式保存binlog

binlog 查看配置命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
mysql> show variables like 'version';
+---------------+--------+
| Variable_name | Value |
+---------------+--------+
| version | 8.0.13 |
+---------------+--------+
1 row in set, 1 warning (0.00 sec)

mysql> show variables like 'log_bin';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_bin | ON |
+---------------+-------+
1 row in set, 1 warning (0.01 sec)
mysql> show variables like '%log_bin%';
+---------------------------------+---------------------------------------------------+
| Variable_name | Value |
+---------------------------------+---------------------------------------------------+
| log_bin | ON |
| log_bin_basename | D:\tools\mysql-8.0.13-winx64\data\mysql-bin |
| log_bin_index | D:\tools\mysql-8.0.13-winx64\data\mysql-bin.index |
| log_bin_trust_function_creators | OFF |
| log_bin_use_v1_row_events | OFF |
| sql_log_bin | ON |
+---------------------------------+---------------------------------------------------+
6 rows in set, 1 warning (0.01 sec)

binlog配置

1
2
3
4
#文件名设置
log-bin=mysql-bin
#日志格式
binlog_format=ROW

binlog 文件操作命令

1
2
3
4
5
6
#会多一个最新的bin-log日志
flush logs
#查看最后一个bin-log日志的相关信息
show master status
#清空所有的bin-log日志
reset master

sql 操作演示

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
mysql> create database binlog;
Query OK, 1 row affected (0.06 sec)

mysql> use binlog;
Database changed
mysql> create table test(id int auto_increment not null primary key, val int,data varchar(20));
Query OK, 0 rows affected (0.15 sec)
#分一个新的日志文件 000002
mysql> flush logs;
Query OK, 0 rows affected (0.04 sec)

mysql> insert into test(val, data) values (1, 'zhang');
Query OK, 1 row affected (0.10 sec)

mysql> insert into test(val, data) values (2, 'wang');
Query OK, 1 row affected (0.04 sec)
#分一个新的日志文件 000003
mysql> flush logs;
Query OK, 0 rows affected (0.02 sec)

mysql> insert into test(val, data) values (3, 'zhao');
Query OK, 1 row affected (0.11 sec)
#分一个新的日志文件 000003
mysql> flush logs;
Query OK, 0 rows affected (0.03 sec)

mysql> delete from test where id=3
mysql> drop table test;
Query OK, 0 rows affected (0.09 sec)

mysql> drop database binlog;
Query OK, 0 rows affected (0.09 sec)

mysql> show binary logs;
+------------------+-----------+
| Log_name | File_size |
+------------------+-----------+
| mysql-bin.000001 | 643 |
| mysql-bin.000002 | 789 |
| mysql-bin.000003 | 495 |
| mysql-bin.000004 | 841 |
+------------------+-----------+
4 rows in set (0.00 sec)
#恢复mysql-bin.000001 创建的数据(不能包含删除数据库sql,不然恢复不成功,这里单独把删除sql存放在了第二个binlog文件)
D:\tools\mysql-8.0.13-winx64\bin>mysqlbinlog ..\data\mysql-bin.000001 | mysql -u root -p
Enter password: ******
#binlog解析成txt文件
D:\tools\mysql-8.0.13-winx64\bin>mysqlbinlog ..\data\mysql-bin.000003 > d:/log.txt

日志文件查看

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
mysql> show binary logs;
+------------------+-----------+
| Log_name | File_size |
+------------------+-----------+
| mysql-bin.000001 | 4883 |
+------------------+-----------+
1 row in set (0.01 sec)

mysql> show binlog events;
+------------------+------+----------------+-----------+-------------+--------------------------------------+
| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |
+------------------+------+----------------+-----------+-------------+--------------------------------------+
| mysql-bin.000001 | 4 | Format_desc | 1 | 124 | Server ver: 8.0.13, Binlog ver: 4 |
| mysql-bin.000001 | 124 | Previous_gtids | 1 | 155 | |
| mysql-bin.000001 | 155 | Anonymous_Gtid | 1 | 230 | SET @@SESSION.GTID_NEXT= 'ANONYMOUS' |
| mysql-bin.000001 | 230 | Query | 1 | 309 | BEGIN |
| mysql-bin.000001 | 309 | Table_map | 1 | 368 | table_id: 66 (binlog.test) |
| mysql-bin.000001 | 368 | Write_rows | 1 | 418 | table_id: 66 flags: STMT_END_F |
| mysql-bin.000001 | 418 | Xid | 1 | 449 | COMMIT /* xid=13 */ |
| mysql-bin.000001 | 449 | Anonymous_Gtid | 1 | 524 | SET @@SESSION.GTID_NEXT= 'ANONYMOUS' |
| mysql-bin.000001 | 524 | Query | 1 | 603 | BEGIN |
| mysql-bin.000001 | 603 | Table_map | 1 | 662 | table_id: 66 (binlog.test) |
| mysql-bin.000001 | 662 | Write_rows | 1 | 711 | table_id: 66 flags: STMT_END_F |
| mysql-bin.000001 | 711 | Xid | 1 | 742 | COMMIT /* xid=98 */ |
+------------------+------+----------------+-----------+-------------+--------------------------------------+
...略
82 rows in set (0.01 sec)
# 查看指定binlog文件
mysql> show binlog events in 'mysql-bin.000001';
#获取指定内容
mysql> show binlog events from 4800;
+------------------+------+------------+-----------+-------------+--------------------------------+
| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |
+------------------+------+------------+-----------+-------------+--------------------------------+
| mysql-bin.000001 | 4800 | Write_rows | 1 | 4852 | table_id: 66 flags: STMT_END_F |
| mysql-bin.000001 | 4852 | Xid | 1 | 4883 | COMMIT /* xid=302 */ |
+------------------+------+------------+-----------+-------------+--------------------------------+
2 rows in set (0.00 sec)

日志文件内容查看

当有数据变更会在本地数据库数据存放目录下面生成以下:

1
2
3
4
5
6
#索引文件
mysql-bin.index
#binlog文件
mysql-bin.000001
#将binlog文件解析成 txt
D:\tools\mysql-8.0.13-winx64\bin>mysqlbinlog ..\data\mysql-bin.000001 > d:/log1.txt

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#190731 17:41:00 server id 1 end_log_pos 124 CRC32 0xe904f22c Start: binlog v 4, server v 8.0.13 created 190731 17:41:00
BINLOG '
LGJBXQ8BAAAAeAAAAHwAAAAAAAQAOC4wLjEzAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAEwANAAgAAAAABAAEAAAAYAAEGggAAAAICAgCAAAACgoKKioAEjQA
CgEs8gTp
'/*!*/;
# at 124
#190731 17:41:00 server id 1 end_log_pos 155 CRC32 0xd12224e5 Previous-GTIDs
# [empty]
# at 155
#190731 17:41:15 server id 1 end_log_pos 230 CRC32 0x25b4eff3 Anonymous_GTID last_committed=0 sequence_number=1 rbr_only=yes original_committed_timestamp=1564566075183241 immediate_commit_timestamp=1564566075183241 transaction_length=294
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
# original_commit_timestamp=1564566075183241 (2019-07-31 17:41:15.183241 ?D1ú±ê×?ê±??)
# immediate_commit_timestamp=1564566075183241 (2019-07-31 17:41:15.183241 ?D1ú±ê×?ê±??)
/*!80001 SET @@session.original_commit_timestamp=1564566075183241*//*!*/;
SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/;
# at 230
#190731 17:41:15 server id 1 end_log_pos 309 CRC32 0xb95da169 Query thread_id=9 exec_time=0 error_code=0
SET TIMESTAMP=1564566075/*!*/;
SET @@session.pseudo_thread_id=9/*!*/;
SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/;
SET @@session.sql_mode=1168113696/*!*/;
SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/;
/*!\C utf8 *//*!*/;
SET @@session.character_set_client=33,@@session.collation_connection=33,@@session.collation_server=33/*!*/;
SET @@session.lc_time_names=0/*!*/;
SET @@session.collation_database=DEFAULT/*!*/;
/*!80011 SET @@session.default_collation_for_utf8mb4=255*//*!*/;
/*!80013 SET @@session.sql_require_primary_key=0*//*!*/;
BEGIN
/*!*/;
# at 309
#190731 17:41:15 server id 1 end_log_pos 368 CRC32 0x7203817f Table_map: `binlog`.`test` mapped to number 115
# at 368
#190731 17:41:15 server id 1 end_log_pos 418 CRC32 0x8be3841b Write_rows: table id 115 flags: STMT_END_F

BINLOG '
O2JBXRMBAAAAOwAAAHABAAAAAHMAAAAAAAEABmJpbmxvZwAEdGVzdAADAwMPAjwABgEBAAIBIX+B
A3I=
O2JBXR4BAAAAMgAAAKIBAAAAAHMAAAAAAAEAAgAD/wABAAAAAQAAAAV6aGFuZxuE44s=
'/*!*/;
# at 418
#190731 17:41:15 server id 1 end_log_pos 449 CRC32 0xfc026d64 Xid = 177
COMMIT/*!*/;
# at 449
#190731 17:41:22 server id 1 end_log_pos 524 CRC32 0x113e4745 Anonymous_GTID last_committed=1 sequence_number=2 rbr_only=yes original_committed_timestamp=1564566082696808 immediate_commit_timestamp=1564566082696808 transaction_length=293
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
# original_commit_timestamp=1564566082696808 (2019-07-31 17:41:22.696808 ?D1ú±ê×?ê±??)
# immediate_commit_timestamp=1564566082696808 (2019-07-31 17:41:22.696808 ?D1ú±ê×?ê±??)
/*!80001 SET @@session.original_commit_timestamp=1564566082696808*//*!*/;
SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/;
# at 524
#190731 17:41:22 server id 1 end_log_pos 603 CRC32 0xe309f990 Query thread_id=9 exec_time=0 error_code=0
SET TIMESTAMP=1564566082/*!*/;
/*!80013 SET @@session.sql_require_primary_key=0*//*!*/;
BEGIN
/*!*/;
# at 603
#190731 17:41:22 server id 1 end_log_pos 662 CRC32 0xc07d94f6 Table_map: `binlog`.`test` mapped to number 115
# at 662
#190731 17:41:22 server id 1 end_log_pos 711 CRC32 0x42d695e3 Write_rows: table id 115 flags: STMT_END_F

BINLOG '
QmJBXRMBAAAAOwAAAJYCAAAAAHMAAAAAAAEABmJpbmxvZwAEdGVzdAADAwMPAjwABgEBAAIBIfaU
fcA=
QmJBXR4BAAAAMQAAAMcCAAAAAHMAAAAAAAEAAgAD/wACAAAAAgAAAAR3YW5n45XWQg==
'/*!*/;
# at 711
#190731 17:41:22 server id 1 end_log_pos 742 CRC32 0x8bde4f70 Xid = 178
COMMIT/*!*/;
# at 742
#190731 17:41:38 server id 1 end_log_pos 789 CRC32 0xcf82cc85 Rotate to mysql-bin.000003 pos: 4
SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;
DELIMITER ;
# End of log file
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;

#恢复第一条数据
D:\tools\mysql-8.0.13-winx64\bin>mysqlbinlog ..\data\mysql-bin.000002 --stop-position=603 | mysql -uroot -p
Enter password: ******
#恢复第二条数据
D:\tools\mysql-8.0.13-winx64\bin>mysqlbinlog ..\data\mysql-bin.000002 --start-position=603 --stop-position=742 | mysql -uroot -p
Enter password: ******

恢复数据mysql-bin.000003

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
BEGIN
/*!*/;
# at 309
#190731 17:41:44 server id 1 end_log_pos 368 CRC32 0x6286ca84 Table_map: `binlog`.`test` mapped to number 115
# at 368
#190731 17:41:44 server id 1 end_log_pos 417 CRC32 0x5004c380 Write_rows: table id 115 flags: STMT_END_F

BINLOG '
WGJBXRMBAAAAOwAAAHABAAAAAHMAAAAAAAEABmJpbmxvZwAEdGVzdAADAwMPAjwABgEBAAIBIYTK
hmI=
WGJBXR4BAAAAMQAAAKEBAAAAAHMAAAAAAAEAAgAD/wADAAAAAwAAAAR6aGFvgMMEUA==
'/*!*/;
# at 417
#190731 17:41:44 server id 1 end_log_pos 448 CRC32 0x8433520a Xid = 180
COMMIT/*!*/;
#恢复第三条数据
D:\tools\mysql-8.0.13-winx64\bin>mysqlbinlog ..\data\mysql-bin.000003 --start-position=309 --stop-position=417 | mysql -uroot -p
Enter password: ******

文件转码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
D:\tools\mysql-8.0.13-winx64\bin>mysqlbinlog  --base64-output=DECODE-ROWS -v  ..\data\mysql-bin.000005 > d:/log6.txt

BEGIN
/*!*/;
# at 309
#190731 19:29:49 server id 1 end_log_pos 368 CRC32 0x0bc503bf Table_map: `binlog`.`test` mapped to number 62
# at 368
#190731 19:29:49 server id 1 end_log_pos 416 CRC32 0x5a878e6d Write_rows: table id 62 flags: STMT_END_F
### INSERT INTO `binlog`.`test`
### SET
### @1=4
### @2=4
### @3='tan'
# at 416
#190731 19:29:49 server id 1 end_log_pos 447 CRC32 0x5064407d Xid = 39
COMMIT/*!*/;

常用命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
是否启用了日志
mysql>show variables like 'log_bin';

怎样知道当前的日志
mysql> show master status;

查看mysql binlog模式
show variables like 'binlog_format';

获取binlog文件列表
show binary logs;

查看当前正在写入的binlog文件
show master status\G

查看指定binlog文件的内容
show binlog events in 'mysql-bin.000002';

ZooKeeper Linux 服务器安装

需要安装 Java 环境

环境检查

1
2
3
4
5
6
7
8
9
10
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)

$ java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

$ echo $JAVA_HOME
/home/redhad/soft/jdk1.8.0_181

安装

1
2
$ wget http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz
$ tar -zxvf zookeeper-3.4.10.tar.gz

进入解压目录

1
$ cd zookeeper-3.4.10/

conf/目录下有个zoo_sample.cfg,复制一份命名为zoo.cfg

1
$ cp zoo_sample.cfg zoo.cfg

zoo.cfg 文件内容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
$ vi conf/zoo.cfg 

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/tmp/zookeeper/data
dataLogDir=/tmp/zookeeper/logs
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

启动服务

1
2
3
4
$ bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/boat/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

查看状态

1
2
3
4
$ bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/boat/zookeeper-3.4.10/bin/../conf/zoo.cfg
Mode: standalone

查看端口号是否被占用

1
$ netstat -lpn | grep 2181

启动 Cli

1
2
3
4
5
6
7
8
9
10
11
#zkCli.sh命令默认以主机号 127.0.0.1,端口号 2181 来连接zk
$ bin/zkCli.sh

Connecting to localhost:2181
2019-02-17 23:42:15,403 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
Welcome to ZooKeeper!

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]

创建节点及查看

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[zk: localhost:2181(CONNECTED) 0] create /zktest 186
Created /zktest
[zk: localhost:2181(CONNECTED) 1] ls /
[dubbo, zktest, zookeeper]
[zk: localhost:2181(CONNECTED) 2] get /zktest
186
cZxid = 0xf4
ctime = Fri Jun 21 02:06:52 PDT 2019
mZxid = 0xf4
mtime = Fri Jun 21 02:06:52 PDT 2019
pZxid = 0xf4
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 3
numChildren = 0
[zk: localhost:2181(CONNECTED) 3] ls /dubbo
[com.example.demoWeb.DemoService, com.example.dubbo.DemoService]

停用 zookeeper

1
2
3
4
$ bin/zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /home/boat/zookeeper-3.4.10/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED

Security 自定义登录过滤器

通过 Security 实现登录自定义参数和Json请求方式

UsernamePasswordAuthenticationFilter 源码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
public static final String SPRING_SECURITY_FORM_USERNAME_KEY = "username";
public static final String SPRING_SECURITY_FORM_PASSWORD_KEY = "password";

private String usernameParameter = SPRING_SECURITY_FORM_USERNAME_KEY;
private String passwordParameter = SPRING_SECURITY_FORM_PASSWORD_KEY;
private boolean postOnly = true;

// ~ Constructors
// ===================================================================================================

public UsernamePasswordAuthenticationFilter() {
super(new AntPathRequestMatcher("/login", "POST"));
}

// ~ Methods
// ========================================================================================================

public Authentication attemptAuthentication(HttpServletRequest request,
HttpServletResponse response) throws AuthenticationException {
if (postOnly && !request.getMethod().equals("POST")) {
throw new AuthenticationServiceException(
"Authentication method not supported: " + request.getMethod());
}

String username = obtainUsername(request);
String password = obtainPassword(request);

if (username == null) {
username = "";
}

if (password == null) {
password = "";
}

username = username.trim();

UsernamePasswordAuthenticationToken authRequest = new UsernamePasswordAuthenticationToken(
username, password);

// Allow subclasses to set the "details" property
setDetails(request, authRequest);

return this.getAuthenticationManager().authenticate(authRequest);
}

通过重写attemptAuthentication 实现自定义参数

新建自定义过滤器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
public class UserLoginAuthenticationFilter extends UsernamePasswordAuthenticationFilter {

private static final Logger LOGGER = LoggerFactory.getLogger(UserAuthenticationManager.class);

@Reference(version = "1.0")
private UserService userService;

@Override
public Authentication attemptAuthentication(HttpServletRequest request,
HttpServletResponse response) throws AuthenticationException {
System.out.println("UserLoginAuthenticationFilter --> attemptAuthentication");

if (!request.getMethod().equals("POST")) {
throw new AuthenticationServiceException(
"Authentication method not supported: " + request.getMethod());
}
LoginRequest loginRequest = builderLoginRequest(request);
TeResponse resp = userService.login(loginRequest);
if (!resp.isStatus()) {
throw new BadCredentialsException(resp.getMessage()+"="+resp.getStatusCode());
}

UsernamePasswordAuthenticationToken authRequest = new UsernamePasswordAuthenticationToken(
loginRequest.getAccount(), BCrypt.hashpw(loginRequest.getPassword(), BCrypt.gensalt()));

setDetails(request, authRequest);
return this.getAuthenticationManager().authenticate(authRequest);
}

/**
* 构造请求参数
**/
@SuppressWarnings("unchecked")
private LoginRequest builderLoginRequest(HttpServletRequest request){
if(request.getContentType().equals(MediaType.APPLICATION_JSON_UTF8_VALUE)
||request.getContentType().equals(MediaType.APPLICATION_JSON_VALUE)){
ObjectMapper mapper = new ObjectMapper();
try (InputStream is = request.getInputStream()){
Map<String,String> map = mapper.readValue(is, Map.class);
return getLoginRequest(map);
}catch (IOException e) {
LOGGER.error("builderLoginRequest error-->",e);
throw new BadCredentialsException("请求格式异常=1003");
}
}
else{
return getLoginRequest(request);
}
}

/**
* 构造请求参数
* application/x-www-form-urlencoded
**/
private LoginRequest getLoginRequest(HttpServletRequest request){
LoginRequest loginRequest = new LoginRequest();
loginRequest.setUuid(request.getParameter(USER_UUID_KEY));
loginRequest.setVerifyCode(request.getParameter(USER_VERIFY_CODE_KEY));
loginRequest.setAccount(request.getParameter(USER_ACCOUNT_KEY));
loginRequest.setPassword(request.getParameter(USER_PASSWORD_KEY));
return loginRequest;
}
/**
* 构造请求参数
* application/json
**/
private LoginRequest getLoginRequest(Map<String,String> parameters ){
LoginRequest loginRequest = new LoginRequest();
loginRequest.setUuid(parameters.get(USER_UUID_KEY));
loginRequest.setVerifyCode(parameters.get(USER_VERIFY_CODE_KEY));
loginRequest.setAccount(parameters.get(USER_ACCOUNT_KEY));
loginRequest.setPassword(parameters.get(USER_PASSWORD_KEY));
return loginRequest;
}

WebSecurityConfig 配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
@Configuration
@EnableWebSecurity
@EnableGlobalMethodSecurity(prePostEnabled = true, securedEnabled = true)
public class WebSecurityConfig extends WebSecurityConfigurerAdapter {

private static String[] pathArray = new String[]{
"/user/captcha/base64Code","/user/forget/password"
};

@Autowired
private UserAuthenticationManager userAuthenticationManager;

@Override
protected void configure(HttpSecurity http) throws Exception {

http.cors().configurationSource(getSource());
http.csrf().disable();

http.authorizeRequests()
.antMatchers(pathArray).permitAll()
.anyRequest().authenticated();
// 添加登录自定义过滤器
http.addFilterAt(userLoginAuthenticationFilter(),UsernamePasswordAuthenticationFilter.class);


http.logout()
.logoutUrl("/logout")
.deleteCookies("JSESSIONID")
.invalidateHttpSession(true)
.logoutSuccessHandler(logoutSuccessHandler())
.permitAll();

http.exceptionHandling()
.authenticationEntryPoint(new LoginUrlAuthenticationEntryPoint())
.accessDeniedHandler(accessDeniedHandler());
}

@Bean
UserLoginAuthenticationFilter userLoginAuthenticationFilter(){
UserLoginAuthenticationFilter filter = new UserLoginAuthenticationFilter();
filter.setRequiresAuthenticationRequestMatcher(new AntPathRequestMatcher("/user/login", "POST"));
filter.setAuthenticationFailureHandler(failureHandler());
filter.setAuthenticationSuccessHandler(authSuccessHandler());
filter.setAuthenticationManager(userAuthenticationManager);
return filter;
}

private UrlBasedCorsConfigurationSource getSource() {
CorsConfiguration configuration = new CorsConfiguration();
String corsAllowedOrigins = "http://127.0.0.1:20880,http://127.0.0.1:20881";
configuration.setAllowedOrigins(Arrays.asList(corsAllowedOrigins.split(",")));
String corsAllowedMethod = "POST, GET, OPTIONS";
configuration.setAllowedMethods(Arrays.asList(corsAllowedMethod.split(",")));
configuration.applyPermitDefaultValues();
UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource();
String corsPath = "/**";
source.registerCorsConfiguration(corsPath, configuration);
return source;
}

@Override
public void configure(WebSecurity web) throws Exception {
web.securityInterceptor(new SecurityInterceptor());
}

@Bean
AuthenticationSuccessHandler authSuccessHandler() {
return new UserAuthenticationHandler();
}

@Bean
AuthenticationFailureHandler failureHandler() {
return new UserAuthenticationFailureHandler();
}

@Bean
LogoutSuccessHandler logoutSuccessHandler() {
return new UserLogoutSuccessHandler();
}

@Bean
AccessDeniedHandler accessDeniedHandler() {
return new UserAccessDeniedHandler();
}
}

Security 登录功能实现

通过 Security 登录功能实现

Security pom包引用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-test</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
</dependencies>

WebSecurityConfig 配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
@Configuration
@EnableWebSecurity
@EnableGlobalMethodSecurity(prePostEnabled = true, securedEnabled = true, jsr250Enabled = true)
public class WebSecurityConfig extends WebSecurityConfigurerAdapter {

@Override
protected void configure(HttpSecurity http) throws Exception {

String[] pathArray = new String[] {"/user/captcha/base64Code","/user/send/sms/verifyCode","/user/forget/password"};
http.cors();
http.csrf().disable();
http.authorizeRequests()
.anyRequest().authenticated()
// 请求访问控制
.accessDecisionManager(accessDecisionManager())
.antMatchers(pathArray).permitAll()
.anyRequest().authenticated();

http.formLogin()
.loginProcessingUrl("/user/login")
//登录成功
.successHandler(authSuccessHandler())
//登录失败
.failureHandler(failureHandler())
.permitAll();

http.logout()
.logoutUrl("/logout")
.deleteCookies("JSESSIONID")
.invalidateHttpSession(true)
//退出成功
.logoutSuccessHandler(logoutSuccessHandler())
.permitAll();
http.exceptionHandling()
//未授权拦截调转
.authenticationEntryPoint(new LoginUrlAuthenticationEntryPoint())
.accessDeniedHandler(accessDeniedHandler());
}
@Autowired
private UserDetailsService userDetailsService;

@Override
protected void configure(AuthenticationManagerBuilder auth) throws Exception {
System.out.println("configure");
auth.userDetailsService(userDetailsService);
}

@Bean
@Override
protected AuthenticationManager authenticationManager() {
System.out.println("authenticationManager");
return new UserAuthenticationManager();
}

@Bean
AuthenticationSuccessHandler authSuccessHandler() {
return new UserAuthenticationHandler();
}

@Bean
AuthenticationFailureHandler failureHandler() {
return new UserFailureHandler();
}

@Bean
public AccessDecisionManager accessDecisionManager() {
List<AccessDecisionVoter<?>> decisionVoters = Arrays.asList(
new RoleVoter(),
authorityVoter()
);
return new UnanimousBased(decisionVoters);
}

@Bean
UserAccessDecisionVoter authorityVoter() {
return new UserAccessDecisionVoter();
}

@Bean
LogoutSuccessHandler logoutSuccessHandler() {
return new UserLogoutSuccessHandler();
}

@Bean
AccessDeniedHandler accessDeniedHandler() {
return new UserAccessDeniedHandler();
}

UserDetailsService 用户查询服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
@Service
public class UserDetailsServiceImpl implements UserDetailsService {
/**
* 通过用户名查询用户详细信息
* @param username
* @return
* @throws UsernameNotFoundException
*/
@Override
public UserDetails loadUserByUsername(String username) throws UsernameNotFoundException {
System.out.println("loadUserByUsername");
//查询 User
return null;
}
}

UserAccessDecisionVoter 投票器

可以动态实现对请求地址的拦截

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
public class UserAccessDecisionVoter implements AccessDecisionVoter {
@Override
public boolean supports(ConfigAttribute attribute) {
return true;
}

@Override
public int vote(Authentication authentication, Object object, Collection collection) {
System.out.println("UserAccessDecisionVoter");
FilterInvocation fi = (FilterInvocation) object;
System.out.println(fi.getRequestUrl());
return 1;
}

@Override
public boolean supports(Class clazz) {
return clazz.equals(FilterInvocation.class);
}
}

UserAuthenticationManager 用户授权管理

对不同方式登录的用户授权,如快捷登录,扫码登录

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
public class UserAuthenticationManager implements AuthenticationManager {

private static final Logger LOGGER = LoggerFactory.getLogger(UserAuthenticationManager.class);
@Reference(version = "1.0")
private UserService userService;

private final List<GrantedAuthority> AUTHORITIES = new ArrayList<>();

public UserAuthenticationManager() {
AUTHORITIES.add(new SimpleGrantedAuthority("ROLE_USER"));
}

@Override
public Authentication authenticate(Authentication auth) throws AuthenticationException {

if (check(auth)) {
LOGGER.info("UserAuthenticationManager -->");
return new UsernamePasswordAuthenticationToken(auth.getPrincipal(), auth.getCredentials(), AUTHORITIES);
}
throw new BadCredentialsException("Bad Credentials");
}
...
}

UserAuthenticationHandler 登录成功

1
2
3
4
5
6
7
8
9
10
public class UserAuthenticationHandler extends SimpleUrlAuthenticationSuccessHandler {
@Override
public void onAuthenticationSuccess(HttpServletRequest request
, HttpServletResponse response, Authentication authentication) throws IOException, ServletException {
response.setCharacterEncoding("UTF-8");
System.out.println("onAuthenticationSuccess");
// 定义调转方式
// JSON.writeJSONString(response.getWriter(), result);
}
}

UserFailureHandler 登录失败

对登录失败异常定义

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
public class UserFailureHandler extends SimpleUrlAuthenticationFailureHandler {
@Override
public void onAuthenticationFailure(HttpServletRequest request, HttpServletResponse response
, AuthenticationException exception) throws IOException, ServletException {
response.setCharacterEncoding("UTF-8");
logger.info("onAuthenticationFailure "+exception.getMessage(),exception);

if(exception instanceof BadCredentialsException){
String[] array = exception.getMessage().split("=");
if(array.length>1){
JSON.writeJSONString(response.getWriter(), TeResponse.error(array[0],array[1]));
return;
}
}
JSON.writeJSONString(response.getWriter(), TeResponse.error(StateEnum.S_1004));
}
}

UserLogoutSuccessHandler 退出成功

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
public class UserLogoutSuccessHandler implements LogoutSuccessHandler {
@Override
public void onLogoutSuccess(HttpServletRequest request, HttpServletResponse response, Authentication authentication) throws IOException, ServletException {
SessionUtil.removeSessionAttribute(Const.LOGIN_SESSION_KEY, request);
SecurityContextHolder.clearContext();
response.setCharacterEncoding("UTF-8");
JSON.writeJSONString(response.getWriter(), TeResponse.success());
}
}

### UserAccessDeniedHandler 自定义权限不足
```java
public class UserAccessDeniedHandler implements AccessDeniedHandler {
@Override
public void handle(HttpServletRequest request, HttpServletResponse response, AccessDeniedException accessDeniedException) throws IOException, ServletException {
TeUser user = SessionUtil.getSessionAttribute(Const.LOGIN_SESSION_KEY, request);
System.out.println("UserAccessDeniedHandler");
response.setCharacterEncoding("UTF-8");
if (null == user) {
JSON.writeJSONString(response.getWriter(), TeResponse.error(StateEnum.S_401));
}
else{
JSON.writeJSONString(response.getWriter(), TeResponse.error(StateEnum.S_403));
}
}
}

UserAccessDeniedHandler 认证用户无权限访问异常

1
2
3
4
5
6
7
8
9
10
11
12
13
14
public class UserAccessDeniedHandler implements AccessDeniedHandler {
@Override
public void handle(HttpServletRequest request, HttpServletResponse response, AccessDeniedException accessDeniedException) throws IOException, ServletException {
TeUser user = SessionUtil.getSessionAttribute(Const.LOGIN_SESSION_KEY, request);
System.out.println("UserAccessDeniedHandler");
response.setCharacterEncoding("UTF-8");
if (null == user) {
JSON.writeJSONString(response.getWriter(), TeResponse.error(StateEnum.S_401));
}
else{
JSON.writeJSONString(response.getWriter(), TeResponse.error(StateEnum.S_403));
}
}
}

LoginUrlAuthenticationEntryPoint 未登录无权限访问异常

1
2
3
4
5
6
7
8
9
10
11
public class LoginUrlAuthenticationEntryPoint implements AuthenticationEntryPoint {
@Override
public void commence(HttpServletRequest request, HttpServletResponse response, AuthenticationException authException) throws IOException, ServletException {
System.out.println("LoginUrlAuthenticationEntryPoint");
response.setContentType("application/json;charset=utf-8");
PrintWriter out = response.getWriter();
out.write(JsonUtil.toJSON(TeResponse.error(StateEnum.S_401)));
out.flush();
out.close();
}
}

Elasticsearch Linux 服务器安装

需要安装 Java 8 环境

环境检查

1
2
3
4
5
6
7
8
9
10
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)

$ java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

$ echo $JAVA_HOME
/home/redhad/soft/jdk1.8.0_181

安装

1
2
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.1.zip
$ unzip elasticsearch-5.5.1.zip

进入解压目录

1
$ cd elasticsearch-5.5.1/

启动elasticsearch

1
$ ./bin/elasticsearch

启动错误提示

1、max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

1
$ sysctl -w vm.max_map_count=262144

2、max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

1
2
3
4
$ vi /etc/security/limits.conf
#添加以下内容
* soft nofile 65536
* hard nofile 131072

默认情况下只能本机访问,远程访问需要修改
安装目录下config/elasticsearch.yml文件
去掉network.host 注释,将值改成0.0.0.0

1
network.host: 0.0.0.0

Elasticsearch head客户端连接失败

1
2
3
4
#查看firewalld 状态
$ systemctl status firewalld
#关闭 firewalld
$ service firewalld stop

关闭firewalld客户端连接成功
image

DubboFilter 使用


前言

项目中使用了dubbo,想实现拦截的功能,查看源码对dubbo-filter 做下了解

dubbo-filter-validation 源码查看

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
/**
* ValidationFilter
*/
@Activate(group = {Constants.CONSUMER, Constants.PROVIDER}, value = Constants.VALIDATION_KEY, order = 10000)
public class ValidationFilter implements Filter {

private Validation validation;

public void setValidation(Validation validation) {
this.validation = validation;
}

public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
if (validation != null && !invocation.getMethodName().startsWith("$")
&& ConfigUtils.isNotEmpty(invoker.getUrl().getMethodParameter(invocation.getMethodName(), Constants.VALIDATION_KEY))) {
try {
Validator validator = validation.getValidator(invoker.getUrl());
if (validator != null) {
validator.validate(invocation.getMethodName(), invocation.getParameterTypes(), invocation.getArguments());
}
} catch (RpcException e) {
throw e;
} catch (Throwable t) {
return new RpcResult(t);
}
}
return invoker.invoke(invocation);
}

}

ValidationFilter 类继承Filter接口, 重写 invoke方法对调用方法参数进行拦截,每个Filter都要加上@Activate注解用于激活,group当其中一个组匹配时,激活当前扩展,order为Filter排序,可选参数。

dubbo-filter-validation配置

配置在META-INF/dubbo/internal/com.alibaba.dubbo.rpc.filter 文件
文件内容
validation=com.alibaba.dubbo.validation.filter.ValidationFilter

dubbo-filter-exception 项目中使用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
@Activate(group = Constants.PROVIDER)
public class DubboFilter implements Filter {

private final static Logger logger = LoggerFactory.getLogger(DubboFilter.class);

@Override
public Result invoke(Invoker<?> invoker, Invocation invocation) {
logger.info("#############DubboFilter start###############" );
Result result = invoker.invoke(invocation);
Throwable exception = result.getException();
if (exception != null) {
logger.error("dubbo " + invocation.getMethodName()+" error-->" , exception);
result = new RpcResult(Result.error("请求异常拦截", "-1"));
}
logger.info("#############DubboFilter end###############" );
return result;
}
}

dubbo-filter-exception 配置

配置在META-INF/dubbo/com.alibaba.dubbo.rpc.filter 文件
文件内容
dubboFilter=com.xx.xx.xx.filter.DubboFilter

运行结果

1
2
3
[10-11 13:37:31.031] [WARN] [cat] Cat is lazy initialized!
this app is start
{"message":"请求异常拦截","result":null,"status":false,"statusCode":"-1"}

总结

可以通过filter 对接口请求、返回、异常 全局进行拦截和过滤。