服务器之家

服务器之家 > 正文

PostgreSQL中Slony-I同步复制部署教程

时间：2020-04-30 15:27     来源/作者：一泽涟漪

前言

本文主要介绍了关于PostgreSQL中Slony-I同步复制部署的相关内容，分享出来供大家参考学习，下面话不多说了，来一起看看详细的介绍吧

本次测试环境

 

IP

10.189.102.118

10.189.100.195

10.189.100.226

PGHOME

/usr/local/pgsql

/usr/local/pgsql

/usr/local/pgsql

Role

Origin,Providers

Subscribers,Providers

Subscribers

Database

region_il 

region_il 

region_il 

Table

il_longlive.entity

il_longlive.entity_result

il_longlive.result

il_longlive.result_logbook

il_longlive.entity

il_longlive.entity_result

il_longlive.result

il_longlive.result_logbook

il_longlive.entity

il_longlive.entity_result

il_longlive.result

il_longlive.result_logbook

Sequences

il_longlive.entity_entity_id_seq

il_longlive.entity_result_id_seq

il_longlive.result_id_seq

il_longlive.result_logbook_record_id_seq

il_longlive.entity_entity_id_seq

il_longlive.entity_result_id_seq

il_longlive.result_id_seq

il_longlive.result_logbook_record_id_seq

 

il_longlive.entity_entity_id_seq

il_longlive.entity_result_id_seq

il_longlive.result_id_seq

il_longlive.result_logbook_record_id_seq

 

Slony-I一般要求表有主键,或者唯一键,Slony-I会自动识别主键,如果表没有主键,需要手工指定唯一键,如果没有指定唯一键,则不能复制,下图为Slony-I架构图.

PostgreSQL中Slony-I同步复制部署教程

1. 查看主库表信息

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
region_il=# \d
   List of relations
 Schema |  Name  | Type | Owner
-------------+------------------------------+----------+----------
 il_longlive | entity   | table | postgres
 il_longlive | entity_entity_id_seq  | sequence | postgres
 il_longlive | entity_result  | table | postgres
 il_longlive | entity_result_id_seq  | sequence | postgres
 il_longlive | result   | table | postgres
 il_longlive | result_id_seq  | sequence | postgres
 il_longlive | result_logbook  | table | postgres
 il_longlive | result_logbook_record_id_seq | sequence | postgres
 il_longlive | result_vendor_schema  | table | postgres
 il_longlive | result_vendor_schema_id_seq | sequence | postgres
 il_longlive | snapshot   | table | postgres
 il_longlive | snapshot_id_seq  | sequence | postgres
(12 rows)
 
region_il=# \d entity
     Table "il_longlive.entity"
 Column |  Type  |    Modifiers   
--------------+------------------------+------------------------------------------------------------
 entity_id | integer  | not null default nextval('entity_entity_id_seq'::regclass)
 country_code | character varying(3) |
 state_code | character varying(50) |
 name  | character varying(128) |
 islive | boolean  |
Indexes:
 "entity_pkey" PRIMARY KEY, btree (entity_id)
 
region_il=# select count(*) from entity;
 count
---------
(1 row)

2. 在所有节点安装Slony-I

?
1
2
3
4
5
6
$ wget http://www.slony.info/downloads/2.2/source/slony1-2.2.4.tar.bz2
$ tar -jxf slony1-2.2.4.tar.bz2
$ cd slony1-2.2.4
$ ./configure --prefix=/usr/local/pgsql --with-pgconfigdir=/usr/local/pgsql/bin --with-perltools
$ make
$ make install

3. 在所有节点创建Slony-I使用的专用用户

?
1
$ psql -c "CREATE USER slonyadmin WITH SUPERUSER REPLICATION LOGIN PASSWORD '888888';"

4. 在所有节点设置允许Slony-I用户远程登录

?
1
2
$ grep slonyadmin /usr/local/pgsql/data/pg_hba.conf
host all  slonyadmin  0.0.0.0/0  md5

5. 在所有节点的对应数据库创建plpgsql语言

?
1
$ createlang plpgsql -d region_il

6. 在主节点创建Slony-I配置文件,并复制到所有其他节点

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
$ cat /usr/local/pgsql/etc/slon_tools.conf
if ($ENV{"SLONYNODES"}) {
 require $ENV{"SLONYNODES"};
} else {
 $CLUSTER_NAME = 'region_il';
 $PIDFILE_DIR = '/usr/local/pgsql/data';
 $LOGDIR = '/usr/local/pgsql/data/pg_log';
 $MASTERNODE = 1;
 $DEBUGLEVEL = 0;
    $START_WATCHDOG = 1;
 $SLEEP_TIME = 60;
 $WATCHDOG_VERBOSE = 0;
 
 add_node(node => 1,
  host => '10.189.102.118',
  dbname => 'region_il',
  port => 5432,
  user => 'slonyadmin',
  password => '888888',);
 
 add_node(node => 2,
  parent => 1,
  host => '10.189.100.195',
  dbname => 'region_il',
  port => 5432,
  user => 'slonyadmin',
  password => '888888',);
 
 add_node(node => 3,
  parent => 2,
  host => '10.189.100.226',
  dbname => 'region_il',
  port => 5432,
  user => 'slonyadmin',
  password => '888888',);
}
 
 
 
$SLONY_SETS = {
 "il_longlive_schema" => {
 "set_id" => 1,
 "table_id" => 101,
 "sequence_id" => 101,
 "pkeyedtables" => [
    'il_longlive.entity',
    'il_longlive.entity_result',
    'il_longlive.result',
    'il_longlive.result_logbook',
    ],
 
 "keyedtables" => {},
 
 "sequences" => ['il_longlive.entity_entity_id_seq',
         'il_longlive.entity_result_id_seq',
            'il_longlive.result_id_seq',
            'il_longlive.result_logbook_record_id_seq',
   ],
},
 
};
 
if ($ENV{"SLONYSET"}) {
 require $ENV{"SLONYSET"};
}
 
# Please do not add or change anything below this point.
 1;
$ scp /usr/local/pgsql/etc/slon_tools.conf postgres@10.189.100.195:/usr/local/pgsql/etc/
$ scp /usr/local/pgsql/etc/slon_tools.conf postgres@10.189.100.226:/usr/local/pgsql/etc/

7. 将需要复制的表结构从主节点导出并导入到所有辅节点中

?
1
2
$ pg_dump -U slonyadmin -s -t il_longlive.entity -t il_longlive.entity_result -t il_longlive.result -t il_longlive.result_logbook region_il | psql -h 10.189.100.195 -U slonyadmin region_il
$ pg_dump -U slonyadmin -s -t il_longlive.entity -t il_longlive.entity_result -t il_longlive.result -t il_longlive.result_logbook region_il | psql -h 10.189.100.226 -U slonyadmin region_il

 8. 在主节点初始化Slony-I集群

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
$ /usr/local/pgsql/bin/slonik_init_cluster
 
# INIT CLUSTER
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 init cluster (id = 1, comment = 'Node 1 - region_il@10.189.102.118');
 
# STORE NODE
 store node (id = 2, event node = 1, comment = 'Node 2 - region_il@10.189.100.195');
 store node (id = 3, event node = 1, comment = 'Node 3 - region_il@10.189.100.226');
 echo 'Set up replication nodes';
 
# STORE PATH
 echo 'Next: configure paths for each node/origin';
 store path (server = 1, client = 2, conninfo = 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888');
 store path (server = 1, client = 3, conninfo = 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888');
 store path (server = 2, client = 1, conninfo = 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888');
 store path (server = 2, client = 3, conninfo = 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888');
 store path (server = 3, client = 1, conninfo = 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888');
 store path (server = 3, client = 2, conninfo = 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888');
 echo 'Replication nodes prepared';
 echo 'Please start a slon replication daemon for each node';
$ /usr/local/pgsql/bin/slonik_init_cluster | slonik
<stdin>:7: Possible unsupported PostgreSQL version (90505) 9.5, defaulting to 8.4 support
<stdin>:10: Possible unsupported PostgreSQL version (90505) 9.5, defaulting to 8.4 support
<stdin>:11: Possible unsupported PostgreSQL version (90505) 9.5, defaulting to 8.4 support
<stdin>:12: Set up replication nodes
<stdin>:15: Next: configure paths for each node/origin
<stdin>:22: Replication nodes prepared
<stdin>:23: Please start a slon replication daemon for each node

9. 在所有节点启动slon进程

在主节点启动slon进程

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ slon_start 1
Invoke slon for node 1 - /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node1.pid -s 1000 -d0 region_il 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' > /usr/local/pgsql/data/pg_log/node1/region_il-2017-06-14.log 2>&1 &
Slon successfully started for cluster region_il, node node1
PID [2454]
Start the watchdog process as well...
$ slon_status 1
Slon is running for the 'region_il' cluster on node1.
$ ps -ef | grep slon
postgres 2454 1 0 22:54 pts/1 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node1.pid -s 1000 -d0 region_il host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888
postgres 2455 2454 0 22:54 pts/1 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node1.pid -s 1000 -d0 region_il host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888
postgres 2459 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24667) idle
postgres 2467 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24669) idle
postgres 2468 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24673) idle
postgres 2469 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24675) idle
postgres 2470 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24677) idle
postgres 2471 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24681) idle
postgres 2473 1 0 22:54 pts/1 00:00:00 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node1 60

在辅节点1启动slave进程

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ slon_start 2
Invoke slon for node 2 - /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' > /usr/local/pgsql/data/pg_log/node2/region_il-2017-06-14.log 2>&1 &
Slon successfully started for cluster region_il, node node2
PID [15535]
Start the watchdog process as well...
$ slon_status 2
Slon is running for the 'region_il' cluster on node2.
$ ps -ef | grep slon
postgres 15515 12843 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(38804) idle
postgres 15535 1 0 22:57 pts/0 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888
postgres 15536 15535 0 22:57 pts/0 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888
postgres 15540 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54629) idle
postgres 15546 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54631) idle
postgres 15547 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54633) idle
postgres 15548 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54635) idle
postgres 15549 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54639) idle
postgres 15552 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54643) idle
postgres 15555 1 0 22:57 pts/0 00:00:00 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node2 60
postgres 15559 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(38812) idle

在辅节点2启动slave进程

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ slon_start 3
Invoke slon for node 3 - /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3.pid -s 1000 -d0 region_il 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' > /usr/local/pgsql/data/pg_log/node3/region_il-2017-06-14.log 2>&1 &
Slon successfully started for cluster region_il, node node3
PID [19212]
Start the watchdog process as well...
$ slon_status 3
Slon is running for the 'region_il' cluster on node3.
$ ps -ef | grep slon
postgres 6033 1 0 Jun02 ? 00:00:02 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=ec2t-pgtest-03 dbname=region_il user=slonyadmin port=5432 password=mypna123
postgres 6050 1 0 Jun02 ? 00:00:02 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node2 60
postgres 19059 15237 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(63615) idle
postgres 19109 15237 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(26781) idle
postgres 19212 1 0 23:00 pts/2 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3.pid -s 1000 -d0 region_il host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888
postgres 19213 19212 0 23:00 pts/2 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3.pid -s 1000 -d0 region_il host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888
postgres 19217 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41213) idle
postgres 19225 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41215) idle
postgres 19226 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41219) idle
postgres 19227 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41221) idle
postgres 19228 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41223) idle
postgres 19229 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41227) idle
postgres 19232 1 0 23:00 pts/2 00:00:00 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node3 60
postgres 19234 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(63635) idle
postgres 19235 6033 0 23:00 ? 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=ec2t-pgtest-03 dbname=region_il user=slonyadmin port=5432 password=mypna123
postgres 19239 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(63637) idle

10. 在主节点创建复制集

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
$ /usr/local/pgsql/bin/slonik_create_set 1
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 
# TABLE ADD KEY
 
# CREATE SET
 create set (id = 1, origin = 1, comment = 'Set 1 (il_longlive_schema) for region_il');
 
# SET ADD TABLE
 echo 'Subscription set 1 (il_longlive_schema) created';
 echo 'Adding tables to the subscription set';
 set add table (set id = 1, origin = 1, id = 101,
   full qualified name = 'il_longlive.entity',
   comment = 'Table il_longlive.entity with primary key');
 echo 'Add primary keyed table il_longlive.entity';
 set add table (set id = 1, origin = 1, id = 102,
   full qualified name = 'il_longlive.entity_result',
   comment = 'Table il_longlive.entity_result with primary key');
 echo 'Add primary keyed table il_longlive.entity_result';
 set add table (set id = 1, origin = 1, id = 103,
   full qualified name = 'il_longlive.result',
   comment = 'Table il_longlive.result with primary key');
 echo 'Add primary keyed table il_longlive.result';
 set add table (set id = 1, origin = 1, id = 104,
   full qualified name = 'il_longlive.result_logbook',
   comment = 'Table il_longlive.result_logbook with primary key');
 echo 'Add primary keyed table il_longlive.result_logbook';
 
# SET ADD SEQUENCE
 echo 'Adding sequences to the subscription set';
 set add sequence (set id = 1, origin = 1, id = 101,
   full qualified name = 'il_longlive.entity_entity_id_seq',
   comment = 'Sequence il_longlive.entity_entity_id_seq');
 echo 'Add sequence il_longlive.entity_entity_id_seq';
 set add sequence (set id = 1, origin = 1, id = 102,
   full qualified name = 'il_longlive.entity_result_id_seq',
   comment = 'Sequence il_longlive.entity_result_id_seq');
 echo 'Add sequence il_longlive.entity_result_id_seq';
 set add sequence (set id = 1, origin = 1, id = 103,
   full qualified name = 'il_longlive.result_id_seq',
   comment = 'Sequence il_longlive.result_id_seq');
 echo 'Add sequence il_longlive.result_id_seq';
 set add sequence (set id = 1, origin = 1, id = 104,
   full qualified name = 'il_longlive.result_logbook_record_id_seq',
   comment = 'Sequence il_longlive.result_logbook_record_id_seq');
 echo 'Add sequence il_longlive.result_logbook_record_id_seq';
 echo 'All tables added';
$ /usr/local/pgsql/bin/slonik_create_set 1 | slonik
<stdin>:12: Subscription set 1 (il_longlive_schema) created
<stdin>:13: Adding tables to the subscription set
<stdin>:17: Add primary keyed table il_longlive.entity
<stdin>:21: Add primary keyed table il_longlive.entity_result
<stdin>:25: Add primary keyed table il_longlive.result
<stdin>:29: Add primary keyed table il_longlive.result_logbook
<stdin>:32: Adding sequences to the subscription set
<stdin>:36: Add sequence il_longlive.entity_entity_id_seq
<stdin>:40: Add sequence il_longlive.entity_result_id_seq
<stdin>:44: Add sequence il_longlive.result_id_seq
<stdin>:48: Add sequence il_longlive.result_logbook_record_id_seq
<stdin>:49: All tables added

11. 在主节点创建复制订阅者

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ slonik_subscribe_set 1 2
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 
 subscribe set (id = 1, provider = 1, receiver = 2, forward = yes);
 echo 'Subscribed nodes to set 1';
$ slonik_subscribe_set 1 2 | slonik
<stdin>:7: Subscribed nodes to set 1
$ slonik_subscribe_set 1 3
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 
 subscribe set (id = 1, provider = 2, receiver = 3, forward = yes);
 echo 'Subscribed nodes to set 1';
$ slonik_subscribe_set 1 3 | slonik
<stdin>:7: Subscribed nodes to set 1

12.查看各节点Slony-I日志信息

主节点查看Slony-I日志信息

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
$ cat region_il-2017-06-14.log
2017-06-14 22:54:42 PDT CONFIG main: slon version 2.2.4 starting up
2017-06-14 22:54:42 PDT INFO slon: watchdog process started
2017-06-14 22:54:42 PDT CONFIG slon: watchdog ready - pid = 2454
2017-06-14 22:54:42 PDT CONFIG slon: worker process created - pid = 2455
2017-06-14 22:54:42 PDT CONFIG main: Integer option vac_frequency = 3
2017-06-14 22:54:42 PDT CONFIG main: Integer option log_level = 0
2017-06-14 22:54:42 PDT CONFIG main: Integer option sync_interval = 1000
2017-06-14 22:54:42 PDT CONFIG main: Integer option sync_interval_timeout = 10000
2017-06-14 22:54:42 PDT CONFIG main: Integer option sync_group_maxsize = 20
2017-06-14 22:54:42 PDT CONFIG main: Integer option syslog = 0
2017-06-14 22:54:42 PDT CONFIG main: Integer option quit_sync_provider = 0
2017-06-14 22:54:42 PDT CONFIG main: Integer option remote_listen_timeout = 300
2017-06-14 22:54:42 PDT CONFIG main: Integer option monitor_interval = 500
2017-06-14 22:54:42 PDT CONFIG main: Integer option explain_interval = 0
2017-06-14 22:54:42 PDT CONFIG main: Integer option tcp_keepalive_idle = 0
2017-06-14 22:54:42 PDT CONFIG main: Integer option tcp_keepalive_interval = 0
2017-06-14 22:54:42 PDT CONFIG main: Integer option tcp_keepalive_count = 0
2017-06-14 22:54:42 PDT CONFIG main: Integer option apply_cache_size = 100
2017-06-14 22:54:42 PDT CONFIG main: Boolean option log_pid = 0
2017-06-14 22:54:42 PDT CONFIG main: Boolean option log_timestamp = 1
2017-06-14 22:54:42 PDT CONFIG main: Boolean option tcp_keepalive = 1
2017-06-14 22:54:42 PDT CONFIG main: Boolean option monitor_threads = 1
2017-06-14 22:54:42 PDT CONFIG main: Real option real_placeholder = 0.000000
2017-06-14 22:54:42 PDT CONFIG main: String option cluster_name = region_il
2017-06-14 22:54:42 PDT CONFIG main: String option conn_info = host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888
2017-06-14 22:54:42 PDT CONFIG main: String option pid_file = /usr/local/pgsql/data/region_il_node1.pid
2017-06-14 22:54:42 PDT CONFIG main: String option log_timestamp_format = %Y-%m-%d %H:%M:%S %Z
2017-06-14 22:54:42 PDT CONFIG main: String option archive_dir = [NULL]
2017-06-14 22:54:42 PDT CONFIG main: String option sql_on_connection = [NULL]
2017-06-14 22:54:42 PDT CONFIG main: String option lag_interval = [NULL]
2017-06-14 22:54:42 PDT CONFIG main: String option command_on_logarchive = [NULL]
2017-06-14 22:54:42 PDT CONFIG main: String option syslog_facility = LOCAL0
2017-06-14 22:54:42 PDT CONFIG main: String option syslog_ident = slon
2017-06-14 22:54:42 PDT CONFIG main: String option cleanup_interval = 10 minutes
2017-06-14 22:54:42 PDT CONFIG main: local node id = 1
2017-06-14 22:54:42 PDT INFO main: main process started
2017-06-14 22:54:42 PDT CONFIG main: launching sched_start_mainloop
2017-06-14 22:54:42 PDT CONFIG main: loading current cluster configuration
2017-06-14 22:54:42 PDT CONFIG storeNode: no_id=2 no_comment='Node 2 - region_il@10.189.100.195'
2017-06-14 22:54:42 PDT CONFIG storeNode: no_id=3 no_comment='Node 3 - region_il@10.189.100.226'
2017-06-14 22:54:42 PDT CONFIG storePath: pa_server=2 pa_client=1 pa_conninfo="host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10
2017-06-14 22:54:42 PDT CONFIG storePath: pa_server=3 pa_client=1 pa_conninfo="host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 22:54:42 PDT CONFIG main: last local event sequence = 5000000007
2017-06-14 22:54:42 PDT CONFIG main: configuration complete - starting threads
2017-06-14 22:54:42 PDT INFO localListenThread: thread starts
2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:54:42 PDT CONFIG enableNode: no_id=2
2017-06-14 22:54:42 PDT CONFIG enableNode: no_id=3
2017-06-14 22:54:42 PDT INFO main: running scheduler mainloop
2017-06-14 22:54:42 PDT INFO remoteWorkerThread_3: thread starts
2017-06-14 22:54:42 PDT INFO remoteListenThread_3: thread starts
2017-06-14 22:54:42 PDT CONFIG cleanupThread: thread starts
2017-06-14 22:54:42 PDT INFO syncThread: thread starts
2017-06-14 22:54:42 PDT INFO monitorThread: thread starts
2017-06-14 22:54:42 PDT INFO remoteListenThread_2: thread starts
2017-06-14 22:54:42 PDT INFO remoteWorkerThread_2: thread starts
2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:54:42 PDT CONFIG cleanupThread: bias = 60
2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3
2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3
2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2
2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3
2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 22:57:50 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 22:57:50 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:50 PDT INFO remoteWorkerThread_2: SYNC 5000000003 done in 0.014 seconds
2017-06-14 22:57:58 PDT INFO remoteWorkerThread_2: SYNC 5000000004 done in 0.003 seconds
2017-06-14 22:58:11 PDT INFO remoteWorkerThread_2: SYNC 5000000005 done in 0.002 seconds
2017-06-14 22:58:19 PDT INFO remoteWorkerThread_2: SYNC 5000000006 done in 0.002 seconds
2017-06-14 22:58:27 PDT INFO remoteWorkerThread_2: SYNC 5000000007 done in 0.003 seconds
2017-06-14 22:58:35 PDT INFO remoteWorkerThread_2: SYNC 5000000008 done in 0.002 seconds
2017-06-14 22:58:43 PDT INFO remoteWorkerThread_2: SYNC 5000000009 done in 0.002 seconds
2017-06-14 22:59:01 PDT INFO remoteWorkerThread_2: SYNC 5000000010 done in 0.003 seconds
2017-06-14 22:59:09 PDT INFO remoteWorkerThread_2: SYNC 5000000011 done in 0.002 seconds
2017-06-14 22:59:17 PDT INFO remoteWorkerThread_2: SYNC 5000000012 done in 0.002 seconds
2017-06-14 22:59:25 PDT INFO remoteWorkerThread_2: SYNC 5000000013 done in 0.002 seconds
2017-06-14 22:59:38 PDT INFO remoteWorkerThread_2: SYNC 5000000014 done in 0.003 seconds
2017-06-14 22:59:51 PDT INFO remoteWorkerThread_2: SYNC 5000000015 done in 0.002 seconds
2017-06-14 22:59:54 PDT INFO remoteWorkerThread_2: SYNC 5000000016 done in 0.002 seconds
2017-06-14 23:00:05 PDT INFO remoteWorkerThread_2: SYNC 5000000017 done in 0.002 seconds
2017-06-14 23:00:19 PDT INFO remoteWorkerThread_2: SYNC 5000000018 done in 0.003 seconds
2017-06-14 23:00:32 PDT INFO remoteWorkerThread_2: SYNC 5000000019 done in 0.003 seconds
2017-06-14 23:00:35 PDT INFO remoteWorkerThread_2: SYNC 5000000020 done in 0.003 seconds
2017-06-14 23:00:43 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:00:43 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:43 PDT INFO remoteWorkerThread_3: SYNC 5000000003 done in 0.045 seconds
2017-06-14 23:00:47 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:47 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:47 PDT INFO remoteWorkerThread_2: SYNC 5000000021 done in 0.014 seconds
2017-06-14 23:00:55 PDT INFO remoteWorkerThread_3: SYNC 5000000004 done in 0.002 seconds
2017-06-14 23:00:55 PDT INFO remoteWorkerThread_2: SYNC 5000000022 done in 0.005 seconds
2017-06-14 23:01:08 PDT INFO remoteWorkerThread_3: SYNC 5000000005 done in 0.002 seconds
2017-06-14 23:01:08 PDT INFO remoteWorkerThread_2: SYNC 5000000023 done in 0.005 seconds
2017-06-14 23:01:16 PDT INFO remoteWorkerThread_3: SYNC 5000000006 done in 0.002 seconds
2017-06-14 23:01:20 PDT INFO remoteWorkerThread_3: SYNC 5000000007 done in 0.002 seconds
2017-06-14 23:01:20 PDT INFO remoteWorkerThread_2: SYNC 5000000024 done in 0.005 seconds
2017-06-14 23:01:24 PDT INFO remoteWorkerThread_2: SYNC 5000000025 done in 0.005 seconds
2017-06-14 23:01:38 PDT INFO remoteWorkerThread_3: SYNC 5000000008 done in 0.003 seconds
2017-06-14 23:01:38 PDT INFO remoteWorkerThread_2: SYNC 5000000026 done in 0.005 seconds
2017-06-14 23:01:41 PDT INFO remoteWorkerThread_3: SYNC 5000000009 done in 0.002 seconds
2017-06-14 23:01:44 PDT INFO remoteWorkerThread_2: SYNC 5000000027 done in 0.004 seconds
2017-06-14 23:01:52 PDT INFO remoteWorkerThread_3: SYNC 5000000010 done in 0.003 seconds
2017-06-14 23:01:55 PDT INFO remoteWorkerThread_2: SYNC 5000000028 done in 0.005 seconds
2017-06-14 23:02:08 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:02:08 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:02:08 PDT INFO remoteWorkerThread_2: SYNC 5000000029 done in 0.012 seconds
2017-06-14 23:02:08 PDT INFO remoteWorkerThread_3: SYNC 5000000011 done in 0.016 seconds
2017-06-14 23:02:11 PDT INFO remoteWorkerThread_3: SYNC 5000000012 done in 0.004 seconds
2017-06-14 23:02:19 PDT INFO remoteWorkerThread_2: SYNC 5000000030 done in 0.005 seconds
2017-06-14 23:02:22 PDT INFO remoteWorkerThread_3: SYNC 5000000013 done in 0.005 seconds
2017-06-14 23:02:25 PDT INFO remoteWorkerThread_2: SYNC 5000000031 done in 0.004 seconds
2017-06-14 23:02:38 PDT INFO remoteWorkerThread_2: SYNC 5000000032 done in 0.004 seconds
2017-06-14 23:02:38 PDT INFO remoteWorkerThread_3: SYNC 5000000014 done in 0.005 seconds
2017-06-14 23:02:46 PDT INFO remoteWorkerThread_3: SYNC 5000000015 done in 0.005 seconds
2017-06-14 23:02:50 PDT INFO remoteWorkerThread_3: SYNC 5000000016 done in 0.004 seconds
2017-06-14 23:02:50 PDT INFO remoteWorkerThread_2: SYNC 5000000033 done in 0.005 seconds
2017-06-14 23:02:54 PDT INFO remoteWorkerThread_2: SYNC 5000000034 done in 0.004 seconds
2017-06-14 23:03:08 PDT INFO remoteWorkerThread_3: SYNC 5000000017 done in 0.004 seconds
2017-06-14 23:03:08 PDT INFO remoteWorkerThread_2: SYNC 5000000035 done in 0.005 seconds
2017-06-14 23:03:11 PDT CONFIG storeSet: set_id=1 set_origin=1 set_comment='Set 1 (il_longlive_schema) for region_il'
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_3: SYNC 5000000018 done in 0.005 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_2: SYNC 5000000036 done in 0.005 seconds
2017-06-14 23:03:21 PDT INFO remoteWorkerThread_3: SYNC 5000000019 done in 0.004 seconds
2017-06-14 23:03:29 PDT INFO remoteWorkerThread_2: SYNC 5000000037 done in 0.016 seconds
2017-06-14 23:03:32 PDT INFO remoteWorkerThread_3: SYNC 5000000020 done in 0.012 seconds
2017-06-14 23:03:35 PDT INFO remoteWorkerThread_2: SYNC 5000000038 done in 0.004 seconds
2017-06-14 23:03:48 PDT INFO remoteWorkerThread_2: SYNC 5000000039 done in 0.004 seconds
2017-06-14 23:03:48 PDT INFO remoteWorkerThread_3: SYNC 5000000021 done in 0.005 seconds
2017-06-14 23:03:56 PDT INFO remoteWorkerThread_3: SYNC 5000000022 done in 0.005 seconds
2017-06-14 23:03:57 PDT INFO remoteWorkerThread_2: SYNC 5000000040 done in 0.005 seconds
2017-06-14 23:04:01 PDT INFO remoteWorkerThread_3: SYNC 5000000023 done in 0.005 seconds
2017-06-14 23:04:04 PDT INFO remoteWorkerThread_2: SYNC 5000000041 done in 0.005 seconds
2017-06-14 23:04:17 PDT INFO remoteWorkerThread_2: SYNC 5000000042 done in 0.005 seconds
2017-06-14 23:04:17 PDT INFO remoteWorkerThread_3: SYNC 5000000024 done in 0.005 seconds
2017-06-14 23:04:20 PDT INFO remoteWorkerThread_3: SYNC 5000000025 done in 0.004 seconds
2017-06-14 23:04:23 PDT INFO remoteWorkerThread_2: SYNC 5000000043 done in 0.008 seconds
2017-06-14 23:04:31 PDT INFO remoteWorkerThread_3: SYNC 5000000026 done in 0.005 seconds
2017-06-14 23:04:34 PDT INFO remoteWorkerThread_2: SYNC 5000000044 done in 0.004 seconds
2017-06-14 23:04:42 PDT INFO remoteWorkerThread_3: SYNC 5000000027 done in 0.005 seconds
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2498
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2499
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2500
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2526
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: Logswitch to sl_log_2 initiated
CONTEXT: SQL statement "SELECT "_region_il".logswitch_start()"
PL/pgSQL function _region_il.cleanupevent(interval) line 97 at PERFORM
2017-06-14 23:04:43 PDT INFO cleanupThread: 0.011 seconds for cleanupEvent()
2017-06-14 23:04:45 PDT INFO remoteWorkerThread_2: SYNC 5000000045 done in 0.004 seconds
2017-06-14 23:04:53 PDT INFO remoteWorkerThread_2: SYNC 5000000046 done in 0.005 seconds
2017-06-14 23:04:53 PDT INFO remoteWorkerThread_3: SYNC 5000000028 done in 0.005 seconds
2017-06-14 23:05:06 PDT INFO remoteWorkerThread_3: SYNC 5000000029 done in 0.005 seconds
2017-06-14 23:05:06 PDT INFO remoteWorkerThread_2: SYNC 5000000047 done in 0.005 seconds
2017-06-14 23:05:19 PDT INFO remoteWorkerThread_2: SYNC 5000000048 done in 0.004 seconds
2017-06-14 23:05:19 PDT INFO remoteWorkerThread_3: SYNC 5000000030 done in 0.012 seconds
2017-06-14 23:05:22 PDT INFO remoteWorkerThread_3: SYNC 5000000031 done in 0.004 seconds
2017-06-14 23:05:25 PDT INFO remoteWorkerThread_2: SYNC 5000000049 done in 0.005 seconds
2017-06-14 23:05:38 PDT INFO remoteWorkerThread_2: SYNC 5000000050 done in 0.004 seconds
2017-06-14 23:05:38 PDT INFO remoteWorkerThread_3: SYNC 5000000032 done in 0.005 seconds
2017-06-14 23:05:41 PDT INFO remoteWorkerThread_3: SYNC 5000000033 done in 0.005 seconds
2017-06-14 23:05:45 PDT INFO remoteWorkerThread_2: SYNC 5000000051 done in 0.005 seconds
2017-06-14 23:05:53 PDT INFO remoteWorkerThread_3: SYNC 5000000034 done in 0.004 seconds
2017-06-14 23:05:53 PDT INFO remoteWorkerThread_2: SYNC 5000000052 done in 0.005 seconds
2017-06-14 23:06:06 PDT INFO remoteWorkerThread_3: SYNC 5000000035 done in 0.005 seconds
2017-06-14 23:06:06 PDT INFO remoteWorkerThread_2: SYNC 5000000053 done in 0.005 seconds
2017-06-14 23:06:14 PDT INFO remoteWorkerThread_2: SYNC 5000000054 done in 0.005 seconds
2017-06-14 23:06:14 PDT INFO remoteWorkerThread_3: SYNC 5000000036 done in 0.006 seconds
2017-06-14 23:06:22 PDT INFO remoteWorkerThread_3: SYNC 5000000037 done in 0.005 seconds
2017-06-14 23:06:25 PDT INFO remoteWorkerThread_2: SYNC 5000000055 done in 0.004 seconds
2017-06-14 23:06:30 PDT INFO remoteWorkerThread_3: SYNC 5000000038 done in 0.005 seconds
2017-06-14 23:06:36 PDT INFO remoteWorkerThread_2: SYNC 5000000056 done in 0.005 seconds
2017-06-14 23:06:40 PDT INFO remoteWorkerThread_3: SYNC 5000000039 done in 0.005 seconds
2017-06-14 23:06:48 PDT INFO remoteWorkerThread_2: SYNC 5000000057 done in 0.007 seconds
2017-06-14 23:06:56 PDT INFO remoteWorkerThread_3: SYNC 5000000040 done in 0.004 seconds
2017-06-14 23:06:56 PDT INFO remoteWorkerThread_2: SYNC 5000000058 done in 0.005 seconds
2017-06-14 23:07:09 PDT INFO remoteWorkerThread_3: SYNC 5000000041 done in 0.004 seconds
2017-06-14 23:07:09 PDT INFO remoteWorkerThread_2: SYNC 5000000059 done in 0.005 seconds
2017-06-14 23:07:10 PDT INFO remoteWorkerThread_3: SYNC 5000000042 done in 0.004 seconds
2017-06-14 23:07:19 PDT INFO remoteWorkerThread_2: SYNC 5000000060 done in 0.005 seconds
2017-06-14 23:07:23 PDT INFO remoteWorkerThread_3: SYNC 5000000043 done in 0.015 seconds
2017-06-14 23:07:23 PDT INFO remoteWorkerThread_2: SYNC 5000000061 done in 0.017 seconds
2017-06-14 23:07:31 PDT INFO remoteWorkerThread_3: SYNC 5000000044 done in 0.004 seconds
2017-06-14 23:07:39 PDT INFO remoteWorkerThread_2: SYNC 5000000062 done in 0.013 seconds
2017-06-14 23:07:42 PDT INFO remoteWorkerThread_3: SYNC 5000000045 done in 0.006 seconds
2017-06-14 23:07:43 PDT INFO remoteWorkerThread_2: SYNC 5000000063 done in 0.005 seconds
2017-06-14 23:07:51 PDT INFO remoteWorkerThread_3: SYNC 5000000046 done in 0.008 seconds
2017-06-14 23:07:56 PDT INFO remoteWorkerThread_2: SYNC 5000000064 done in 0.005 seconds
2017-06-14 23:08:02 PDT INFO remoteWorkerThread_3: SYNC 5000000047 done in 0.022 seconds
2017-06-14 23:08:07 PDT INFO remoteWorkerThread_2: SYNC 5000000065 done in 0.005 seconds
2017-06-14 23:08:13 PDT INFO remoteWorkerThread_3: SYNC 5000000048 done in 0.008 seconds
2017-06-14 23:08:18 PDT INFO remoteWorkerThread_2: SYNC 5000000066 done in 0.011 seconds
2017-06-14 23:08:24 PDT INFO remoteWorkerThread_3: SYNC 5000000049 done in 0.004 seconds
2017-06-14 23:08:24 PDT INFO remoteWorkerThread_2: SYNC 5000000067 done in 0.017 seconds
2017-06-14 23:08:35 PDT INFO remoteWorkerThread_3: SYNC 5000000050 done in 0.005 seconds
2017-06-14 23:08:39 PDT INFO remoteWorkerThread_2: SYNC 5000000068 done in 0.005 seconds
2017-06-14 23:08:40 PDT INFO remoteWorkerThread_3: SYNC 5000000051 done in 0.004 seconds
2017-06-14 23:08:49 PDT INFO remoteWorkerThread_2: SYNC 5000000069 done in 0.005 seconds
2017-06-14 23:08:53 PDT INFO remoteWorkerThread_2: SYNC 5000000070 done in 0.005 seconds
2017-06-14 23:08:54 PDT INFO remoteWorkerThread_3: SYNC 5000000052 done in 0.005 seconds
2017-06-14 23:09:02 PDT INFO remoteWorkerThread_3: SYNC 5000000053 done in 0.006 seconds
2017-06-14 23:09:07 PDT INFO remoteWorkerThread_2: SYNC 5000000071 done in 0.005 seconds
2017-06-14 23:09:13 PDT INFO remoteWorkerThread_3: SYNC 5000000054 done in 0.005 seconds
2017-06-14 23:09:18 PDT INFO remoteWorkerThread_2: SYNC 5000000072 done in 0.005 seconds
2017-06-14 23:09:24 PDT INFO remoteWorkerThread_3: SYNC 5000000055 done in 0.005 seconds
2017-06-14 23:09:29 PDT INFO remoteWorkerThread_2: SYNC 5000000073 done in 0.005 seconds
2017-06-14 23:09:33 PDT INFO remoteWorkerThread_2: SYNC 5000000074 done in 0.005 seconds
2017-06-14 23:09:34 PDT INFO remoteWorkerThread_3: SYNC 5000000056 done in 0.005 seconds
2017-06-14 23:09:48 PDT INFO remoteWorkerThread_3: SYNC 5000000057 done in 0.004 seconds
2017-06-14 23:09:48 PDT INFO remoteWorkerThread_2: SYNC 5000000075 done in 0.005 seconds
2017-06-14 23:09:52 PDT INFO remoteWorkerThread_3: SYNC 5000000058 done in 0.005 seconds
2017-06-14 23:09:55 PDT INFO remoteWorkerThread_2: SYNC 5000000076 done in 0.004 seconds
2017-06-14 23:10:00 PDT INFO remoteWorkerThread_3: SYNC 5000000059 done in 0.005 seconds
2017-06-14 23:10:06 PDT INFO remoteWorkerThread_2: SYNC 5000000077 done in 0.005 seconds
2017-06-14 23:10:11 PDT INFO remoteWorkerThread_3: SYNC 5000000060 done in 0.005 seconds
2017-06-14 23:10:17 PDT INFO remoteWorkerThread_2: SYNC 5000000078 done in 0.005 seconds
2017-06-14 23:10:22 PDT INFO remoteWorkerThread_3: SYNC 5000000061 done in 0.005 seconds
2017-06-14 23:10:28 PDT INFO remoteWorkerThread_2: SYNC 5000000079 done in 0.005 seconds
2017-06-14 23:10:33 PDT INFO remoteWorkerThread_3: SYNC 5000000062 done in 0.005 seconds
2017-06-14 23:10:37 PDT INFO remoteWorkerThread_2: SYNC 5000000080 done in 0.005 seconds
2017-06-14 23:10:41 PDT INFO remoteWorkerThread_3: SYNC 5000000063 done in 0.005 seconds
2017-06-14 23:10:46 PDT INFO remoteWorkerThread_2: SYNC 5000000081 done in 0.005 seconds
2017-06-14 23:10:52 PDT INFO remoteWorkerThread_3: SYNC 5000000064 done in 0.004 seconds
2017-06-14 23:10:57 PDT INFO remoteWorkerThread_2: SYNC 5000000082 done in 0.004 seconds
2017-06-14 23:11:03 PDT INFO remoteWorkerThread_3: SYNC 5000000065 done in 0.004 seconds
2017-06-14 23:11:08 PDT INFO remoteWorkerThread_2: SYNC 5000000083 done in 0.005 seconds
2017-06-14 23:11:14 PDT INFO remoteWorkerThread_3: SYNC 5000000066 done in 0.004 seconds
2017-06-14 23:11:19 PDT INFO remoteWorkerThread_2: SYNC 5000000084 done in 0.005 seconds
2017-06-14 23:11:23 PDT INFO remoteWorkerThread_2: SYNC 5000000085 done in 0.005 seconds
2017-06-14 23:11:25 PDT INFO remoteWorkerThread_3: SYNC 5000000067 done in 0.005 seconds
2017-06-14 23:11:34 PDT INFO remoteWorkerThread_2: SYNC 5000000086 done in 0.006 seconds
2017-06-14 23:11:38 PDT INFO remoteWorkerThread_3: SYNC 5000000068 done in 0.005 seconds
2017-06-14 23:11:42 PDT INFO remoteWorkerThread_3: SYNC 5000000069 done in 0.005 seconds
2017-06-14 23:11:46 PDT INFO remoteWorkerThread_2: SYNC 5000000087 done in 0.005 seconds
2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3
2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: connection for provider 3 terminated
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 3
2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3
2017-06-14 23:11:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:11:54 PDT INFO remoteWorkerThread_3: SYNC 5000000070 done in 0.005 seconds
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_3: connection for provider 3 terminated
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 3
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_3: connection for provider 2 terminated
2017-06-14 23:11:54 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 2
2017-06-14 23:11:58 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:11:58 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:11:58 PDT INFO remoteWorkerThread_2: SYNC 5000000088 done in 0.012 seconds
2017-06-14 23:12:02 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:12:02 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:12:02 PDT INFO remoteWorkerThread_3: SYNC 5000000071 done in 0.020 seconds
2017-06-14 23:12:06 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:12:06 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:12:07 PDT INFO remoteWorkerThread_2: SYNC 5000000089 done in 0.278 seconds
2017-06-14 23:12:13 PDT INFO remoteWorkerThread_3: SYNC 5000000072 done in 0.003 seconds
2017-06-14 23:12:24 PDT INFO remoteWorkerThread_3: SYNC 5000000073 done in 0.003 seconds
2017-06-14 23:12:33 PDT INFO remoteWorkerThread_2: SYNC 5000000090 done in 0.005 seconds
2017-06-14 23:12:37 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:12:37 PDT INFO remoteWorkerThread_2: SYNC 5000000091 done in 0.004 seconds
2017-06-14 23:12:37 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:12:38 PDT INFO remoteWorkerThread_3: SYNC 5000000074 done in 0.522 seconds
2017-06-14 23:12:41 PDT INFO remoteWorkerThread_3: SYNC 5000000075 done in 0.005 seconds
2017-06-14 23:12:52 PDT INFO remoteWorkerThread_3: SYNC 5000000076 done in 0.005 seconds
2017-06-14 23:12:55 PDT INFO remoteWorkerThread_2: SYNC 5000000092 done in 0.005 seconds
2017-06-14 23:13:03 PDT INFO remoteWorkerThread_2: SYNC 5000000093 done in 0.016 seconds
2017-06-14 23:13:09 PDT INFO remoteWorkerThread_3: SYNC 5000000077 done in 0.005 seconds
2017-06-14 23:13:10 PDT INFO remoteWorkerThread_3: SYNC 5000000078 done in 0.005 seconds
2017-06-14 23:13:14 PDT INFO remoteWorkerThread_2: SYNC 5000000094 done in 0.005 seconds
2017-06-14 23:13:21 PDT INFO remoteWorkerThread_3: SYNC 5000000079 done in 0.005 seconds
2017-06-14 23:13:25 PDT INFO remoteWorkerThread_2: SYNC 5000000095 done in 0.004 seconds
2017-06-14 23:13:32 PDT INFO remoteWorkerThread_3: SYNC 5000000080 done in 0.005 seconds
2017-06-14 23:13:36 PDT INFO remoteWorkerThread_2: SYNC 5000000096 done in 0.012 seconds
2017-06-14 23:13:43 PDT INFO remoteWorkerThread_3: SYNC 5000000081 done in 0.005 seconds
2017-06-14 23:13:47 PDT INFO remoteWorkerThread_2: SYNC 5000000097 done in 0.089 seconds
2017-06-14 23:13:54 PDT INFO remoteWorkerThread_3: SYNC 5000000082 done in 0.005 seconds
2017-06-14 23:13:58 PDT INFO remoteWorkerThread_2: SYNC 5000000098 done in 0.006 seconds
2017-06-14 23:14:06 PDT INFO remoteWorkerThread_3: SYNC 5000000083 done in 0.005 seconds
2017-06-14 23:14:10 PDT INFO remoteWorkerThread_3: SYNC 5000000084 done in 0.004 seconds
2017-06-14 23:14:11 PDT INFO remoteWorkerThread_2: SYNC 5000000099 done in 0.005 seconds
2017-06-14 23:14:21 PDT INFO remoteWorkerThread_3: SYNC 5000000085 done in 0.015 seconds
2017-06-14 23:14:21 PDT INFO remoteWorkerThread_2: SYNC 5000000100 done in 0.016 seconds
2017-06-14 23:14:32 PDT INFO remoteWorkerThread_3: SYNC 5000000086 done in 0.009 seconds
2017-06-14 23:14:35 PDT INFO remoteWorkerThread_2: SYNC 5000000101 done in 0.005 seconds
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2529
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2552
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2567
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=2568
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: log switch to sl_log_2 complete - truncate sl_log_1
CONTEXT: PL/pgSQL function _region_il.cleanupevent(interval) line 95 at assignment
2017-06-14 23:14:43 PDT INFO cleanupThread: 0.086 seconds for cleanupEvent()
2017-06-14 23:14:49 PDT INFO remoteWorkerThread_3: SYNC 5000000087 done in 0.005 seconds
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_3: SYNC 5000000088 done in 0.005 seconds
2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3
2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: connection for provider 3 terminated
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 3
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: connection for provider 3 terminated
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 3
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: connection for provider 2 terminated
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 2
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=2
2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=3
2017-06-14 23:14:54 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=2
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:14:54 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:14:54 PDT INFO remoteWorkerThread_2: SYNC 5000000102 done in 0.037 seconds
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated
2017-06-14 23:14:54 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2
2017-06-14 23:15:02 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:15:02 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:15:02 PDT INFO remoteWorkerThread_2: SYNC 5000000103 done in 0.013 seconds
2017-06-14 23:15:06 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:15:06 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:15:06 PDT INFO remoteWorkerThread_3: SYNC 5000000089 done in 0.013 seconds
2017-06-14 23:15:14 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:15:14 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:15:17 PDT INFO remoteWorkerThread_3: SYNC 5000000090 done in 2.438 seconds
2017-06-14 23:15:20 PDT INFO remoteWorkerThread_2: SYNC 5000000104 done in 0.003 seconds
2017-06-14 23:15:21 PDT INFO remoteWorkerThread_2: SYNC 5000000105 done in 0.002 seconds
2017-06-14 23:15:25 PDT INFO remoteWorkerThread_3: SYNC 5000000091 done in 0.016 seconds
2017-06-14 23:15:32 PDT INFO remoteWorkerThread_2: SYNC 5000000106 done in 0.002 seconds
2017-06-14 23:15:36 PDT INFO remoteWorkerThread_3: SYNC 5000000092 done in 0.005 seconds
2017-06-14 23:15:43 PDT INFO remoteWorkerThread_2: SYNC 5000000107 done in 0.003 seconds
2017-06-14 23:15:47 PDT INFO remoteWorkerThread_3: SYNC 5000000093 done in 0.006 seconds
2017-06-14 23:15:55 PDT INFO remoteWorkerThread_2: SYNC 5000000108 done in 0.003 seconds
2017-06-14 23:15:55 PDT INFO remoteWorkerThread_3: SYNC 5000000094 done in 0.013 seconds
2017-06-14 23:16:06 PDT INFO remoteWorkerThread_2: SYNC 5000000109 done in 0.003 seconds
2017-06-14 23:16:10 PDT INFO remoteWorkerThread_3: SYNC 5000000095 done in 0.004 seconds
2017-06-14 23:16:11 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:16:11 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:16:11 PDT INFO remoteWorkerThread_2: SYNC 5000000110 done in 0.058 seconds
2017-06-14 23:16:12 PDT INFO remoteWorkerThread_3: SYNC 5000000096 done in 0.012 seconds
2017-06-14 23:16:27 PDT INFO remoteWorkerThread_2: SYNC 5000000111 done in 0.008 seconds
2017-06-14 23:16:31 PDT INFO remoteWorkerThread_2: SYNC 5000000112 done in 0.013 seconds
2017-06-14 23:16:31 PDT INFO remoteWorkerThread_3: SYNC 5000000097 done in 0.037 seconds
2017-06-14 23:16:42 PDT INFO remoteWorkerThread_3: SYNC 5000000098 done in 0.006 seconds
2017-06-14 23:16:42 PDT INFO remoteWorkerThread_2: SYNC 5000000113 done in 0.006 seconds
2017-06-14 23:16:53 PDT INFO remoteWorkerThread_3: SYNC 5000000099 done in 0.005 seconds
2017-06-14 23:16:53 PDT INFO remoteWorkerThread_2: SYNC 5000000114 done in 0.005 seconds
2017-06-14 23:17:04 PDT INFO remoteWorkerThread_3: SYNC 5000000100 done in 0.005 seconds
2017-06-14 23:17:04 PDT INFO remoteWorkerThread_2: SYNC 5000000115 done in 0.006 seconds
2017-06-14 23:17:08 PDT INFO remoteWorkerThread_3: SYNC 5000000101 done in 0.005 seconds
2017-06-14 23:17:14 PDT INFO remoteWorkerThread_2: SYNC 5000000116 done in 0.004 seconds
2017-06-14 23:17:19 PDT INFO remoteWorkerThread_3: SYNC 5000000102 done in 0.005 seconds
2017-06-14 23:17:25 PDT INFO remoteWorkerThread_2: SYNC 5000000117 done in 0.005 seconds
2017-06-14 23:17:30 PDT INFO remoteWorkerThread_3: SYNC 5000000103 done in 0.005 seconds
2017-06-14 23:17:34 PDT INFO remoteWorkerThread_2: SYNC 5000000118 done in 0.013 seconds
2017-06-14 23:17:38 PDT INFO remoteWorkerThread_3: SYNC 5000000104 done in 0.012 seconds
2017-06-14 23:17:43 PDT INFO remoteWorkerThread_2: SYNC 5000000119 done in 0.005 seconds
2017-06-14 23:17:49 PDT INFO remoteWorkerThread_3: SYNC 5000000105 done in 0.004 seconds
2017-06-14 23:17:53 PDT INFO remoteWorkerThread_2: SYNC 5000000120 done in 0.005 seconds
2017-06-14 23:18:01 PDT INFO remoteWorkerThread_2: SYNC 5000000121 done in 0.005 seconds
2017-06-14 23:18:01 PDT INFO remoteWorkerThread_3: SYNC 5000000106 done in 0.006 seconds
2017-06-14 23:18:14 PDT INFO remoteWorkerThread_2: SYNC 5000000122 done in 0.005 seconds
2017-06-14 23:18:14 PDT INFO remoteWorkerThread_3: SYNC 5000000107 done in 0.006 seconds
2017-06-14 23:18:27 PDT INFO remoteWorkerThread_2: SYNC 5000000123 done in 0.005 seconds
2017-06-14 23:18:27 PDT INFO remoteWorkerThread_3: SYNC 5000000108 done in 0.006 seconds
2017-06-14 23:18:30 PDT INFO remoteWorkerThread_3: SYNC 5000000109 done in 0.005 seconds
2017-06-14 23:18:33 PDT INFO remoteWorkerThread_2: SYNC 5000000124 done in 0.004 seconds
2017-06-14 23:18:38 PDT INFO remoteWorkerThread_3: SYNC 5000000110 done in 0.004 seconds
2017-06-14 23:18:44 PDT INFO remoteWorkerThread_2: SYNC 5000000125 done in 0.004 seconds
2017-06-14 23:18:49 PDT INFO remoteWorkerThread_3: SYNC 5000000111 done in 0.005 seconds
2017-06-14 23:18:55 PDT INFO remoteWorkerThread_2: SYNC 5000000126 done in 0.005 seconds
2017-06-14 23:19:00 PDT INFO remoteWorkerThread_3: SYNC 5000000112 done in 0.011 seconds
2017-06-14 23:19:04 PDT INFO remoteWorkerThread_2: SYNC 5000000127 done in 0.005 seconds
2017-06-14 23:19:08 PDT INFO remoteWorkerThread_3: SYNC 5000000113 done in 0.005 seconds
2017-06-14 23:19:12 PDT INFO remoteWorkerThread_2: SYNC 5000000128 done in 0.005 seconds
2017-06-14 23:19:20 PDT INFO remoteWorkerThread_3: SYNC 5000000114 done in 0.004 seconds
2017-06-14 23:19:21 PDT INFO remoteWorkerThread_2: SYNC 5000000129 done in 0.004 seconds
2017-06-14 23:19:34 PDT INFO remoteWorkerThread_3: SYNC 5000000115 done in 0.005 seconds
2017-06-14 23:19:34 PDT INFO remoteWorkerThread_2: SYNC 5000000130 done in 0.006 seconds
2017-06-14 23:19:47 PDT INFO remoteWorkerThread_2: SYNC 5000000131 done in 0.005 seconds
2017-06-14 23:19:47 PDT INFO remoteWorkerThread_3: SYNC 5000000116 done in 0.005 seconds
2017-06-14 23:19:48 PDT INFO remoteWorkerThread_3: SYNC 5000000117 done in 0.005 seconds
2017-06-14 23:19:51 PDT INFO remoteWorkerThread_2: SYNC 5000000132 done in 0.005 seconds
2017-06-14 23:20:02 PDT INFO remoteWorkerThread_3: SYNC 5000000118 done in 0.005 seconds
2017-06-14 23:20:02 PDT INFO remoteWorkerThread_2: SYNC 5000000133 done in 0.006 seconds
2017-06-14 23:20:16 PDT INFO remoteWorkerThread_3: SYNC 5000000119 done in 0.005 seconds
2017-06-14 23:20:16 PDT INFO remoteWorkerThread_2: SYNC 5000000134 done in 0.006 seconds
2017-06-14 23:20:19 PDT INFO remoteWorkerThread_3: SYNC 5000000120 done in 0.005 seconds
2017-06-14 23:20:22 PDT INFO remoteWorkerThread_2: SYNC 5000000135 done in 0.005 seconds
2017-06-14 23:20:35 PDT INFO remoteWorkerThread_3: SYNC 5000000121 done in 0.005 seconds
2017-06-14 23:20:35 PDT INFO remoteWorkerThread_2: SYNC 5000000136 done in 0.006 seconds
2017-06-14 23:20:38 PDT INFO remoteWorkerThread_3: SYNC 5000000122 done in 0.005 seconds
2017-06-14 23:20:41 PDT INFO remoteWorkerThread_2: SYNC 5000000137 done in 0.005 seconds

辅节点1查看Slony-I日志信息

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
$ cat /usr/local/pgsql/data/pg_log/node2/region_il-2017-06-14.log
2017-06-14 22:57:41 PDT CONFIG main: slon version 2.2.4 starting up
2017-06-14 22:57:41 PDT INFO slon: watchdog process started
2017-06-14 22:57:41 PDT CONFIG slon: watchdog ready - pid = 15535
2017-06-14 22:57:41 PDT CONFIG slon: worker process created - pid = 15536
2017-06-14 22:57:41 PDT CONFIG main: Integer option vac_frequency = 3
2017-06-14 22:57:41 PDT CONFIG main: Integer option log_level = 0
2017-06-14 22:57:41 PDT CONFIG main: Integer option sync_interval = 1000
2017-06-14 22:57:41 PDT CONFIG main: Integer option sync_interval_timeout = 10000
2017-06-14 22:57:41 PDT CONFIG main: Integer option sync_group_maxsize = 20
2017-06-14 22:57:41 PDT CONFIG main: Integer option syslog = 0
2017-06-14 22:57:41 PDT CONFIG main: Integer option quit_sync_provider = 0
2017-06-14 22:57:41 PDT CONFIG main: Integer option remote_listen_timeout = 300
2017-06-14 22:57:41 PDT CONFIG main: Integer option monitor_interval = 500
2017-06-14 22:57:41 PDT CONFIG main: Integer option explain_interval = 0
2017-06-14 22:57:41 PDT CONFIG main: Integer option tcp_keepalive_idle = 0
2017-06-14 22:57:41 PDT CONFIG main: Integer option tcp_keepalive_interval = 0
2017-06-14 22:57:41 PDT CONFIG main: Integer option tcp_keepalive_count = 0
2017-06-14 22:57:41 PDT CONFIG main: Integer option apply_cache_size = 100
2017-06-14 22:57:41 PDT CONFIG main: Boolean option log_pid = 0
2017-06-14 22:57:41 PDT CONFIG main: Boolean option log_timestamp = 1
2017-06-14 22:57:41 PDT CONFIG main: Boolean option tcp_keepalive = 1
2017-06-14 22:57:41 PDT CONFIG main: Boolean option monitor_threads = 1
2017-06-14 22:57:41 PDT CONFIG main: Real option real_placeholder = 0.000000
2017-06-14 22:57:41 PDT CONFIG main: String option cluster_name = region_il
2017-06-14 22:57:41 PDT CONFIG main: String option conn_info = host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888
2017-06-14 22:57:41 PDT CONFIG main: String option pid_file = /usr/local/pgsql/data/region_il_node2.pid
2017-06-14 22:57:41 PDT CONFIG main: String option log_timestamp_format = %Y-%m-%d %H:%M:%S %Z
2017-06-14 22:57:41 PDT CONFIG main: String option archive_dir = [NULL]
2017-06-14 22:57:41 PDT CONFIG main: String option sql_on_connection = [NULL]
2017-06-14 22:57:41 PDT CONFIG main: String option lag_interval = [NULL]
2017-06-14 22:57:41 PDT CONFIG main: String option command_on_logarchive = [NULL]
2017-06-14 22:57:41 PDT CONFIG main: String option syslog_facility = LOCAL0
2017-06-14 22:57:41 PDT CONFIG main: String option syslog_ident = slon
2017-06-14 22:57:41 PDT CONFIG main: String option cleanup_interval = 10 minutes
2017-06-14 22:57:41 PDT CONFIG main: local node id = 2
2017-06-14 22:57:41 PDT INFO main: main process started
2017-06-14 22:57:41 PDT CONFIG main: launching sched_start_mainloop
2017-06-14 22:57:41 PDT CONFIG main: loading current cluster configuration
2017-06-14 22:57:41 PDT CONFIG storeNode: no_id=1 no_comment='Node 1 - region_il@10.189.102.118'
2017-06-14 22:57:41 PDT CONFIG storeNode: no_id=3 no_comment='<event pending>'
2017-06-14 22:57:41 PDT CONFIG storePath: pa_server=1 pa_client=2 pa_conninfo="host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10
2017-06-14 22:57:41 PDT CONFIG storePath: pa_server=3 pa_client=2 pa_conninfo="host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG main: last local event sequence = 5000000002
2017-06-14 22:57:41 PDT CONFIG main: configuration complete - starting threads
2017-06-14 22:57:41 PDT INFO localListenThread: thread starts
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT CONFIG enableNode: no_id=1
2017-06-14 22:57:41 PDT INFO main: running scheduler mainloop
2017-06-14 22:57:41 PDT CONFIG cleanupThread: thread starts
2017-06-14 22:57:41 PDT INFO syncThread: thread starts
2017-06-14 22:57:41 PDT INFO monitorThread: thread starts
2017-06-14 22:57:41 PDT INFO remoteListenThread_1: thread starts
2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: thread starts
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT CONFIG cleanupThread: bias = 60
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG storeNode: no_id=3 no_comment='Node 3 - region_il@10.189.100.226' - update node
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG enableNode: no_id=3
2017-06-14 22:57:41 PDT INFO remoteListenThread_3: thread starts
2017-06-14 22:57:41 PDT INFO remoteWorkerThread_3: thread starts
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: SYNC 5000000008 done in 0.015 seconds
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: SYNC 5000000010 done in 0.002 seconds
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1
2017-06-14 22:57:41 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=3
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: SYNC 5000000014 done in 0.011 seconds
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1
2017-06-14 22:57:41 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 22:57:41 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: SYNC 5000000022 done in 0.012 seconds
2017-06-14 22:57:41 PDT INFO remoteWorkerThread_1: SYNC 5000000025 done in 0.002 seconds
2017-06-14 22:57:45 PDT INFO remoteWorkerThread_1: SYNC 5000000026 done in 0.003 seconds
2017-06-14 22:57:56 PDT INFO remoteWorkerThread_1: SYNC 5000000027 done in 0.003 seconds
2017-06-14 22:58:10 PDT INFO remoteWorkerThread_1: SYNC 5000000028 done in 0.003 seconds
2017-06-14 22:58:14 PDT INFO remoteWorkerThread_1: SYNC 5000000029 done in 0.003 seconds
2017-06-14 22:58:28 PDT INFO remoteWorkerThread_1: SYNC 5000000030 done in 0.003 seconds
2017-06-14 22:58:39 PDT INFO remoteWorkerThread_1: SYNC 5000000031 done in 0.003 seconds
2017-06-14 22:58:43 PDT INFO remoteWorkerThread_1: SYNC 5000000032 done in 0.003 seconds
2017-06-14 22:59:01 PDT INFO remoteWorkerThread_1: SYNC 5000000033 done in 0.003 seconds
2017-06-14 22:59:05 PDT INFO remoteWorkerThread_1: SYNC 5000000034 done in 0.003 seconds
2017-06-14 22:59:19 PDT INFO remoteWorkerThread_1: SYNC 5000000035 done in 0.003 seconds
2017-06-14 22:59:23 PDT INFO remoteWorkerThread_1: SYNC 5000000036 done in 0.002 seconds
2017-06-14 22:59:41 PDT INFO remoteWorkerThread_1: SYNC 5000000037 done in 0.003 seconds
2017-06-14 22:59:45 PDT INFO remoteWorkerThread_1: SYNC 5000000038 done in 0.002 seconds
2017-06-14 22:59:59 PDT INFO remoteWorkerThread_1: SYNC 5000000039 done in 0.003 seconds
2017-06-14 23:00:03 PDT INFO remoteWorkerThread_1: SYNC 5000000040 done in 0.003 seconds
2017-06-14 23:00:21 PDT INFO remoteWorkerThread_1: SYNC 5000000041 done in 0.003 seconds
2017-06-14 23:00:25 PDT INFO remoteWorkerThread_1: SYNC 5000000042 done in 0.002 seconds
2017-06-14 23:00:39 PDT INFO remoteWorkerThread_1: SYNC 5000000043 done in 0.003 seconds
2017-06-14 23:00:43 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:00:43 PDT INFO remoteWorkerThread_1: SYNC 5000000044 done in 0.003 seconds
2017-06-14 23:00:43 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:43 PDT INFO remoteWorkerThread_3: SYNC 5000000003 done in 0.014 seconds
2017-06-14 23:00:56 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:00:56 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:56 PDT INFO remoteWorkerThread_3: SYNC 5000000004 done in 0.022 seconds
2017-06-14 23:01:00 PDT INFO remoteWorkerThread_3: SYNC 5000000005 done in 0.013 seconds
2017-06-14 23:01:01 PDT INFO remoteWorkerThread_1: SYNC 5000000045 done in 0.003 seconds
2017-06-14 23:01:04 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:01:04 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:01:04 PDT INFO remoteWorkerThread_1: SYNC 5000000046 done in 0.014 seconds
2017-06-14 23:01:15 PDT INFO remoteWorkerThread_3: SYNC 5000000006 done in 0.005 seconds
2017-06-14 23:01:15 PDT INFO remoteWorkerThread_1: SYNC 5000000047 done in 0.006 seconds
2017-06-14 23:01:19 PDT INFO remoteWorkerThread_3: SYNC 5000000007 done in 0.004 seconds
2017-06-14 23:01:26 PDT INFO remoteWorkerThread_1: SYNC 5000000048 done in 0.005 seconds
2017-06-14 23:01:30 PDT INFO remoteWorkerThread_3: SYNC 5000000008 done in 0.005 seconds
2017-06-14 23:01:37 PDT INFO remoteWorkerThread_1: SYNC 5000000049 done in 0.005 seconds
2017-06-14 23:01:41 PDT INFO remoteWorkerThread_3: SYNC 5000000009 done in 0.005 seconds
2017-06-14 23:01:49 PDT INFO remoteWorkerThread_1: SYNC 5000000050 done in 0.013 seconds
2017-06-14 23:01:53 PDT INFO remoteWorkerThread_3: SYNC 5000000010 done in 0.005 seconds
2017-06-14 23:01:53 PDT INFO remoteWorkerThread_1: SYNC 5000000051 done in 0.006 seconds
2017-06-14 23:02:01 PDT INFO remoteWorkerThread_3: SYNC 5000000011 done in 0.006 seconds
2017-06-14 23:02:07 PDT INFO remoteWorkerThread_1: SYNC 5000000052 done in 0.005 seconds
2017-06-14 23:02:11 PDT INFO remoteWorkerThread_3: SYNC 5000000012 done in 0.005 seconds
2017-06-14 23:02:15 PDT INFO remoteWorkerThread_1: SYNC 5000000053 done in 0.005 seconds
2017-06-14 23:02:19 PDT INFO remoteWorkerThread_3: SYNC 5000000013 done in 0.005 seconds
2017-06-14 23:02:26 PDT INFO remoteWorkerThread_1: SYNC 5000000054 done in 0.005 seconds
2017-06-14 23:02:30 PDT INFO remoteWorkerThread_3: SYNC 5000000014 done in 0.005 seconds
2017-06-14 23:02:37 PDT INFO remoteWorkerThread_1: SYNC 5000000055 done in 0.005 seconds
2017-06-14 23:02:41 PDT INFO remoteWorkerThread_3: SYNC 5000000015 done in 0.005 seconds
2017-06-14 23:02:45 PDT INFO remoteWorkerThread_1: SYNC 5000000056 done in 0.006 seconds
2017-06-14 23:02:49 PDT INFO remoteWorkerThread_3: SYNC 5000000016 done in 0.006 seconds
2017-06-14 23:02:55 PDT INFO remoteWorkerThread_1: SYNC 5000000057 done in 0.005 seconds
2017-06-14 23:03:00 PDT INFO remoteWorkerThread_3: SYNC 5000000017 done in 0.005 seconds
2017-06-14 23:03:06 PDT INFO remoteWorkerThread_1: SYNC 5000000058 done in 0.009 seconds
2017-06-14 23:03:11 PDT INFO remoteWorkerThread_3: SYNC 5000000018 done in 0.004 seconds
2017-06-14 23:03:17 PDT CONFIG storeSet: set_id=1 set_origin=1 set_comment='Set 1 (il_longlive_schema) for region_il'
2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated
2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1
2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: connection for provider 3 terminated
2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 3
2017-06-14 23:03:17 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:03:17 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000060 done in 0.014 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000062 done in 0.002 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000064 done in 0.002 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000066 done in 0.002 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000068 done in 0.002 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000070 done in 0.002 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000072 done in 0.002 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000074 done in 0.002 seconds
2017-06-14 23:03:17 PDT INFO remoteWorkerThread_1: SYNC 5000000076 done in 0.002 seconds
2017-06-14 23:03:21 PDT INFO remoteWorkerThread_3: SYNC 5000000019 done in 0.005 seconds
2017-06-14 23:03:25 PDT INFO remoteWorkerThread_1: SYNC 5000000077 done in 0.002 seconds
2017-06-14 23:03:29 PDT INFO remoteWorkerThread_3: SYNC 5000000020 done in 0.005 seconds
2017-06-14 23:03:36 PDT INFO remoteWorkerThread_1: SYNC 5000000078 done in 0.003 seconds
2017-06-14 23:03:40 PDT INFO remoteWorkerThread_3: SYNC 5000000021 done in 0.005 seconds
2017-06-14 23:03:44 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:03:44 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:03:44 PDT INFO remoteWorkerThread_1: SYNC 5000000079 done in 0.015 seconds
2017-06-14 23:03:54 PDT INFO remoteWorkerThread_1: SYNC 5000000080 done in 0.005 seconds
2017-06-14 23:03:58 PDT INFO remoteWorkerThread_3: SYNC 5000000022 done in 0.005 seconds
2017-06-14 23:04:01 PDT INFO remoteWorkerThread_3: SYNC 5000000023 done in 0.005 seconds
2017-06-14 23:04:06 PDT INFO remoteWorkerThread_1: SYNC 5000000081 done in 0.005 seconds
2017-06-14 23:04:12 PDT INFO remoteWorkerThread_3: SYNC 5000000024 done in 0.005 seconds
2017-06-14 23:04:16 PDT INFO remoteWorkerThread_1: SYNC 5000000082 done in 0.005 seconds
2017-06-14 23:04:20 PDT INFO remoteWorkerThread_3: SYNC 5000000025 done in 0.005 seconds
2017-06-14 23:04:24 PDT INFO remoteWorkerThread_1: SYNC 5000000083 done in 0.005 seconds
2017-06-14 23:04:32 PDT INFO remoteWorkerThread_3: SYNC 5000000026 done in 0.005 seconds
2017-06-14 23:04:33 PDT INFO remoteWorkerThread_1: SYNC 5000000084 done in 0.004 seconds
2017-06-14 23:04:44 PDT INFO remoteWorkerThread_3: SYNC 5000000027 done in 0.005 seconds
2017-06-14 23:04:44 PDT INFO remoteWorkerThread_1: SYNC 5000000085 done in 0.006 seconds
2017-06-14 23:04:54 PDT INFO remoteWorkerThread_3: SYNC 5000000028 done in 0.005 seconds
2017-06-14 23:04:58 PDT INFO remoteWorkerThread_1: SYNC 5000000086 done in 0.005 seconds
2017-06-14 23:05:01 PDT INFO remoteWorkerThread_3: SYNC 5000000029 done in 0.006 seconds
2017-06-14 23:05:05 PDT INFO remoteWorkerThread_1: SYNC 5000000087 done in 0.005 seconds
2017-06-14 23:05:13 PDT INFO remoteWorkerThread_3: SYNC 5000000030 done in 0.005 seconds
2017-06-14 23:05:14 PDT INFO remoteWorkerThread_1: SYNC 5000000088 done in 0.005 seconds
2017-06-14 23:05:25 PDT INFO remoteWorkerThread_3: SYNC 5000000031 done in 0.005 seconds
2017-06-14 23:05:25 PDT INFO remoteWorkerThread_1: SYNC 5000000089 done in 0.006 seconds
2017-06-14 23:05:29 PDT INFO remoteWorkerThread_3: SYNC 5000000032 done in 0.005 seconds
2017-06-14 23:05:36 PDT INFO remoteWorkerThread_1: SYNC 5000000090 done in 0.006 seconds
2017-06-14 23:05:40 PDT INFO remoteWorkerThread_3: SYNC 5000000033 done in 0.004 seconds
2017-06-14 23:05:47 PDT INFO remoteWorkerThread_1: SYNC 5000000091 done in 0.005 seconds
2017-06-14 23:05:51 PDT INFO remoteWorkerThread_3: SYNC 5000000034 done in 0.004 seconds
2017-06-14 23:05:58 PDT INFO remoteWorkerThread_1: SYNC 5000000092 done in 0.005 seconds
2017-06-14 23:06:02 PDT INFO remoteWorkerThread_3: SYNC 5000000035 done in 0.005 seconds
2017-06-14 23:06:09 PDT INFO remoteWorkerThread_1: SYNC 5000000093 done in 0.005 seconds
2017-06-14 23:06:13 PDT INFO remoteWorkerThread_1: SYNC 5000000094 done in 0.005 seconds
2017-06-14 23:06:13 PDT INFO remoteWorkerThread_3: SYNC 5000000036 done in 0.004 seconds
2017-06-14 23:06:21 PDT INFO remoteWorkerThread_3: SYNC 5000000037 done in 0.004 seconds
2017-06-14 23:06:27 PDT INFO remoteWorkerThread_1: SYNC 5000000095 done in 0.004 seconds
2017-06-14 23:06:31 PDT INFO remoteWorkerThread_3: SYNC 5000000038 done in 0.005 seconds
2017-06-14 23:06:35 PDT INFO remoteWorkerThread_1: SYNC 5000000096 done in 0.005 seconds
2017-06-14 23:06:39 PDT INFO remoteWorkerThread_3: SYNC 5000000039 done in 0.004 seconds
2017-06-14 23:06:46 PDT INFO remoteWorkerThread_1: SYNC 5000000097 done in 0.005 seconds
2017-06-14 23:06:50 PDT INFO remoteWorkerThread_3: SYNC 5000000040 done in 0.004 seconds
2017-06-14 23:06:57 PDT INFO remoteWorkerThread_1: SYNC 5000000098 done in 0.005 seconds
2017-06-14 23:07:01 PDT INFO remoteWorkerThread_3: SYNC 5000000041 done in 0.005 seconds
2017-06-14 23:07:08 PDT INFO remoteWorkerThread_1: SYNC 5000000099 done in 0.004 seconds
2017-06-14 23:07:12 PDT INFO remoteWorkerThread_3: SYNC 5000000042 done in 0.005 seconds
2017-06-14 23:07:13 PDT INFO remoteWorkerThread_1: SYNC 5000000100 done in 0.004 seconds
2017-06-14 23:07:22 PDT INFO remoteWorkerThread_3: SYNC 5000000043 done in 0.005 seconds
2017-06-14 23:07:27 PDT INFO remoteWorkerThread_1: SYNC 5000000101 done in 0.030 seconds
2017-06-14 23:07:31 PDT INFO remoteWorkerThread_3: SYNC 5000000044 done in 0.005 seconds
2017-06-14 23:07:35 PDT INFO remoteWorkerThread_1: SYNC 5000000102 done in 0.004 seconds
2017-06-14 23:07:40 PDT INFO remoteWorkerThread_3: SYNC 5000000045 done in 0.009 seconds
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15654
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15655
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15658
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: Logswitch to sl_log_2 initiated
CONTEXT: SQL statement "SELECT "_region_il".logswitch_start()"
PL/pgSQL function _region_il.cleanupevent(interval) line 97 at PERFORM
2017-06-14 23:07:42 PDT INFO cleanupThread: 0.019 seconds for cleanupEvent()
2017-06-14 23:07:46 PDT INFO remoteWorkerThread_1: SYNC 5000000103 done in 0.009 seconds
2017-06-14 23:07:51 PDT INFO remoteWorkerThread_3: SYNC 5000000046 done in 0.005 seconds
2017-06-14 23:07:57 PDT INFO remoteWorkerThread_1: SYNC 5000000104 done in 0.010 seconds
2017-06-14 23:08:02 PDT INFO remoteWorkerThread_3: SYNC 5000000047 done in 0.009 seconds
2017-06-14 23:08:08 PDT INFO remoteWorkerThread_1: SYNC 5000000105 done in 0.012 seconds
2017-06-14 23:08:13 PDT INFO remoteWorkerThread_1: SYNC 5000000106 done in 0.008 seconds
2017-06-14 23:08:13 PDT INFO remoteWorkerThread_3: SYNC 5000000048 done in 0.013 seconds
2017-06-14 23:08:24 PDT INFO remoteWorkerThread_3: SYNC 5000000049 done in 0.009 seconds
2017-06-14 23:08:24 PDT INFO remoteWorkerThread_1: SYNC 5000000107 done in 0.010 seconds
2017-06-14 23:08:35 PDT INFO remoteWorkerThread_3: SYNC 5000000050 done in 0.005 seconds
2017-06-14 23:08:39 PDT INFO remoteWorkerThread_1: SYNC 5000000108 done in 0.006 seconds
2017-06-14 23:08:39 PDT INFO remoteWorkerThread_3: SYNC 5000000051 done in 0.005 seconds
2017-06-14 23:08:43 PDT INFO remoteWorkerThread_1: SYNC 5000000109 done in 0.005 seconds
2017-06-14 23:08:54 PDT INFO remoteWorkerThread_3: SYNC 5000000052 done in 0.006 seconds
2017-06-14 23:08:54 PDT INFO remoteWorkerThread_1: SYNC 5000000110 done in 0.006 seconds
2017-06-14 23:09:04 PDT INFO remoteWorkerThread_3: SYNC 5000000053 done in 0.005 seconds
2017-06-14 23:09:08 PDT INFO remoteWorkerThread_1: SYNC 5000000111 done in 0.005 seconds
2017-06-14 23:09:11 PDT INFO remoteWorkerThread_3: SYNC 5000000054 done in 0.005 seconds
2017-06-14 23:09:16 PDT INFO remoteWorkerThread_1: SYNC 5000000112 done in 0.005 seconds
2017-06-14 23:09:22 PDT INFO remoteWorkerThread_3: SYNC 5000000055 done in 0.005 seconds
2017-06-14 23:09:27 PDT INFO remoteWorkerThread_1: SYNC 5000000113 done in 0.005 seconds
2017-06-14 23:09:33 PDT INFO remoteWorkerThread_3: SYNC 5000000056 done in 0.005 seconds
2017-06-14 23:09:34 PDT INFO remoteWorkerThread_1: SYNC 5000000114 done in 0.005 seconds
2017-06-14 23:09:45 PDT INFO remoteWorkerThread_1: SYNC 5000000115 done in 0.005 seconds
2017-06-14 23:09:48 PDT INFO remoteWorkerThread_3: SYNC 5000000057 done in 0.005 seconds
2017-06-14 23:09:49 PDT INFO remoteWorkerThread_3: SYNC 5000000058 done in 0.005 seconds
2017-06-14 23:09:59 PDT INFO remoteWorkerThread_1: SYNC 5000000116 done in 0.005 seconds
2017-06-14 23:10:00 PDT INFO remoteWorkerThread_3: SYNC 5000000059 done in 0.005 seconds
2017-06-14 23:10:04 PDT INFO remoteWorkerThread_1: SYNC 5000000117 done in 0.005 seconds
2017-06-14 23:10:10 PDT INFO remoteWorkerThread_3: SYNC 5000000060 done in 0.005 seconds
2017-06-14 23:10:14 PDT INFO remoteWorkerThread_1: SYNC 5000000118 done in 0.005 seconds
2017-06-14 23:10:22 PDT INFO remoteWorkerThread_3: SYNC 5000000061 done in 0.005 seconds
2017-06-14 23:10:28 PDT INFO remoteWorkerThread_1: SYNC 5000000119 done in 0.005 seconds
2017-06-14 23:10:33 PDT INFO remoteWorkerThread_3: SYNC 5000000062 done in 0.005 seconds
2017-06-14 23:10:39 PDT INFO remoteWorkerThread_1: SYNC 5000000120 done in 0.004 seconds
2017-06-14 23:10:43 PDT INFO remoteWorkerThread_3: SYNC 5000000063 done in 0.005 seconds
2017-06-14 23:10:43 PDT INFO remoteWorkerThread_1: SYNC 5000000121 done in 0.005 seconds
2017-06-14 23:10:52 PDT INFO remoteWorkerThread_3: SYNC 5000000064 done in 0.005 seconds
2017-06-14 23:10:57 PDT INFO remoteWorkerThread_1: SYNC 5000000122 done in 0.004 seconds
2017-06-14 23:11:03 PDT INFO remoteWorkerThread_3: SYNC 5000000065 done in 0.005 seconds
2017-06-14 23:11:08 PDT INFO remoteWorkerThread_1: SYNC 5000000123 done in 0.005 seconds
2017-06-14 23:11:14 PDT INFO remoteWorkerThread_3: SYNC 5000000066 done in 0.005 seconds
2017-06-14 23:11:19 PDT INFO remoteWorkerThread_1: SYNC 5000000124 done in 0.005 seconds
2017-06-14 23:11:25 PDT INFO remoteWorkerThread_3: SYNC 5000000067 done in 0.005 seconds
2017-06-14 23:11:30 PDT INFO remoteWorkerThread_1: SYNC 5000000125 done in 0.005 seconds
2017-06-14 23:11:34 PDT INFO remoteWorkerThread_1: SYNC 5000000126 done in 0.005 seconds
2017-06-14 23:11:36 PDT INFO remoteWorkerThread_3: SYNC 5000000068 done in 0.005 seconds
2017-06-14 23:11:40 PDT INFO remoteWorkerThread_3: SYNC 5000000069 done in 0.004 seconds
2017-06-14 23:11:45 PDT INFO remoteWorkerThread_1: SYNC 5000000127 done in 0.005 seconds
2017-06-14 23:11:51 PDT INFO remoteWorkerThread_3: SYNC 5000000070 done in 0.004 seconds
2017-06-14 23:11:56 PDT CONFIG storeSubscribe: sub_set=1 sub_provider=1 sub_forward='t'
2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1
2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=3
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: connection for provider 3 terminated
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 3
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_3: connection for provider 1 terminated
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 1
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_3: connection for provider 3 terminated
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 3
2017-06-14 23:11:56 PDT INFO copy_set 1 - omit=f - bool=0
2017-06-14 23:11:56 PDT INFO omit is FALSE
2017-06-14 23:11:56 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: connected to provider DB
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive"."entity"
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive"."entity_result"
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive"."result"
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive"."result_logbook"
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: all tables for set 1 found on subscriber
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive"."entity_entity_id_seq"
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive"."entity_result_id_seq"
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive"."result_id_seq"
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive"."result_logbook_record_id_seq"
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive"."entity"
2017-06-14 23:11:57 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive"."entity"
NOTICE: truncate of "il_longlive"."entity" succeeded
2017-06-14 23:12:02 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:12:02 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:12:02 PDT INFO remoteWorkerThread_3: SYNC 5000000071 done in 0.029 seconds
2017-06-14 23:12:13 PDT INFO remoteWorkerThread_3: SYNC 5000000072 done in 0.003 seconds
2017-06-14 23:12:28 PDT INFO remoteWorkerThread_3: SYNC 5000000073 done in 0.016 seconds
2017-06-14 23:12:33 PDT CONFIG remoteWorkerThread_1: 125045870 bytes copied for table "il_longlive"."entity"
2017-06-14 23:12:35 PDT INFO remoteWorkerThread_3: SYNC 5000000074 done in 0.010 seconds
2017-06-14 23:12:43 PDT CONFIG remoteWorkerThread_1: 46.468 seconds to copy table "il_longlive"."entity"
2017-06-14 23:12:43 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive"."entity_result"
2017-06-14 23:12:43 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive"."entity_result"
NOTICE: truncate of "il_longlive"."entity_result" succeeded
2017-06-14 23:12:46 PDT INFO remoteWorkerThread_3: SYNC 5000000075 done in 0.003 seconds
2017-06-14 23:12:50 PDT INFO remoteWorkerThread_3: SYNC 5000000076 done in 0.003 seconds
2017-06-14 23:13:12 PDT CONFIG remoteWorkerThread_1: 66996342 bytes copied for table "il_longlive"."entity_result"
2017-06-14 23:13:14 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:13:14 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:13:14 PDT INFO remoteWorkerThread_3: SYNC 5000000078 done in 0.025 seconds
2017-06-14 23:13:23 PDT INFO remoteWorkerThread_3: SYNC 5000000079 done in 0.013 seconds
2017-06-14 23:13:32 PDT INFO remoteWorkerThread_3: SYNC 5000000080 done in 0.005 seconds
2017-06-14 23:13:41 PDT INFO remoteWorkerThread_3: SYNC 5000000081 done in 0.005 seconds
2017-06-14 23:13:55 PDT INFO remoteWorkerThread_3: SYNC 5000000082 done in 0.005 seconds
2017-06-14 23:13:57 PDT CONFIG remoteWorkerThread_1: 74.255 seconds to copy table "il_longlive"."entity_result"
2017-06-14 23:13:57 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive"."result"
2017-06-14 23:13:57 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive"."result"
NOTICE: truncate of "il_longlive"."result" succeeded
2017-06-14 23:14:07 PDT INFO remoteWorkerThread_3: SYNC 5000000083 done in 0.005 seconds
2017-06-14 23:14:20 PDT INFO remoteWorkerThread_3: SYNC 5000000084 done in 0.020 seconds
2017-06-14 23:14:23 PDT INFO remoteWorkerThread_3: SYNC 5000000085 done in 0.005 seconds
2017-06-14 23:14:29 PDT CONFIG remoteWorkerThread_1: 105961503 bytes copied for table "il_longlive"."result"
2017-06-14 23:14:33 PDT INFO remoteWorkerThread_3: SYNC 5000000086 done in 0.005 seconds
2017-06-14 23:14:41 PDT INFO remoteWorkerThread_3: SYNC 5000000087 done in 0.005 seconds
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: 52.781 seconds to copy table "il_longlive"."result"
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive"."result_logbook"
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive"."result_logbook"
NOTICE: truncate of "il_longlive"."result_logbook" succeeded
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: 79 bytes copied for table "il_longlive"."result_logbook"
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: 0.322 seconds to copy table "il_longlive"."result_logbook"
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 103 ("il_longlive"."result_id_seq") to 2611527
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 102 ("il_longlive"."entity_result_id_seq") to 2611527
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 104 ("il_longlive"."result_logbook_record_id_seq") to 1
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 101 ("il_longlive"."entity_entity_id_seq") to 2528612
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: 0.002 seconds to copy sequences
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: copy_set SYNC found, use event seqno 5000000130.
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: 0.005 seconds to build initial setsync status
2017-06-14 23:14:50 PDT INFO copy_set 1 done in 173.913 seconds
2017-06-14 23:14:50 PDT CONFIG enableSubscription: sub_set=1
2017-06-14 23:14:50 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 23:14:50 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1
2017-06-14 23:14:50 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 1
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_3: connection for provider 3 terminated
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 3
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_3: connection for provider 1 terminated
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 1
2017-06-14 23:14:50 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: SYNC 5000000131 done in 0.053 seconds
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:14:50 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 1
2017-06-14 23:14:50 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: SYNC 5000000135 done in 0.016 seconds
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: SYNC 5000000143 done in 0.005 seconds
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: SYNC 5000000147 done in 0.005 seconds
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1
2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 1
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: connection for provider 3 terminated
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 3
2017-06-14 23:14:52 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=3
2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=3 li_receiver=2 li_provider=1
2017-06-14 23:14:52 PDT CONFIG storeListen: li_origin=1 li_receiver=2 li_provider=1
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 1
2017-06-14 23:14:52 PDT INFO remoteWorkerThread_3: SYNC 5000000088 done in 0.013 seconds
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_3: connection for provider 1 terminated
2017-06-14 23:14:52 PDT CONFIG remoteWorkerThread_3: disconnecting from data provider 1
2017-06-14 23:15:00 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:15:00 PDT INFO remoteWorkerThread_1: SYNC 5000000150 done in 0.006 seconds
2017-06-14 23:15:05 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:15:05 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:15:05 PDT INFO remoteWorkerThread_3: SYNC 5000000089 done in 0.054 seconds
2017-06-14 23:15:11 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:15:11 PDT INFO remoteWorkerThread_1: SYNC 5000000151 done in 0.004 seconds
2017-06-14 23:15:15 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:15:15 PDT INFO remoteWorkerThread_1: SYNC 5000000152 done in 0.004 seconds
2017-06-14 23:15:16 PDT INFO remoteWorkerThread_3: SYNC 5000000090 done in 0.005 seconds
2017-06-14 23:15:26 PDT CONFIG remoteWorkerThread_3: update provider configuration
2017-06-14 23:15:26 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:15:26 PDT INFO remoteWorkerThread_1: SYNC 5000000153 done in 0.007 seconds
2017-06-14 23:15:26 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:15:26 PDT INFO remoteWorkerThread_3: SYNC 5000000091 done in 0.025 seconds
2017-06-14 23:15:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:15:37 PDT INFO remoteWorkerThread_1: SYNC 5000000154 done in 0.005 seconds
2017-06-14 23:15:37 PDT INFO remoteWorkerThread_3: SYNC 5000000092 done in 0.006 seconds
2017-06-14 23:15:47 PDT INFO remoteWorkerThread_3: SYNC 5000000093 done in 0.005 seconds
2017-06-14 23:15:51 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:15:51 PDT INFO remoteWorkerThread_1: SYNC 5000000155 done in 0.004 seconds
2017-06-14 23:15:55 PDT INFO remoteWorkerThread_3: SYNC 5000000094 done in 0.005 seconds
2017-06-14 23:15:59 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:15:59 PDT INFO remoteWorkerThread_1: SYNC 5000000156 done in 0.005 seconds
2017-06-14 23:16:09 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:16:09 PDT INFO remoteWorkerThread_1: SYNC 5000000157 done in 0.004 seconds
2017-06-14 23:16:12 PDT INFO remoteWorkerThread_3: SYNC 5000000095 done in 0.009 seconds
2017-06-14 23:16:13 PDT INFO remoteWorkerThread_3: SYNC 5000000096 done in 0.015 seconds
2017-06-14 23:16:17 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:16:17 PDT INFO remoteWorkerThread_1: SYNC 5000000158 done in 0.004 seconds
2017-06-14 23:16:28 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:16:28 PDT INFO remoteWorkerThread_1: SYNC 5000000159 done in 0.004 seconds
2017-06-14 23:16:31 PDT INFO remoteWorkerThread_3: SYNC 5000000097 done in 0.005 seconds
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: SYNC 5000000160 done in 0.004 seconds
2017-06-14 23:16:39 PDT INFO remoteWorkerThread_3: SYNC 5000000098 done in 0.005 seconds
2017-06-14 23:16:46 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:16:46 PDT INFO remoteWorkerThread_1: SYNC 5000000161 done in 0.004 seconds
2017-06-14 23:16:50 PDT INFO remoteWorkerThread_3: SYNC 5000000099 done in 0.005 seconds
2017-06-14 23:16:57 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:16:57 PDT INFO remoteWorkerThread_1: SYNC 5000000162 done in 0.004 seconds
2017-06-14 23:17:01 PDT INFO remoteWorkerThread_3: SYNC 5000000100 done in 0.008 seconds
2017-06-14 23:17:09 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:17:09 PDT INFO remoteWorkerThread_3: SYNC 5000000101 done in 0.006 seconds
2017-06-14 23:17:09 PDT INFO remoteWorkerThread_1: SYNC 5000000163 done in 0.005 seconds
2017-06-14 23:17:22 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:17:22 PDT INFO remoteWorkerThread_1: SYNC 5000000164 done in 0.005 seconds
2017-06-14 23:17:22 PDT INFO remoteWorkerThread_3: SYNC 5000000102 done in 0.005 seconds
2017-06-14 23:17:25 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:17:25 PDT INFO remoteWorkerThread_1: SYNC 5000000165 done in 0.004 seconds
2017-06-14 23:17:30 PDT INFO remoteWorkerThread_3: SYNC 5000000103 done in 0.005 seconds
2017-06-14 23:17:36 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:17:36 PDT INFO remoteWorkerThread_1: SYNC 5000000166 done in 0.004 seconds
2017-06-14 23:17:40 PDT INFO remoteWorkerThread_3: SYNC 5000000104 done in 0.013 seconds
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15559
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15656
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15675
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15683
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15825
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15838
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15853
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15939
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15940
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=15941
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: log switch to sl_log_2 complete - truncate sl_log_1
CONTEXT: PL/pgSQL function _region_il.cleanupevent(interval) line 95 at assignment
2017-06-14 23:17:42 PDT INFO cleanupThread: 0.047 seconds for cleanupEvent()
2017-06-14 23:17:48 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:17:48 PDT INFO remoteWorkerThread_3: SYNC 5000000105 done in 0.005 seconds
2017-06-14 23:17:48 PDT INFO remoteWorkerThread_1: SYNC 5000000167 done in 0.005 seconds
2017-06-14 23:18:01 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:18:01 PDT INFO remoteWorkerThread_3: SYNC 5000000106 done in 0.005 seconds
2017-06-14 23:18:01 PDT INFO remoteWorkerThread_1: SYNC 5000000168 done in 0.005 seconds
2017-06-14 23:18:14 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:18:14 PDT INFO remoteWorkerThread_3: SYNC 5000000107 done in 0.006 seconds
2017-06-14 23:18:14 PDT INFO remoteWorkerThread_1: SYNC 5000000169 done in 0.006 seconds
2017-06-14 23:18:17 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:18:17 PDT INFO remoteWorkerThread_1: SYNC 5000000170 done in 0.005 seconds
2017-06-14 23:18:20 PDT INFO remoteWorkerThread_3: SYNC 5000000108 done in 0.005 seconds
2017-06-14 23:18:28 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:18:28 PDT INFO remoteWorkerThread_3: SYNC 5000000109 done in 0.005 seconds
2017-06-14 23:18:28 PDT INFO remoteWorkerThread_1: SYNC 5000000171 done in 0.005 seconds
2017-06-14 23:18:41 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:18:41 PDT INFO remoteWorkerThread_1: SYNC 5000000172 done in 0.005 seconds
2017-06-14 23:18:41 PDT INFO remoteWorkerThread_3: SYNC 5000000110 done in 0.005 seconds
2017-06-14 23:18:54 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:18:54 PDT INFO remoteWorkerThread_3: SYNC 5000000111 done in 0.005 seconds
2017-06-14 23:18:54 PDT INFO remoteWorkerThread_1: SYNC 5000000173 done in 0.005 seconds
2017-06-14 23:18:57 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:18:57 PDT INFO remoteWorkerThread_1: SYNC 5000000174 done in 0.013 seconds
2017-06-14 23:19:00 PDT INFO remoteWorkerThread_3: SYNC 5000000112 done in 0.009 seconds
2017-06-14 23:19:08 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:19:08 PDT INFO remoteWorkerThread_3: SYNC 5000000113 done in 0.005 seconds
2017-06-14 23:19:08 PDT INFO remoteWorkerThread_1: SYNC 5000000175 done in 0.005 seconds
2017-06-14 23:19:21 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:19:21 PDT INFO remoteWorkerThread_3: SYNC 5000000114 done in 0.006 seconds
2017-06-14 23:19:21 PDT INFO remoteWorkerThread_1: SYNC 5000000176 done in 0.005 seconds
2017-06-14 23:19:34 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:19:34 PDT INFO remoteWorkerThread_3: SYNC 5000000115 done in 0.006 seconds
2017-06-14 23:19:34 PDT INFO remoteWorkerThread_1: SYNC 5000000177 done in 0.006 seconds
2017-06-14 23:19:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:19:37 PDT INFO remoteWorkerThread_1: SYNC 5000000178 done in 0.005 seconds
2017-06-14 23:19:40 PDT INFO remoteWorkerThread_3: SYNC 5000000116 done in 0.005 seconds
2017-06-14 23:19:48 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:19:48 PDT INFO remoteWorkerThread_3: SYNC 5000000117 done in 0.005 seconds
2017-06-14 23:19:48 PDT INFO remoteWorkerThread_1: SYNC 5000000179 done in 0.005 seconds
2017-06-14 23:20:01 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:20:01 PDT INFO remoteWorkerThread_1: SYNC 5000000180 done in 0.006 seconds
2017-06-14 23:20:01 PDT INFO remoteWorkerThread_3: SYNC 5000000118 done in 0.006 seconds
2017-06-14 23:20:14 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:20:14 PDT INFO remoteWorkerThread_1: SYNC 5000000181 done in 0.005 seconds
2017-06-14 23:20:14 PDT INFO remoteWorkerThread_3: SYNC 5000000119 done in 0.005 seconds
2017-06-14 23:20:17 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:20:17 PDT INFO remoteWorkerThread_1: SYNC 5000000182 done in 0.005 seconds
2017-06-14 23:20:18 PDT INFO remoteWorkerThread_3: SYNC 5000000120 done in 0.005 seconds
2017-06-14 23:20:27 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:20:27 PDT INFO remoteWorkerThread_1: SYNC 5000000183 done in 0.005 seconds
2017-06-14 23:20:32 PDT INFO remoteWorkerThread_3: SYNC 5000000121 done in 0.006 seconds
2017-06-14 23:20:38 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:20:38 PDT INFO remoteWorkerThread_1: SYNC 5000000184 done in 0.004 seconds
2017-06-14 23:20:39 PDT INFO remoteWorkerThread_3: SYNC 5000000122 done in 0.005 seconds
2017-06-14 23:20:53 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:20:53 PDT INFO remoteWorkerThread_3: SYNC 5000000123 done in 0.006 seconds
2017-06-14 23:20:53 PDT INFO remoteWorkerThread_1: SYNC 5000000185 done in 0.006 seconds
2017-06-14 23:21:01 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:21:01 PDT INFO remoteWorkerThread_3: SYNC 5000000124 done in 0.006 seconds
2017-06-14 23:21:01 PDT INFO remoteWorkerThread_1: SYNC 5000000186 done in 0.006 seconds
2017-06-14 23:21:09 PDT INFO remoteWorkerThread_3: SYNC 5000000125 done in 0.005 seconds
2017-06-14 23:21:14 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:21:14 PDT INFO remoteWorkerThread_1: SYNC 5000000187 done in 0.004 seconds
2017-06-14 23:21:15 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:21:15 PDT INFO remoteWorkerThread_1: SYNC 5000000188 done in 0.004 seconds
2017-06-14 23:21:20 PDT INFO remoteWorkerThread_3: SYNC 5000000126 done in 0.005 seconds
2017-06-14 23:21:26 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:21:26 PDT INFO remoteWorkerThread_1: SYNC 5000000189 done in 0.004 seconds
2017-06-14 23:21:31 PDT INFO remoteWorkerThread_3: SYNC 5000000127 done in 0.005 seconds
2017-06-14 23:21:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:21:37 PDT INFO remoteWorkerThread_1: SYNC 5000000190 done in 0.005 seconds
2017-06-14 23:21:42 PDT INFO remoteWorkerThread_3: SYNC 5000000128 done in 0.005 seconds
2017-06-14 23:21:48 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:21:48 PDT INFO remoteWorkerThread_1: SYNC 5000000191 done in 0.008 seconds
2017-06-14 23:21:52 PDT INFO remoteWorkerThread_3: SYNC 5000000129 done in 0.006 seconds
2017-06-14 23:22:00 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:22:00 PDT INFO remoteWorkerThread_1: SYNC 5000000192 done in 0.005 seconds
2017-06-14 23:22:00 PDT INFO remoteWorkerThread_3: SYNC 5000000130 done in 0.006 seconds
2017-06-14 23:22:13 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:22:13 PDT INFO remoteWorkerThread_3: SYNC 5000000131 done in 0.005 seconds
2017-06-14 23:22:13 PDT INFO remoteWorkerThread_1: SYNC 5000000193 done in 0.005 seconds
2017-06-14 23:22:16 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:22:16 PDT INFO remoteWorkerThread_1: SYNC 5000000194 done in 0.004 seconds
2017-06-14 23:22:21 PDT INFO remoteWorkerThread_3: SYNC 5000000132 done in 0.005 seconds
2017-06-14 23:22:27 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:22:27 PDT INFO remoteWorkerThread_1: SYNC 5000000195 done in 0.005 seconds
2017-06-14 23:22:32 PDT INFO remoteWorkerThread_3: SYNC 5000000133 done in 0.005 seconds
2017-06-14 23:22:38 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:22:38 PDT INFO remoteWorkerThread_1: SYNC 5000000196 done in 0.006 seconds
2017-06-14 23:22:42 PDT INFO remoteWorkerThread_3: SYNC 5000000134 done in 0.005 seconds
2017-06-14 23:22:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:22:50 PDT INFO remoteWorkerThread_1: SYNC 5000000197 done in 0.005 seconds
2017-06-14 23:22:54 PDT INFO remoteWorkerThread_3: SYNC 5000000135 done in 0.005 seconds
2017-06-14 23:22:55 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:22:55 PDT INFO remoteWorkerThread_1: SYNC 5000000198 done in 0.004 seconds
2017-06-14 23:22:59 PDT INFO remoteWorkerThread_3: SYNC 5000000136 done in 0.005 seconds
2017-06-14 23:23:10 PDT INFO remoteWorkerThread_3: SYNC 5000000137 done in 0.005 seconds
2017-06-14 23:23:13 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:23:13 PDT INFO remoteWorkerThread_1: SYNC 5000000199 done in 0.005 seconds
2017-06-14 23:23:21 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:23:21 PDT INFO remoteWorkerThread_1: SYNC 5000000200 done in 0.004 seconds
2017-06-14 23:23:25 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:23:25 PDT INFO remoteWorkerThread_3: SYNC 5000000138 done in 0.005 seconds
2017-06-14 23:23:25 PDT INFO remoteWorkerThread_1: SYNC 5000000201 done in 0.005 seconds
2017-06-14 23:23:34 PDT INFO remoteWorkerThread_3: SYNC 5000000139 done in 0.005 seconds
2017-06-14 23:23:39 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:23:39 PDT INFO remoteWorkerThread_1: SYNC 5000000202 done in 0.005 seconds
2017-06-14 23:23:43 PDT INFO remoteWorkerThread_3: SYNC 5000000140 done in 0.005 seconds
2017-06-14 23:23:47 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 1
2017-06-14 23:23:47 PDT INFO remoteWorkerThread_1: SYNC 5000000203 done in 0.004 seconds

辅节点2查看Slony-I日志信息

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
$ cat /usr/local/pgsql/data/pg_log/node3/region_il-2017-06-14.log
2017-06-14 23:00:38 PDT CONFIG main: slon version 2.2.4 starting up
2017-06-14 23:00:38 PDT INFO slon: watchdog process started
2017-06-14 23:00:38 PDT CONFIG slon: watchdog ready - pid = 19212
2017-06-14 23:00:38 PDT CONFIG slon: worker process created - pid = 19213
2017-06-14 23:00:38 PDT CONFIG main: Integer option vac_frequency = 3
2017-06-14 23:00:38 PDT CONFIG main: Integer option log_level = 0
2017-06-14 23:00:38 PDT CONFIG main: Integer option sync_interval = 1000
2017-06-14 23:00:38 PDT CONFIG main: Integer option sync_interval_timeout = 10000
2017-06-14 23:00:38 PDT CONFIG main: Integer option sync_group_maxsize = 20
2017-06-14 23:00:38 PDT CONFIG main: Integer option syslog = 0
2017-06-14 23:00:38 PDT CONFIG main: Integer option quit_sync_provider = 0
2017-06-14 23:00:38 PDT CONFIG main: Integer option remote_listen_timeout = 300
2017-06-14 23:00:38 PDT CONFIG main: Integer option monitor_interval = 500
2017-06-14 23:00:38 PDT CONFIG main: Integer option explain_interval = 0
2017-06-14 23:00:38 PDT CONFIG main: Integer option tcp_keepalive_idle = 0
2017-06-14 23:00:38 PDT CONFIG main: Integer option tcp_keepalive_interval = 0
2017-06-14 23:00:38 PDT CONFIG main: Integer option tcp_keepalive_count = 0
2017-06-14 23:00:38 PDT CONFIG main: Integer option apply_cache_size = 100
2017-06-14 23:00:38 PDT CONFIG main: Boolean option log_pid = 0
2017-06-14 23:00:38 PDT CONFIG main: Boolean option log_timestamp = 1
2017-06-14 23:00:38 PDT CONFIG main: Boolean option tcp_keepalive = 1
2017-06-14 23:00:38 PDT CONFIG main: Boolean option monitor_threads = 1
2017-06-14 23:00:38 PDT CONFIG main: Real option real_placeholder = 0.000000
2017-06-14 23:00:38 PDT CONFIG main: String option cluster_name = region_il
2017-06-14 23:00:38 PDT CONFIG main: String option conn_info = host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888
2017-06-14 23:00:38 PDT CONFIG main: String option pid_file = /usr/local/pgsql/data/region_il_node3.pid
2017-06-14 23:00:38 PDT CONFIG main: String option log_timestamp_format = %Y-%m-%d %H:%M:%S %Z
2017-06-14 23:00:38 PDT CONFIG main: String option archive_dir = [NULL]
2017-06-14 23:00:38 PDT CONFIG main: String option sql_on_connection = [NULL]
2017-06-14 23:00:38 PDT CONFIG main: String option lag_interval = [NULL]
2017-06-14 23:00:38 PDT CONFIG main: String option command_on_logarchive = [NULL]
2017-06-14 23:00:38 PDT CONFIG main: String option syslog_facility = LOCAL0
2017-06-14 23:00:38 PDT CONFIG main: String option syslog_ident = slon
2017-06-14 23:00:38 PDT CONFIG main: String option cleanup_interval = 10 minutes
2017-06-14 23:00:38 PDT CONFIG main: local node id = 3
2017-06-14 23:00:38 PDT INFO main: main process started
2017-06-14 23:00:38 PDT CONFIG main: launching sched_start_mainloop
2017-06-14 23:00:38 PDT CONFIG main: loading current cluster configuration
2017-06-14 23:00:38 PDT CONFIG storeNode: no_id=1 no_comment='Node 1 - region_il@10.189.102.118'
2017-06-14 23:00:38 PDT CONFIG storeNode: no_id=2 no_comment='Node 2 - region_il@10.189.100.195'
2017-06-14 23:00:38 PDT CONFIG storePath: pa_server=1 pa_client=3 pa_conninfo="host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10
2017-06-14 23:00:38 PDT CONFIG storePath: pa_server=2 pa_client=3 pa_conninfo="host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=10
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG main: last local event sequence = 5000000002
2017-06-14 23:00:38 PDT CONFIG main: configuration complete - starting threads
2017-06-14 23:00:38 PDT INFO localListenThread: thread starts
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG enableNode: no_id=1
2017-06-14 23:00:38 PDT CONFIG enableNode: no_id=2
2017-06-14 23:00:38 PDT INFO main: running scheduler mainloop
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: thread starts
2017-06-14 23:00:38 PDT INFO remoteListenThread_2: thread starts
2017-06-14 23:00:38 PDT CONFIG cleanupThread: thread starts
2017-06-14 23:00:38 PDT INFO syncThread: thread starts
2017-06-14 23:00:38 PDT INFO monitorThread: thread starts
2017-06-14 23:00:38 PDT INFO remoteListenThread_1: thread starts
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: thread starts
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:38 PDT CONFIG cleanupThread: bias = 60
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: SYNC 5000000003 done in 0.019 seconds
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1
2017-06-14 23:00:38 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: SYNC 5000000005 done in 0.011 seconds
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000008 done in 0.011 seconds
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2
2017-06-14 23:00:38 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:00:38 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000010 done in 0.002 seconds
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000014 done in 0.002 seconds
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: SYNC 5000000009 done in 0.011 seconds
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000022 done in 0.002 seconds
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: SYNC 5000000017 done in 0.002 seconds
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000038 done in 0.002 seconds
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_2: SYNC 5000000020 done in 0.002 seconds
2017-06-14 23:00:38 PDT INFO remoteWorkerThread_1: SYNC 5000000043 done in 0.002 seconds
2017-06-14 23:00:46 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:00:46 PDT INFO remoteWorkerThread_2: SYNC 5000000021 done in 0.003 seconds
2017-06-14 23:00:46 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:00:46 PDT INFO remoteWorkerThread_1: SYNC 5000000044 done in 0.017 seconds
2017-06-14 23:00:57 PDT INFO remoteWorkerThread_2: SYNC 5000000022 done in 0.002 seconds
2017-06-14 23:01:02 PDT INFO remoteWorkerThread_1: SYNC 5000000045 done in 0.005 seconds
2017-06-14 23:01:03 PDT INFO remoteWorkerThread_1: SYNC 5000000046 done in 0.004 seconds
2017-06-14 23:01:08 PDT INFO remoteWorkerThread_2: SYNC 5000000023 done in 0.003 seconds
2017-06-14 23:01:14 PDT INFO remoteWorkerThread_1: SYNC 5000000047 done in 0.004 seconds
2017-06-14 23:01:19 PDT INFO remoteWorkerThread_2: SYNC 5000000024 done in 0.003 seconds
2017-06-14 23:01:23 PDT INFO remoteWorkerThread_2: SYNC 5000000025 done in 0.003 seconds
2017-06-14 23:01:25 PDT INFO remoteWorkerThread_1: SYNC 5000000048 done in 0.004 seconds
2017-06-14 23:01:34 PDT INFO remoteWorkerThread_2: SYNC 5000000026 done in 0.002 seconds
2017-06-14 23:01:38 PDT INFO remoteWorkerThread_1: SYNC 5000000049 done in 0.004 seconds
2017-06-14 23:01:46 PDT INFO remoteWorkerThread_2: SYNC 5000000027 done in 0.003 seconds
2017-06-14 23:01:50 PDT INFO remoteWorkerThread_1: SYNC 5000000050 done in 0.005 seconds
2017-06-14 23:01:54 PDT INFO remoteWorkerThread_2: SYNC 5000000028 done in 0.002 seconds
2017-06-14 23:01:54 PDT INFO remoteWorkerThread_1: SYNC 5000000051 done in 0.005 seconds
2017-06-14 23:02:05 PDT INFO remoteWorkerThread_2: SYNC 5000000029 done in 0.003 seconds
2017-06-14 23:02:09 PDT INFO remoteWorkerThread_1: SYNC 5000000052 done in 0.005 seconds
2017-06-14 23:02:17 PDT INFO remoteWorkerThread_1: SYNC 5000000053 done in 0.005 seconds
2017-06-14 23:02:21 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:02:21 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:02:21 PDT INFO remoteWorkerThread_2: SYNC 5000000030 done in 0.027 seconds
2017-06-14 23:02:25 PDT INFO remoteWorkerThread_1: SYNC 5000000054 done in 0.007 seconds
2017-06-14 23:02:25 PDT INFO remoteWorkerThread_2: SYNC 5000000031 done in 0.008 seconds
2017-06-14 23:02:36 PDT INFO remoteWorkerThread_1: SYNC 5000000055 done in 0.008 seconds
2017-06-14 23:02:39 PDT INFO remoteWorkerThread_2: SYNC 5000000032 done in 0.005 seconds
2017-06-14 23:02:43 PDT INFO remoteWorkerThread_1: SYNC 5000000056 done in 0.005 seconds
2017-06-14 23:02:47 PDT INFO remoteWorkerThread_2: SYNC 5000000033 done in 0.005 seconds
2017-06-14 23:02:54 PDT INFO remoteWorkerThread_1: SYNC 5000000057 done in 0.004 seconds
2017-06-14 23:02:54 PDT INFO remoteWorkerThread_2: SYNC 5000000034 done in 0.005 seconds
2017-06-14 23:03:05 PDT INFO remoteWorkerThread_1: SYNC 5000000058 done in 0.005 seconds
2017-06-14 23:03:08 PDT INFO remoteWorkerThread_2: SYNC 5000000035 done in 0.005 seconds
2017-06-14 23:03:16 PDT CONFIG storeSet: set_id=1 set_origin=1 set_comment='Set 1 (il_longlive_schema) for region_il'
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_2: SYNC 5000000036 done in 0.005 seconds
2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated
2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1
2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: connection for provider 2 terminated
2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 2
2017-06-14 23:03:16 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:03:16 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000060 done in 0.012 seconds
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000062 done in 0.002 seconds
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000064 done in 0.002 seconds
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000066 done in 0.002 seconds
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000068 done in 0.003 seconds
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000070 done in 0.002 seconds
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000072 done in 0.002 seconds
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000074 done in 0.002 seconds
2017-06-14 23:03:16 PDT INFO remoteWorkerThread_1: SYNC 5000000076 done in 0.002 seconds
2017-06-14 23:03:29 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:03:29 PDT INFO remoteWorkerThread_2: SYNC 5000000037 done in 0.005 seconds
2017-06-14 23:03:29 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:03:29 PDT INFO remoteWorkerThread_1: SYNC 5000000077 done in 0.015 seconds
2017-06-14 23:03:42 PDT INFO remoteWorkerThread_2: SYNC 5000000038 done in 0.013 seconds
2017-06-14 23:03:42 PDT INFO remoteWorkerThread_1: SYNC 5000000078 done in 0.015 seconds
2017-06-14 23:03:43 PDT INFO remoteWorkerThread_1: SYNC 5000000079 done in 0.004 seconds
2017-06-14 23:03:43 PDT INFO remoteWorkerThread_2: SYNC 5000000039 done in 0.005 seconds
2017-06-14 23:03:56 PDT INFO remoteWorkerThread_1: SYNC 5000000080 done in 0.004 seconds
2017-06-14 23:03:56 PDT INFO remoteWorkerThread_2: SYNC 5000000040 done in 0.005 seconds
2017-06-14 23:04:04 PDT INFO remoteWorkerThread_2: SYNC 5000000041 done in 0.005 seconds
2017-06-14 23:04:08 PDT INFO remoteWorkerThread_1: SYNC 5000000081 done in 0.005 seconds
2017-06-14 23:04:16 PDT INFO remoteWorkerThread_1: SYNC 5000000082 done in 0.005 seconds
2017-06-14 23:04:16 PDT INFO remoteWorkerThread_2: SYNC 5000000042 done in 0.006 seconds
2017-06-14 23:04:29 PDT INFO remoteWorkerThread_2: SYNC 5000000043 done in 0.004 seconds
2017-06-14 23:04:29 PDT INFO remoteWorkerThread_1: SYNC 5000000083 done in 0.005 seconds
2017-06-14 23:04:37 PDT INFO remoteWorkerThread_2: SYNC 5000000044 done in 0.004 seconds
2017-06-14 23:04:37 PDT INFO remoteWorkerThread_1: SYNC 5000000084 done in 0.005 seconds
2017-06-14 23:04:48 PDT INFO remoteWorkerThread_2: SYNC 5000000045 done in 0.004 seconds
2017-06-14 23:04:48 PDT INFO remoteWorkerThread_1: SYNC 5000000085 done in 0.005 seconds
2017-06-14 23:05:02 PDT INFO remoteWorkerThread_1: SYNC 5000000086 done in 0.004 seconds
2017-06-14 23:05:02 PDT INFO remoteWorkerThread_2: SYNC 5000000046 done in 0.005 seconds
2017-06-14 23:05:03 PDT INFO remoteWorkerThread_1: SYNC 5000000087 done in 0.005 seconds
2017-06-14 23:05:05 PDT INFO remoteWorkerThread_2: SYNC 5000000047 done in 0.005 seconds
2017-06-14 23:05:14 PDT INFO remoteWorkerThread_1: SYNC 5000000088 done in 0.005 seconds
2017-06-14 23:05:19 PDT INFO remoteWorkerThread_2: SYNC 5000000048 done in 0.004 seconds
2017-06-14 23:05:23 PDT INFO remoteWorkerThread_2: SYNC 5000000049 done in 0.005 seconds
2017-06-14 23:05:25 PDT INFO remoteWorkerThread_1: SYNC 5000000089 done in 0.004 seconds
2017-06-14 23:05:34 PDT INFO remoteWorkerThread_2: SYNC 5000000050 done in 0.006 seconds
2017-06-14 23:05:38 PDT INFO remoteWorkerThread_1: SYNC 5000000090 done in 0.005 seconds
2017-06-14 23:05:46 PDT INFO remoteWorkerThread_1: SYNC 5000000091 done in 0.004 seconds
2017-06-14 23:05:46 PDT INFO remoteWorkerThread_2: SYNC 5000000051 done in 0.005 seconds
2017-06-14 23:05:59 PDT INFO remoteWorkerThread_1: SYNC 5000000092 done in 0.004 seconds
2017-06-14 23:05:59 PDT INFO remoteWorkerThread_2: SYNC 5000000052 done in 0.005 seconds
2017-06-14 23:06:12 PDT INFO remoteWorkerThread_2: SYNC 5000000053 done in 0.004 seconds
2017-06-14 23:06:12 PDT INFO remoteWorkerThread_1: SYNC 5000000093 done in 0.004 seconds
2017-06-14 23:06:15 PDT INFO remoteWorkerThread_2: SYNC 5000000054 done in 0.004 seconds
2017-06-14 23:06:15 PDT INFO remoteWorkerThread_1: SYNC 5000000094 done in 0.005 seconds
2017-06-14 23:06:28 PDT INFO remoteWorkerThread_2: SYNC 5000000055 done in 0.005 seconds
2017-06-14 23:06:28 PDT INFO remoteWorkerThread_1: SYNC 5000000095 done in 0.005 seconds
2017-06-14 23:06:36 PDT INFO remoteWorkerThread_1: SYNC 5000000096 done in 0.004 seconds
2017-06-14 23:06:36 PDT INFO remoteWorkerThread_2: SYNC 5000000056 done in 0.005 seconds
2017-06-14 23:06:47 PDT INFO remoteWorkerThread_1: SYNC 5000000097 done in 0.005 seconds
2017-06-14 23:06:48 PDT INFO remoteWorkerThread_2: SYNC 5000000057 done in 0.004 seconds
2017-06-14 23:06:58 PDT INFO remoteWorkerThread_1: SYNC 5000000098 done in 0.004 seconds
2017-06-14 23:06:58 PDT INFO remoteWorkerThread_2: SYNC 5000000058 done in 0.005 seconds
2017-06-14 23:07:12 PDT INFO remoteWorkerThread_1: SYNC 5000000099 done in 0.004 seconds
2017-06-14 23:07:12 PDT INFO remoteWorkerThread_2: SYNC 5000000059 done in 0.005 seconds
2017-06-14 23:07:15 PDT INFO remoteWorkerThread_2: SYNC 5000000060 done in 0.004 seconds
2017-06-14 23:07:15 PDT INFO remoteWorkerThread_1: SYNC 5000000100 done in 0.005 seconds
2017-06-14 23:07:28 PDT INFO remoteWorkerThread_1: SYNC 5000000101 done in 0.060 seconds
2017-06-14 23:07:28 PDT INFO remoteWorkerThread_2: SYNC 5000000061 done in 0.062 seconds
2017-06-14 23:07:36 PDT INFO remoteWorkerThread_2: SYNC 5000000062 done in 0.004 seconds
2017-06-14 23:07:36 PDT INFO remoteWorkerThread_1: SYNC 5000000102 done in 0.005 seconds
2017-06-14 23:07:47 PDT INFO remoteWorkerThread_1: SYNC 5000000103 done in 0.008 seconds
2017-06-14 23:07:47 PDT INFO remoteWorkerThread_2: SYNC 5000000063 done in 0.009 seconds
2017-06-14 23:07:58 PDT INFO remoteWorkerThread_1: SYNC 5000000104 done in 0.005 seconds
2017-06-14 23:07:58 PDT INFO remoteWorkerThread_2: SYNC 5000000064 done in 0.005 seconds
2017-06-14 23:08:09 PDT INFO remoteWorkerThread_1: SYNC 5000000105 done in 0.007 seconds
2017-06-14 23:08:09 PDT INFO remoteWorkerThread_2: SYNC 5000000065 done in 0.009 seconds
2017-06-14 23:08:13 PDT INFO remoteWorkerThread_1: SYNC 5000000106 done in 0.005 seconds
2017-06-14 23:08:19 PDT INFO remoteWorkerThread_2: SYNC 5000000066 done in 0.010 seconds
2017-06-14 23:08:23 PDT INFO remoteWorkerThread_2: SYNC 5000000067 done in 0.004 seconds
2017-06-14 23:08:24 PDT INFO remoteWorkerThread_1: SYNC 5000000107 done in 0.008 seconds
2017-06-14 23:08:37 PDT INFO remoteWorkerThread_2: SYNC 5000000068 done in 0.006 seconds
2017-06-14 23:08:41 PDT INFO remoteWorkerThread_1: SYNC 5000000108 done in 0.005 seconds
2017-06-14 23:08:45 PDT INFO remoteWorkerThread_1: SYNC 5000000109 done in 0.005 seconds
2017-06-14 23:08:49 PDT INFO remoteWorkerThread_2: SYNC 5000000069 done in 0.005 seconds
2017-06-14 23:08:57 PDT INFO remoteWorkerThread_1: SYNC 5000000110 done in 0.006 seconds
2017-06-14 23:08:57 PDT INFO remoteWorkerThread_2: SYNC 5000000070 done in 0.006 seconds
2017-06-14 23:09:08 PDT INFO remoteWorkerThread_2: SYNC 5000000071 done in 0.005 seconds
2017-06-14 23:09:08 PDT INFO remoteWorkerThread_1: SYNC 5000000111 done in 0.005 seconds
2017-06-14 23:09:19 PDT INFO remoteWorkerThread_1: SYNC 5000000112 done in 0.004 seconds
2017-06-14 23:09:19 PDT INFO remoteWorkerThread_2: SYNC 5000000072 done in 0.005 seconds
2017-06-14 23:09:29 PDT INFO remoteWorkerThread_2: SYNC 5000000073 done in 0.004 seconds
2017-06-14 23:09:29 PDT INFO remoteWorkerThread_1: SYNC 5000000113 done in 0.005 seconds
2017-06-14 23:09:33 PDT INFO remoteWorkerThread_2: SYNC 5000000074 done in 0.005 seconds
2017-06-14 23:09:34 PDT INFO remoteWorkerThread_1: SYNC 5000000114 done in 0.005 seconds
2017-06-14 23:09:43 PDT INFO remoteWorkerThread_2: SYNC 5000000075 done in 0.005 seconds
2017-06-14 23:09:47 PDT INFO remoteWorkerThread_1: SYNC 5000000115 done in 0.004 seconds
2017-06-14 23:09:55 PDT INFO remoteWorkerThread_2: SYNC 5000000076 done in 0.005 seconds
2017-06-14 23:10:01 PDT INFO remoteWorkerThread_1: SYNC 5000000116 done in 0.005 seconds
2017-06-14 23:10:05 PDT INFO remoteWorkerThread_1: SYNC 5000000117 done in 0.005 seconds
2017-06-14 23:10:06 PDT INFO remoteWorkerThread_2: SYNC 5000000077 done in 0.005 seconds
2017-06-14 23:10:14 PDT INFO remoteWorkerThread_1: SYNC 5000000118 done in 0.005 seconds
2017-06-14 23:10:18 PDT INFO remoteWorkerThread_2: SYNC 5000000078 done in 0.005 seconds
2017-06-14 23:10:26 PDT INFO remoteWorkerThread_2: SYNC 5000000079 done in 0.005 seconds
2017-06-14 23:10:30 PDT INFO remoteWorkerThread_1: SYNC 5000000119 done in 0.005 seconds
2017-06-14 23:10:34 PDT INFO remoteWorkerThread_2: SYNC 5000000080 done in 0.005 seconds
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19256
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: Logswitch to sl_log_2 initiated
CONTEXT: SQL statement "SELECT "_region_il".logswitch_start()"
PL/pgSQL function _region_il.cleanupevent(interval) line 97 at PERFORM
2017-06-14 23:10:39 PDT INFO cleanupThread: 0.021 seconds for cleanupEvent()
2017-06-14 23:10:39 PDT INFO remoteWorkerThread_1: SYNC 5000000120 done in 0.005 seconds
2017-06-14 23:10:45 PDT INFO remoteWorkerThread_1: SYNC 5000000121 done in 0.005 seconds
2017-06-14 23:10:45 PDT INFO remoteWorkerThread_2: SYNC 5000000081 done in 0.005 seconds
2017-06-14 23:10:56 PDT INFO remoteWorkerThread_2: SYNC 5000000082 done in 0.005 seconds
2017-06-14 23:10:57 PDT INFO remoteWorkerThread_1: SYNC 5000000122 done in 0.004 seconds
2017-06-14 23:11:07 PDT INFO remoteWorkerThread_1: SYNC 5000000123 done in 0.005 seconds
2017-06-14 23:11:08 PDT INFO remoteWorkerThread_2: SYNC 5000000083 done in 0.004 seconds
2017-06-14 23:11:18 PDT INFO remoteWorkerThread_2: SYNC 5000000084 done in 0.005 seconds
2017-06-14 23:11:19 PDT INFO remoteWorkerThread_1: SYNC 5000000124 done in 0.004 seconds
2017-06-14 23:11:23 PDT INFO remoteWorkerThread_2: SYNC 5000000085 done in 0.004 seconds
2017-06-14 23:11:29 PDT INFO remoteWorkerThread_1: SYNC 5000000125 done in 0.005 seconds
2017-06-14 23:11:34 PDT INFO remoteWorkerThread_1: SYNC 5000000126 done in 0.005 seconds
2017-06-14 23:11:34 PDT INFO remoteWorkerThread_2: SYNC 5000000086 done in 0.005 seconds
2017-06-14 23:11:45 PDT INFO remoteWorkerThread_1: SYNC 5000000127 done in 0.004 seconds
2017-06-14 23:11:45 PDT INFO remoteWorkerThread_2: SYNC 5000000087 done in 0.004 seconds
2017-06-14 23:11:56 PDT INFO remoteWorkerThread_2: SYNC 5000000088 done in 0.005 seconds
2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1
2017-06-14 23:11:56 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: connection for provider 2 terminated
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 2
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_2: connection for provider 1 terminated
2017-06-14 23:11:56 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 1
2017-06-14 23:12:00 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:12:00 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:12:00 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1
2017-06-14 23:12:00 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:12:00 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:12:00 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:12:00 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:12:00 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:12:00 PDT INFO remoteWorkerThread_1: SYNC 5000000130 done in 0.012 seconds
2017-06-14 23:12:08 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:12:08 PDT INFO remoteWorkerThread_1: SYNC 5000000131 done in 0.003 seconds
2017-06-14 23:12:08 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:12:08 PDT INFO remoteWorkerThread_2: SYNC 5000000089 done in 0.014 seconds
2017-06-14 23:12:19 PDT INFO remoteWorkerThread_1: SYNC 5000000132 done in 0.002 seconds
2017-06-14 23:12:30 PDT INFO remoteWorkerThread_1: SYNC 5000000133 done in 0.003 seconds
2017-06-14 23:12:34 PDT INFO remoteWorkerThread_1: SYNC 5000000134 done in 0.003 seconds
2017-06-14 23:12:34 PDT INFO remoteWorkerThread_2: SYNC 5000000090 done in 0.003 seconds
2017-06-14 23:12:38 PDT INFO remoteWorkerThread_2: SYNC 5000000091 done in 0.003 seconds
2017-06-14 23:12:49 PDT INFO remoteWorkerThread_1: SYNC 5000000135 done in 0.002 seconds
2017-06-14 23:12:53 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:12:53 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:12:56 PDT INFO remoteWorkerThread_1: SYNC 5000000136 done in 0.002 seconds
2017-06-14 23:13:02 PDT INFO remoteWorkerThread_2: SYNC 5000000092 done in 8.409 seconds
2017-06-14 23:13:07 PDT INFO remoteWorkerThread_1: SYNC 5000000137 done in 0.003 seconds
2017-06-14 23:13:07 PDT INFO remoteWorkerThread_2: SYNC 5000000093 done in 0.005 seconds
2017-06-14 23:13:18 PDT INFO remoteWorkerThread_1: SYNC 5000000138 done in 0.003 seconds
2017-06-14 23:13:18 PDT INFO remoteWorkerThread_2: SYNC 5000000094 done in 0.005 seconds
2017-06-14 23:13:29 PDT INFO remoteWorkerThread_1: SYNC 5000000139 done in 0.003 seconds
2017-06-14 23:13:29 PDT INFO remoteWorkerThread_2: SYNC 5000000095 done in 0.005 seconds
2017-06-14 23:13:40 PDT INFO remoteWorkerThread_1: SYNC 5000000140 done in 0.003 seconds
2017-06-14 23:13:40 PDT INFO remoteWorkerThread_2: SYNC 5000000096 done in 0.006 seconds
2017-06-14 23:13:44 PDT INFO remoteWorkerThread_1: SYNC 5000000141 done in 0.003 seconds
2017-06-14 23:13:48 PDT INFO remoteWorkerThread_2: SYNC 5000000097 done in 0.006 seconds
2017-06-14 23:13:58 PDT INFO remoteWorkerThread_2: SYNC 5000000098 done in 0.005 seconds
2017-06-14 23:14:02 PDT INFO remoteWorkerThread_1: SYNC 5000000142 done in 0.003 seconds
2017-06-14 23:14:06 PDT INFO remoteWorkerThread_1: SYNC 5000000143 done in 0.002 seconds
2017-06-14 23:14:16 PDT INFO remoteWorkerThread_2: SYNC 5000000099 done in 0.005 seconds
2017-06-14 23:14:20 PDT INFO remoteWorkerThread_2: SYNC 5000000100 done in 0.007 seconds
2017-06-14 23:14:20 PDT INFO remoteWorkerThread_1: SYNC 5000000144 done in 0.003 seconds
2017-06-14 23:14:28 PDT INFO remoteWorkerThread_1: SYNC 5000000145 done in 0.003 seconds
2017-06-14 23:14:34 PDT INFO remoteWorkerThread_2: SYNC 5000000101 done in 0.015 seconds
2017-06-14 23:14:39 PDT INFO remoteWorkerThread_1: SYNC 5000000146 done in 0.003 seconds
2017-06-14 23:14:50 PDT INFO remoteWorkerThread_1: SYNC 5000000147 done in 0.003 seconds
2017-06-14 23:14:51 PDT CONFIG storeSubscribe: sub_set=1 sub_provider=2 sub_forward='t'
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: connection for provider 1 terminated
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 1
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2
2017-06-14 23:14:51 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2017-06-14 23:14:51 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:14:51 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1
2017-06-14 23:14:51 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_1: connection for provider 1 terminated
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_1: disconnecting from data provider 1
2017-06-14 23:14:51 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:14:51 PDT WARN remoteWorkerThread_1: copy set: data provider 2 only on sync 5000000128 - sleep 5 seconds
2017-06-14 23:14:55 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:14:55 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:14:55 PDT INFO remoteWorkerThread_2: SYNC 5000000102 done in 0.012 seconds
2017-06-14 23:14:56 PDT INFO copy_set 1 - omit=f - bool=0
2017-06-14 23:14:56 PDT INFO omit is FALSE
2017-06-14 23:14:56 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: connected to provider DB
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive"."entity"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive"."entity_result"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive"."result"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: prepare to copy table "il_longlive"."result_logbook"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: all tables for set 1 found on subscriber
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive"."entity_entity_id_seq"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive"."entity_result_id_seq"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive"."result_id_seq"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: copy sequence "il_longlive"."result_logbook_record_id_seq"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive"."entity"
2017-06-14 23:14:56 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive"."entity"
NOTICE: truncate of "il_longlive"."entity" succeeded
2017-06-14 23:15:03 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:15:03 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:15:03 PDT INFO remoteWorkerThread_2: SYNC 5000000103 done in 0.033 seconds
2017-06-14 23:15:17 PDT CONFIG remoteWorkerThread_1: 125045870 bytes copied for table "il_longlive"."entity"
2017-06-14 23:15:19 PDT INFO remoteWorkerThread_2: SYNC 5000000104 done in 0.006 seconds
2017-06-14 23:15:22 PDT INFO remoteWorkerThread_2: SYNC 5000000105 done in 0.005 seconds
2017-06-14 23:15:25 PDT CONFIG remoteWorkerThread_1: 28.669 seconds to copy table "il_longlive"."entity"
2017-06-14 23:15:25 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive"."entity_result"
2017-06-14 23:15:25 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive"."entity_result"
NOTICE: truncate of "il_longlive"."entity_result" succeeded
2017-06-14 23:15:34 PDT INFO remoteWorkerThread_2: SYNC 5000000106 done in 0.004 seconds
2017-06-14 23:15:37 PDT CONFIG remoteWorkerThread_1: 66996342 bytes copied for table "il_longlive"."entity_result"
2017-06-14 23:15:45 PDT INFO remoteWorkerThread_2: SYNC 5000000107 done in 0.012 seconds
2017-06-14 23:15:56 PDT INFO remoteWorkerThread_2: SYNC 5000000108 done in 0.004 seconds
2017-06-14 23:16:07 PDT INFO remoteWorkerThread_2: SYNC 5000000109 done in 0.005 seconds
2017-06-14 23:16:10 PDT CONFIG remoteWorkerThread_1: 45.106 seconds to copy table "il_longlive"."entity_result"
2017-06-14 23:16:10 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive"."result"
2017-06-14 23:16:10 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive"."result"
NOTICE: truncate of "il_longlive"."result" succeeded
2017-06-14 23:16:11 PDT INFO remoteWorkerThread_2: SYNC 5000000110 done in 0.015 seconds
2017-06-14 23:16:27 PDT CONFIG remoteWorkerThread_1: 105961503 bytes copied for table "il_longlive"."result"
2017-06-14 23:16:29 PDT INFO remoteWorkerThread_2: SYNC 5000000111 done in 0.005 seconds
2017-06-14 23:16:32 PDT INFO remoteWorkerThread_2: SYNC 5000000112 done in 0.005 seconds
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: 25.058 seconds to copy table "il_longlive"."result"
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: copy table "il_longlive"."result_logbook"
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: Begin COPY of table "il_longlive"."result_logbook"
NOTICE: truncate of "il_longlive"."result_logbook" succeeded
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: 79 bytes copied for table "il_longlive"."result_logbook"
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: 0.079 seconds to copy table "il_longlive"."result_logbook"
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 101 ("il_longlive"."entity_entity_id_seq") to 2528612
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 103 ("il_longlive"."result_id_seq") to 2611527
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 102 ("il_longlive"."entity_result_id_seq") to 2611527
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: set last_value of sequence 104 ("il_longlive"."result_logbook_record_id_seq") to 1
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: 0.002 seconds to copy sequences
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: copy_set SYNC found, use event seqno 5000000147.
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: 0.015 seconds to build initial setsync status
2017-06-14 23:16:35 PDT INFO copy_set 1 done in 98.961 seconds
2017-06-14 23:16:35 PDT CONFIG enableSubscription: sub_set=1
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: connection for provider 2 terminated
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 2
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: connection for provider 1 terminated
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: disconnecting from data provider 1
2017-06-14 23:16:35 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=2
2017-06-14 23:16:35 PDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1
2017-06-14 23:16:35 PDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=2
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 2
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: update provider configuration
2017-06-14 23:16:35 PDT CONFIG remoteWorkerThread_1: added active set 1 to provider 2
2017-06-14 23:16:35 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:16:35 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: SYNC 5000000151 done in 0.042 seconds
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: SYNC 5000000155 done in 0.006 seconds
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:16:35 PDT INFO remoteWorkerThread_1: SYNC 5000000159 done in 0.005 seconds
2017-06-14 23:16:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:16:37 PDT INFO remoteWorkerThread_1: SYNC 5000000160 done in 0.008 seconds
2017-06-14 23:16:41 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:16:41 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:16:41 PDT INFO remoteWorkerThread_2: SYNC 5000000113 done in 0.013 seconds
2017-06-14 23:16:55 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:16:55 PDT INFO remoteWorkerThread_2: SYNC 5000000114 done in 0.005 seconds
2017-06-14 23:16:55 PDT INFO remoteWorkerThread_1: SYNC 5000000161 done in 0.008 seconds
2017-06-14 23:16:59 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:16:59 PDT INFO remoteWorkerThread_1: SYNC 5000000162 done in 0.007 seconds
2017-06-14 23:17:03 PDT INFO remoteWorkerThread_2: SYNC 5000000115 done in 0.003 seconds
2017-06-14 23:17:14 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:17:14 PDT INFO remoteWorkerThread_2: SYNC 5000000116 done in 0.004 seconds
2017-06-14 23:17:14 PDT INFO remoteWorkerThread_1: SYNC 5000000163 done in 0.008 seconds
2017-06-14 23:17:28 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:17:28 PDT INFO remoteWorkerThread_2: SYNC 5000000117 done in 0.004 seconds
2017-06-14 23:17:28 PDT INFO remoteWorkerThread_1: SYNC 5000000165 done in 0.008 seconds
2017-06-14 23:17:32 PDT INFO remoteWorkerThread_2: SYNC 5000000118 done in 0.003 seconds
2017-06-14 23:17:36 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:17:36 PDT INFO remoteWorkerThread_1: SYNC 5000000166 done in 0.007 seconds
2017-06-14 23:17:47 PDT INFO remoteWorkerThread_2: SYNC 5000000119 done in 0.002 seconds
2017-06-14 23:17:48 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:17:48 PDT INFO remoteWorkerThread_1: SYNC 5000000167 done in 0.007 seconds
2017-06-14 23:17:52 PDT INFO remoteWorkerThread_2: SYNC 5000000120 done in 0.003 seconds
2017-06-14 23:18:03 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:18:03 PDT INFO remoteWorkerThread_2: SYNC 5000000121 done in 0.004 seconds
2017-06-14 23:18:03 PDT INFO remoteWorkerThread_1: SYNC 5000000168 done in 0.008 seconds
2017-06-14 23:18:14 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:18:14 PDT INFO remoteWorkerThread_2: SYNC 5000000122 done in 0.004 seconds
2017-06-14 23:18:14 PDT INFO remoteWorkerThread_1: SYNC 5000000169 done in 0.007 seconds
2017-06-14 23:18:18 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:18:18 PDT INFO remoteWorkerThread_1: SYNC 5000000170 done in 0.022 seconds
2017-06-14 23:18:26 PDT INFO remoteWorkerThread_2: SYNC 5000000123 done in 0.003 seconds
2017-06-14 23:18:30 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:18:30 PDT INFO remoteWorkerThread_1: SYNC 5000000171 done in 0.007 seconds
2017-06-14 23:18:31 PDT INFO remoteWorkerThread_2: SYNC 5000000124 done in 0.002 seconds
2017-06-14 23:18:44 PDT CONFIG remoteWorkerThread_2: update provider configuration
2017-06-14 23:18:44 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 90505
2017-06-14 23:18:44 PDT INFO remoteWorkerThread_2: SYNC 5000000125 done in 0.015 seconds
2017-06-14 23:18:49 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:18:49 PDT INFO remoteWorkerThread_1: SYNC 5000000172 done in 0.007 seconds
2017-06-14 23:18:53 PDT INFO remoteWorkerThread_2: SYNC 5000000126 done in 0.004 seconds
2017-06-14 23:18:54 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:18:54 PDT INFO remoteWorkerThread_1: SYNC 5000000173 done in 0.006 seconds
2017-06-14 23:18:58 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:18:58 PDT INFO remoteWorkerThread_1: SYNC 5000000174 done in 0.006 seconds
2017-06-14 23:19:02 PDT INFO remoteWorkerThread_2: SYNC 5000000127 done in 0.012 seconds
2017-06-14 23:19:10 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:19:10 PDT INFO remoteWorkerThread_1: SYNC 5000000175 done in 0.015 seconds
2017-06-14 23:19:11 PDT INFO remoteWorkerThread_2: SYNC 5000000128 done in 0.004 seconds
2017-06-14 23:19:22 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:19:22 PDT INFO remoteWorkerThread_2: SYNC 5000000129 done in 0.005 seconds
2017-06-14 23:19:22 PDT INFO remoteWorkerThread_1: SYNC 5000000176 done in 0.007 seconds
2017-06-14 23:19:36 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:19:36 PDT INFO remoteWorkerThread_2: SYNC 5000000130 done in 0.006 seconds
2017-06-14 23:19:36 PDT INFO remoteWorkerThread_1: SYNC 5000000177 done in 0.007 seconds
2017-06-14 23:19:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:19:37 PDT INFO remoteWorkerThread_1: SYNC 5000000178 done in 0.006 seconds
2017-06-14 23:19:41 PDT INFO remoteWorkerThread_2: SYNC 5000000131 done in 0.005 seconds
2017-06-14 23:19:55 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:19:55 PDT INFO remoteWorkerThread_2: SYNC 5000000132 done in 0.006 seconds
2017-06-14 23:19:55 PDT INFO remoteWorkerThread_1: SYNC 5000000179 done in 0.007 seconds
2017-06-14 23:20:06 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:20:06 PDT INFO remoteWorkerThread_2: SYNC 5000000133 done in 0.005 seconds
2017-06-14 23:20:06 PDT INFO remoteWorkerThread_1: SYNC 5000000180 done in 0.007 seconds
2017-06-14 23:20:16 PDT INFO remoteWorkerThread_2: SYNC 5000000134 done in 0.007 seconds
2017-06-14 23:20:20 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:20:20 PDT INFO remoteWorkerThread_1: SYNC 5000000182 done in 0.007 seconds
2017-06-14 23:20:21 PDT INFO remoteWorkerThread_2: SYNC 5000000135 done in 0.005 seconds
2017-06-14 23:20:35 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:20:35 PDT INFO remoteWorkerThread_2: SYNC 5000000136 done in 0.006 seconds
2017-06-14 23:20:35 PDT INFO remoteWorkerThread_1: SYNC 5000000183 done in 0.008 seconds
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19234
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19239
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19244
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19348
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19566
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19567
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19568
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: cleanup stale sl_nodelock entry for pid=19663
CONTEXT: SQL statement "SELECT "_region_il".cleanupNodelock()"
PL/pgSQL function _region_il.cleanupevent(interval) line 82 at PERFORM
NOTICE: Slony-I: log switch to sl_log_2 complete - truncate sl_log_1
CONTEXT: PL/pgSQL function _region_il.cleanupevent(interval) line 95 at assignment
2017-06-14 23:20:39 PDT INFO cleanupThread: 0.077 seconds for cleanupEvent()
2017-06-14 23:20:39 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:20:39 PDT INFO remoteWorkerThread_1: SYNC 5000000184 done in 0.008 seconds
2017-06-14 23:20:43 PDT INFO remoteWorkerThread_2: SYNC 5000000137 done in 0.005 seconds
2017-06-14 23:20:54 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:20:54 PDT INFO remoteWorkerThread_2: SYNC 5000000138 done in 0.005 seconds
2017-06-14 23:20:54 PDT INFO remoteWorkerThread_1: SYNC 5000000185 done in 0.007 seconds
2017-06-14 23:21:04 PDT INFO remoteWorkerThread_2: SYNC 5000000139 done in 0.009 seconds
2017-06-14 23:21:08 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:21:08 PDT INFO remoteWorkerThread_1: SYNC 5000000186 done in 0.006 seconds
2017-06-14 23:21:11 PDT INFO remoteWorkerThread_2: SYNC 5000000140 done in 0.005 seconds
2017-06-14 23:21:16 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:21:16 PDT INFO remoteWorkerThread_1: SYNC 5000000188 done in 0.007 seconds
2017-06-14 23:21:27 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:21:27 PDT INFO remoteWorkerThread_2: SYNC 5000000141 done in 0.006 seconds
2017-06-14 23:21:27 PDT INFO remoteWorkerThread_1: SYNC 5000000189 done in 0.007 seconds
2017-06-14 23:21:31 PDT INFO remoteWorkerThread_2: SYNC 5000000142 done in 0.005 seconds
2017-06-14 23:21:45 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:21:45 PDT INFO remoteWorkerThread_2: SYNC 5000000143 done in 0.005 seconds
2017-06-14 23:21:45 PDT INFO remoteWorkerThread_1: SYNC 5000000190 done in 0.007 seconds
2017-06-14 23:21:49 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:21:49 PDT INFO remoteWorkerThread_1: SYNC 5000000191 done in 0.008 seconds
2017-06-14 23:21:52 PDT INFO remoteWorkerThread_2: SYNC 5000000144 done in 0.005 seconds
2017-06-14 23:22:05 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:22:05 PDT INFO remoteWorkerThread_2: SYNC 5000000145 done in 0.006 seconds
2017-06-14 23:22:05 PDT INFO remoteWorkerThread_1: SYNC 5000000192 done in 0.008 seconds
2017-06-14 23:22:18 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:22:18 PDT INFO remoteWorkerThread_2: SYNC 5000000146 done in 0.005 seconds
2017-06-14 23:22:18 PDT INFO remoteWorkerThread_1: SYNC 5000000194 done in 0.007 seconds
2017-06-14 23:22:22 PDT INFO remoteWorkerThread_2: SYNC 5000000147 done in 0.005 seconds
2017-06-14 23:22:30 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:22:30 PDT INFO remoteWorkerThread_1: SYNC 5000000195 done in 0.016 seconds
2017-06-14 23:22:31 PDT INFO remoteWorkerThread_2: SYNC 5000000148 done in 0.005 seconds
2017-06-14 23:22:45 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:22:45 PDT INFO remoteWorkerThread_2: SYNC 5000000149 done in 0.006 seconds
2017-06-14 23:22:45 PDT INFO remoteWorkerThread_1: SYNC 5000000196 done in 0.008 seconds
2017-06-14 23:22:56 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:22:56 PDT INFO remoteWorkerThread_2: SYNC 5000000150 done in 0.006 seconds
2017-06-14 23:22:56 PDT INFO remoteWorkerThread_1: SYNC 5000000198 done in 0.008 seconds
2017-06-14 23:23:10 PDT INFO remoteWorkerThread_2: SYNC 5000000151 done in 0.005 seconds
2017-06-14 23:23:13 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:23:13 PDT INFO remoteWorkerThread_2: SYNC 5000000152 done in 0.017 seconds
2017-06-14 23:23:13 PDT INFO remoteWorkerThread_1: SYNC 5000000199 done in 0.019 seconds
2017-06-14 23:23:26 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:23:26 PDT INFO remoteWorkerThread_2: SYNC 5000000153 done in 0.005 seconds
2017-06-14 23:23:26 PDT INFO remoteWorkerThread_1: SYNC 5000000201 done in 0.007 seconds
2017-06-14 23:23:39 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:23:39 PDT INFO remoteWorkerThread_2: SYNC 5000000154 done in 0.006 seconds
2017-06-14 23:23:39 PDT INFO remoteWorkerThread_1: SYNC 5000000202 done in 0.007 seconds
2017-06-14 23:23:43 PDT INFO remoteWorkerThread_2: SYNC 5000000155 done in 0.004 seconds
2017-06-14 23:23:51 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:23:51 PDT INFO remoteWorkerThread_2: SYNC 5000000156 done in 0.005 seconds
2017-06-14 23:23:51 PDT INFO remoteWorkerThread_1: SYNC 5000000203 done in 0.007 seconds
2017-06-14 23:23:59 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:23:59 PDT INFO remoteWorkerThread_1: SYNC 5000000204 done in 0.007 seconds
2017-06-14 23:24:07 PDT INFO remoteWorkerThread_2: SYNC 5000000157 done in 0.005 seconds
2017-06-14 23:24:10 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:24:10 PDT INFO remoteWorkerThread_1: SYNC 5000000205 done in 0.008 seconds
2017-06-14 23:24:13 PDT INFO remoteWorkerThread_2: SYNC 5000000158 done in 0.005 seconds
2017-06-14 23:24:16 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:24:16 PDT INFO remoteWorkerThread_1: SYNC 5000000206 done in 0.006 seconds
2017-06-14 23:24:24 PDT INFO remoteWorkerThread_2: SYNC 5000000159 done in 0.005 seconds
2017-06-14 23:24:37 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:24:37 PDT INFO remoteWorkerThread_2: SYNC 5000000160 done in 0.006 seconds
2017-06-14 23:24:37 PDT INFO remoteWorkerThread_1: SYNC 5000000207 done in 0.008 seconds
2017-06-14 23:24:50 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:24:50 PDT INFO remoteWorkerThread_2: SYNC 5000000161 done in 0.005 seconds
2017-06-14 23:24:50 PDT INFO remoteWorkerThread_1: SYNC 5000000209 done in 0.007 seconds
2017-06-14 23:24:53 PDT INFO remoteWorkerThread_2: SYNC 5000000162 done in 0.005 seconds
2017-06-14 23:25:06 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:25:06 PDT INFO remoteWorkerThread_2: SYNC 5000000163 done in 0.013 seconds
2017-06-14 23:25:06 PDT INFO remoteWorkerThread_1: SYNC 5000000210 done in 0.014 seconds
2017-06-14 23:25:19 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:25:19 PDT INFO remoteWorkerThread_2: SYNC 5000000164 done in 0.005 seconds
2017-06-14 23:25:19 PDT INFO remoteWorkerThread_1: SYNC 5000000211 done in 0.007 seconds
2017-06-14 23:25:22 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:25:22 PDT INFO remoteWorkerThread_2: SYNC 5000000165 done in 0.006 seconds
2017-06-14 23:25:22 PDT INFO remoteWorkerThread_1: SYNC 5000000212 done in 0.008 seconds
2017-06-14 23:25:30 PDT INFO remoteWorkerThread_1: syncing set 1 with 4 table(s) from provider 2
2017-06-14 23:25:30 PDT INFO remoteWorkerThread_1: SYNC 5000000213 done in 0.006 seconds
2017-06-14 23:25:31 PDT INFO remoteWorkerThread_2: SYNC 5000000166 done in 0.004 seconds

13. 查看数据同步状态

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
$ psql -h 10.189.100.226 -U slonyadmin -d region_il -c "\d il_longlive.entity" -W
Password for user slonyadmin:
           Table "il_longlive.entity"
 Column |   Type   |        Modifiers       
--------------+------------------------+------------------------------------------------------------------------
 entity_id | integer    | not null default nextval('il_longlive.entity_entity_id_seq'::regclass)
 country_code | character varying(3) |
 state_code | character varying(50) |
 name   | character varying(128) |
 islive  | boolean    |
Indexes:
 "entity_pkey" PRIMARY KEY, btree (entity_id)
Triggers:        #可以看到Slony-I配置完成后原表多了触发器
 _region_il_denyaccess BEFORE INSERT OR DELETE OR UPDATE ON il_longlive.entity FOR EACH ROW EXECUTE PROCEDURE _region_il.denyaccess('_region_il')
 _region_il_truncatedeny BEFORE TRUNCATE ON il_longlive.entity FOR EACH STATEMENT EXECUTE PROCEDURE _region_il.deny_truncate()
Disabled user triggers:
 _region_il_logtrigger AFTER INSERT OR DELETE OR UPDATE ON il_longlive.entity FOR EACH ROW EXECUTE PROCEDURE _region_il.logtrigger('_region_il', '101', 'k')
 _region_il_truncatetrigger BEFORE TRUNCATE ON il_longlive.entity FOR EACH STATEMENT EXECUTE PROCEDURE _region_il.log_truncate('101')
$ psql -h 10.189.102.118 -U slonyadmin -d region_il -c "select count(*) from il_longlive.entity;" -W
Password for user slonyadmin:
 count
---------
(1 row)
$ psql -h 10.189.100.195 -U slonyadmin -d region_il -c "select count(*) from il_longlive.entity;" -W
Password for user slonyadmin:
 count
---------
   #可以看到数据已经同步到辅节点1了
(1 row)
$ psql -h 10.189.100.226 -U slonyadmin -d region_il -c "select count(*) from il_longlive.entity;" -W
Password for user slonyadmin:
 count
---------
  #可以看到数据已经同步到辅节点2了
(1 row)

14. Slony-I相关表或者视图查看

sl_apply_stats: 每个节点本地的复制表的DML统计状态

sl_components: slon/slonik组件等当前的工作状态

?
1
2
3
4
5
6
7
8
9
10
11
region_il=# select * from sl_components;
  co_actor  | co_pid | co_node | co_connection_pid | co_activity |  co_starttime  | co_event | co_eventtype
----------------------+--------+---------+-------------------+------------------+------------------------+------------+--------------
 local_monitor  | 2455 |  0 |    2470 | thread main loop | 2017-06-14 22:54:42-07 |   | n/a
 local_cleanup  | 2455 |  0 |    2468 | cleanupEvent  | 2017-06-15 00:44:48-07 |   | n/a
 remoteWorkerThread_2 | 2455 |  2 |    2471 | SYNC    | 2017-06-15 00:49:14-07 | 5000000668 | SYNC
 local_listen   | 2455 |  1 |    2459 | thread main loop | 2017-06-15 00:49:20-07 |   | n/a
 remoteWorkerThread_3 | 2455 |  3 |    2467 | SYNC    | 2017-06-15 00:49:17-07 | 5000000653 | SYNC
 remote listener  | 2455 |  3 |    19059 | thread main loop | 2017-06-15 00:49:18-07 |   | n/a
 local_sync   | 2455 |  0 |    2469 | thread main loop | 2017-06-15 00:49:21-07 |   | n/a
(7 rows)

sl_table: 当前集群中的表信息

?
1
2
3
4
5
6
7
8
region_il=# select * from sl_table ;
 tab_id | tab_reloid | tab_relname | tab_nspname | tab_set |  tab_idxname  | tab_altered |     tab_comment    
--------+------------+----------------+-------------+---------+---------------------+-------------+---------------------------------------------------
|  19350 | entity   | il_longlive |  1 | entity_pkey   | f   | Table il_longlive.entity with primary key
|  19355 | entity_result | il_longlive |  1 | entity_result_pkey | f   | Table il_longlive.entity_result with primary key
|  19360 | result   | il_longlive |  1 | result_pkey   | f   | Table il_longlive.result with primary key
|  19365 | result_logbook | il_longlive |  1 | result_logbook_pkey | f   | Table il_longlive.result_logbook with primary key
(4 rows)

sl_set: 当前集群中的集合信息

?
1
2
3
4
5
region_il=# select * from sl_set;
 set_id | set_origin | set_locked |    set_comment   
--------+------------+------------+------------------------------------------
  1 |   1 |   | Set 1 (il_longlive_schema) for region_il
(1 row)

sl_set: 当前集群中的节点信息

?
1
2
3
4
5
6
7
region_il=# select * from sl_node;
 no_id | no_active |   no_comment    | no_failed
-------+-----------+-----------------------------------+-----------
| t   | Node 1 - region_il@10.189.102.118 | f
| t   | Node 2 - region_il@10.189.100.195 | f
| t   | Node 3 - region_il@10.189.100.226 | f
(3 rows)

sl_status: 显示远程节点落后主节点的延迟

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
region_il=# select * from sl_status;
-[ RECORD 1 ]-------------+------------------------------
st_origin     | 1
st_received    | 3
st_last_event    | 5000000800
st_last_event_ts   | 2017-06-15 01:03:19.585161-07
st_last_received   | 5000000799
st_last_received_ts  | 2017-06-15 01:03:14.281766-07
st_last_received_event_ts | 2017-06-15 01:03:09.57357-07
st_lag_num_events   | 1
st_lag_time    | 00:00:11.414753
-[ RECORD 2 ]-------------+------------------------------
st_origin     | 1
st_received    | 2
st_last_event    | 5000000800
st_last_event_ts   | 2017-06-15 01:03:19.585161-07
st_last_received   | 5000000799
st_last_received_ts  | 2017-06-15 01:03:14.193743-07
st_last_received_event_ts | 2017-06-15 01:03:09.57357-07
st_lag_num_events   | 1
st_lag_time    | 00:00:11.414753

sl_subscribe: 显示当前集群中的订阅者

?
1
2
3
4
5
6
region_il=# select * from sl_subscribe;
 sub_set | sub_provider | sub_receiver | sub_forward | sub_active
---------+--------------+--------------+-------------+------------
|   1 |   2 | t   | t
|   2 |   3 | t   | t
(2 rows)

 Slony-I日常维护

   1. 更改主表的表结构结构

     1.1. 查看要修改的表结构信息

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
region_il=# \d result_logbook;
           Table "il_longlive.result_logbook"
  Column  |   Type    |        Modifiers       
------------------+-----------------------------+--------------------------------------------------------------------
 record_id  | integer      | not null default nextval('result_logbook_record_id_seq'::regclass)
 schema   | character varying(255)  | not null
 vendor_code  | character varying(255)  |
 tag    | character varying(255)  |
 total_result_cnt | integer      |
 exist_result_cnt | integer      |
 new_result_cnt | integer      |
 max_result_id | integer      |
 islive   | boolean      | default true
 create_time  | timestamp without time zone | default now()
 update_time  | timestamp without time zone |
Indexes:
 "result_logbook_pkey" PRIMARY KEY, btree (record_id)

     1.2. 创建修改表字段的ddl语句文件

?
1
2
$ cat /home/postgres/alter_table_result_logbook.sql
alter table il_longlive.result_logbook add column address varchar(40);

     1.3. 通过执行slonik_execute_script命令

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ /usr/local/pgsql/bin/slonik_execute_script 1 /home/postgres/alter_table_result_logbook.sql
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 execute script (
 set id = 1,
 filename = '/home/postgres/alter_table_result_logbook.sql',
 event node = 1
 );
$ cat /home/postgres/alter_table_result_logbook.sh
#!/bin/sh
slonik << __EOF__
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 execute script (
 filename = '/home/postgres/alter_table_result_logbook.sql',
 event node = 1
 );
__EOF__
$ chmod +x /home/postgres/alter_table_result_logbook.sh
$ /home/postgres/alter_table_result_logbook.sh

     1.4. 验证所有节点上的表结构修改是否成功

?
1
2
3
4
5
6
7
8
9
$ psql -h 10.189.102.118 -U slonyadmin -d region_il -c "\d il_longlive.result_logbook" | grep address
Password for user slonyadmin:
 address   | character varying(40)
$ psql -h 10.189.100.195 -U slonyadmin -d region_il -c "\d il_longlive.result_logbook" | grep address
Password for user slonyadmin:
 address   | character varying(40)
$ psql -h 10.189.100.226 -U slonyadmin -d region_il -c "\d il_longlive.result_logbook" | grep address
Password for user slonyadmin:
 address   | character varying(40)

   2. 向现有集群中增加一个复制表

     2.1. 查看主库的的表结构信息,确认符合Slony-I的要求(主键)

?
1
2
3
4
5
6
7
8
9
10
11
12
13
region_il=# \d snapshot
          Table "il_longlive.snapshot"
  Column  |    Type    |      Modifiers     
-----------------+--------------------------------+-------------------------------------------------------
 id    | integer      | not null default nextval('snapshot_id_seq'::regclass)
 feed_schema  | character varying(255)   |
 job    | character varying(255)   |
 table_name  | character varying(255)   |
 max_primary_key | bigint       |
 create_time  | timestamp(0) without time zone |
 update_time  | timestamp(0) without time zone |
Indexes:
 "snapshot_pkey" PRIMARY KEY, btree (id)

      2.2. 将主表结构导入到所有辅节点

?
1
2
$ pg_dump -U slonyadmin -s -t il_longlive.snapshot region_il | psql -h 10.189.100.195 -U slonyadmin region_il
$ pg_dump -U slonyadmin -s -t il_longlive.snapshot region_il | psql -h 10.189.100.226 -U slonyadmin region_il

       2.3. 修改Slony-I配置文件slon_tools_conf,增加新复制集,并复制到所有节点(不是必须,为了各节点配置文件完整性)

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
$ cat /usr/local/pgsql/etc/slon_tools.conf
if ($ENV{"SLONYNODES"}) {
 require $ENV{"SLONYNODES"};
} else {
 $CLUSTER_NAME = 'region_il';
 $PIDFILE_DIR = '/usr/local/pgsql/data';
 $LOGDIR = '/usr/local/pgsql/data/pg_log';
 $MASTERNODE = 1;
 $DEBUGLEVEL = 0;
 $START_WATCHDOG = 1;
 $SLEEP_TIME = 60;
 $WATCHDOG_VERBOSE = 0;
 
 add_node(node  => 1,
    host  => '10.189.102.118',
    dbname => 'region_il',
    port  => 5432,
    user  => 'slonyadmin',
    password => '888888',);
 
 add_node(node  => 2,
    parent => 1,
    host  => '10.189.100.195',
    dbname => 'region_il',
    port  => 5432,
    user  => 'slonyadmin',
    password => '888888',);
 
 add_node(node  => 3,
    parent => 2,
    host  => '10.189.100.226',
    dbname => 'region_il',
    port  => 5432,
    user  => 'slonyadmin',
    password => '888888',);
}
 
 
 
$SLONY_SETS = {
 "il_longlive_schema" => {
  "set_id" => 1,
  "table_id" => 101,
  "sequence_id" => 101,
  "pkeyedtables" => [
       'il_longlive.entity',
       'il_longlive.entity_result',
       'il_longlive.result',
       'il_longlive.result_logbook',
       ],
 
  "keyedtables" => {},
 
  "sequences" => ['il_longlive.entity_entity_id_seq',
    'il_longlive.entity_result_id_seq',
   'il_longlive.result_id_seq',
   'il_longlive.result_logbook_record_id_seq',
      ],
 },
 
 "il_longlive_schema1" => {
  "set_id" => 2,
  "table_id" => 201,
  "sequence_id" => 201,
  "pkeyedtables" => [
       'il_longlive.snapshot',
       ],
 
  "keyedtables" => {},
 
  "sequences" => ['il_longlive.snapshot_id_seq'
      ],
 },
 
 
};
 
if ($ENV{"SLONYSET"}) {
 require $ENV{"SLONYSET"};
}
 
# Please do not add or change anything below this point.
 1;
$ scp /usr/local/pgsql/etc/slon_tools.conf postgres@10.189.100.195:/usr/local/pgsql/etc/slon_tools.conf
$ scp /usr/local/pgsql/etc/slon_tools.conf postgres@10.189.100.226:/usr/local/pgsql/etc/slon_tools.conf

       2.4. 在主节点创建新复制集

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
$ /usr/local/pgsql/bin/slonik_create_set --config /usr/local/pgsql/etc/slon_tools.conf 2
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 
# TABLE ADD KEY
 
# CREATE SET
 create set (id = 2, origin = 1, comment = 'Set 2 (il_longlive_schema1) for region_il');
 
# SET ADD TABLE
 echo 'Subscription set 2 (il_longlive_schema1) created';
 echo 'Adding tables to the subscription set';
 set add table (set id = 2, origin = 1, id = 201,
     full qualified name = 'il_longlive.snapshot',
     comment = 'Table il_longlive.snapshot with primary key');
 echo 'Add primary keyed table il_longlive.snapshot';
 
# SET ADD SEQUENCE
 echo 'Adding sequences to the subscription set';
 set add sequence (set id = 2, origin = 1, id = 201,
     full qualified name = 'il_longlive.snapshot_id_seq',
     comment = 'Sequence il_longlive.snapshot_id_seq');
 echo 'Add sequence il_longlive.snapshot_id_seq';
 echo 'All tables added';
$ /usr/local/pgsql/bin/slonik_create_set --config /usr/local/pgsql/etc/slon_tools.conf 2 | slonik
<stdin>:12: Subscription set 2 (il_longlive_schema1) created
<stdin>:13: Adding tables to the subscription set
<stdin>:17: Add primary keyed table il_longlive.snapshot
<stdin>:20: Adding sequences to the subscription set
<stdin>:24: Add sequence il_longlive.snapshot_id_seq
<stdin>:25: All tables added

       2.5. 在主节点创建新复制集的订阅者

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ /usr/local/pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 2 2
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 
 subscribe set (id = 2, provider = 1, receiver = 2, forward = yes);
 echo 'Subscribed nodes to set 2';
$ /usr/local/pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 2 3
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 
 subscribe set (id = 2, provider = 2, receiver = 3, forward = yes);
 echo 'Subscribed nodes to set 2';
$ /usr/local/pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 2 2 | slonik
<stdin>:7: Subscribed nodes to set 2
$ /usr/local/pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 2 3 | slonik
<stdin>:7: Subscribed nodes to set 2

        2.6. 查看所有辅节点数据同步状态

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ psql -h 10.189.102.118 -U slonyadmin -d region_il -c "select count(*) from il_longlive.snapshot;"  #主节点
Password for user slonyadmin:
 count
-------
(1 row)
 
$ psql -h 10.189.100.196 -U slonyadmin -d region_il -c "select count(*) from il_longlive.snapshot;"  #辅节点1
^C
[postgres@ec2t-pgtest-01 bin]$ psql -h 10.189.100.195 -U slonyadmin -d region_il -c "select count(*) from il_longlive.snapshot;"
Password for user slonyadmin:
 count
-------
(1 row)
 
$ psql -h 10.189.100.226 -U slonyadmin -d region_il -c "select count(*) from il_longlive.snapshot;"  #辅节点2
Password for user slonyadmin:
 count
-------
(1 row)

         2.7. 查看Slony-I系统表信息

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
region_il=# select * from sl_set;
 set_id | set_origin | set_locked |    set_comment   
--------+------------+------------+-------------------------------------------
|   1 |   | Set 1 (il_longlive_schema) for region_il
|   1 |   | Set 2 (il_longlive_schema1) for region_il
(2 rows)
 
region_il=# select * from sl_table;
 tab_id | tab_reloid | tab_relname | tab_nspname | tab_set |  tab_idxname  | tab_altered |     tab_comment    
--------+------------+----------------+-------------+---------+---------------------+-------------+---------------------------------------------------
|  19350 | entity   | il_longlive |  1 | entity_pkey   | f   | Table il_longlive.entity with primary key
|  19355 | entity_result | il_longlive |  1 | entity_result_pkey | f   | Table il_longlive.entity_result with primary key
|  19360 | result   | il_longlive |  1 | result_pkey   | f   | Table il_longlive.result with primary key
|  19365 | result_logbook | il_longlive |  1 | result_logbook_pkey | f   | Table il_longlive.result_logbook with primary key
|  19380 | snapshot  | il_longlive |  2 | snapshot_pkey  | f   | Table il_longlive.snapshot with primary key
(5 rows)

         2.8. 合并复制集

?
1
2
3
4
5
6
7
8
9
$ /usr/local/pgsql/bin/slonik_merge_sets --config /usr/local/pgsql/etc/slon_tools.conf 1 1 2
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 merge set (id = 1, add id = 2, origin = 1);
 echo 'Replication set 2 merged into 1 on origin 1. Set 2 no longer exists.';
$ /usr/local/pgsql/bin/slonik_merge_sets --config /usr/local/pgsql/etc/slon_tools.conf 1 1 2 | slonik
<stdin>:6: Replication set 2 merged into 1 on origin 1. Set 2 no longer exists.

         2.9. 查看Slony-I系统表信息

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
region_il=# select * from sl_set;
 set_id | set_origin | set_locked |    set_comment   
--------+------------+------------+------------------------------------------
|   1 |   | Set 1 (il_longlive_schema) for region_il
(1 row)
 
region_il=# select * from sl_table;
 tab_id | tab_reloid | tab_relname | tab_nspname | tab_set |  tab_idxname  | tab_altered |     tab_comment    
--------+------------+----------------+-------------+---------+---------------------+-------------+---------------------------------------------------
|  19350 | entity   | il_longlive |  1 | entity_pkey   | f   | Table il_longlive.entity with primary key
|  19355 | entity_result | il_longlive |  1 | entity_result_pkey | f   | Table il_longlive.entity_result with primary key
|  19360 | result   | il_longlive |  1 | result_pkey   | f   | Table il_longlive.result with primary key
|  19365 | result_logbook | il_longlive |  1 | result_logbook_pkey | f   | Table il_longlive.result_logbook with primary key
|  19380 | snapshot  | il_longlive |  1 | snapshot_pkey  | f   | Table il_longlive.snapshot with primary key
(5 rows)

 3. 向现有集群中移除一个复制表

         3.1. 查看Slony-I系统视图确认要移除的表

?
1
2
3
4
5
6
7
8
9
region_il=# select * from sl_table;
 tab_id | tab_reloid | tab_relname | tab_nspname | tab_set |  tab_idxname  | tab_altered |     tab_comment    
--------+------------+----------------+-------------+---------+---------------------+-------------+---------------------------------------------------
|  19350 | entity   | il_longlive |  1 | entity_pkey   | f   | Table il_longlive.entity with primary key
|  19355 | entity_result | il_longlive |  1 | entity_result_pkey | f   | Table il_longlive.entity_result with primary key
|  19360 | result   | il_longlive |  1 | result_pkey   | f   | Table il_longlive.result with primary key
|  19365 | result_logbook | il_longlive |  1 | result_logbook_pkey | f   | Table il_longlive.result_logbook with primary key
|  19380 | snapshot  | il_longlive |  1 | snapshot_pkey  | f   | Table il_longlive.snapshot with primary key
(5 rows)

         3.2. 移除集群中指定的表

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ /usr/local/pgsql/bin/slonik_drop_table --config /usr/local/pgsql/etc/slon_tools.conf 201 1
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 
# DROP TABLE
 try {
 SET DROP TABLE (id = 201, origin = 1);
 } on error {
 echo 'Could not drop table 201 for region_il!';
 exit 1;
 }
 
$ /usr/local/pgsql/bin/slonik_drop_table --config /usr/local/pgsql/etc/slon_tools.conf 201 1 | slonik

          3.3. 验证集群中指定的表是否移除

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
region_il=# select * from sl_table;
 tab_id | tab_reloid | tab_relname | tab_nspname | tab_set |  tab_idxname  | tab_altered |     tab_comment    
--------+------------+----------------+-------------+---------+---------------------+-------------+---------------------------------------------------
|  19350 | entity   | il_longlive |  1 | entity_pkey   | f   | Table il_longlive.entity with primary key
|  19355 | entity_result | il_longlive |  1 | entity_result_pkey | f   | Table il_longlive.entity_result with primary key
|  19360 | result   | il_longlive |  1 | result_pkey   | f   | Table il_longlive.result with primary key
|  19365 | result_logbook | il_longlive |  1 | result_logbook_pkey | f   | Table il_longlive.result_logbook with primary key
(4 rows)
 
region_il=# \d il_longlive.snapshot  #可以看到对应的表上的触发器也被移除了
            Table "il_longlive.snapshot"
  Column  |    Type    |        Modifiers       
-----------------+--------------------------------+-------------------------------------------------------------------
 id    | integer      | not null default nextval('il_longlive.snapshot_id_seq'::regclass)
 feed_schema  | character varying(255)   |
 job    | character varying(255)   |
 table_name  | character varying(255)   |
 max_primary_key | bigint       |
 create_time  | timestamp(0) without time zone |
 update_time  | timestamp(0) without time zone |
Indexes:
 "snapshot_pkey" PRIMARY KEY, btree (id)

 4. 向现有集群中移除一个节点

         4.1. 查看Slony-I系统视图确认要移除的节点

?
1
2
3
4
5
6
7
region_il=# select * from sl_node;
 no_id | no_active |   no_comment    | no_failed
-------+-----------+-----------------------------------+-----------
| t   | Node 1 - region_il@10.189.102.118 | f
| t   | Node 2 - region_il@10.189.100.195 | f
| t   | Node 3 - region_il@10.189.100.226 | f
(3 rows)

         4.2. 移除集群中指定的节点

?
1
2
3
4
5
6
7
8
9
10
11
$ /usr/local/pgsql/bin/slonik_drop_node --config /usr/local/pgsql/etc/slon_tools.conf 3 1
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 drop node (id = 3, event node = 1);
 echo 'dropped node 3 cluster';
$ /usr/local/pgsql/bin/slonik_drop_node --config /usr/local/pgsql/etc/slon_tools.conf 3 1 | slonik
debug: waiting for 3,5000019408 on 1
debug: waiting for 3,5000019408 on 1
<stdin>:6: dropped node 3 cluster

          4.3. 验证集群中指定的节点是否移除(此时被移除节点的Slony-I系统表等信息已经被移除)

?
1
2
3
4
5
6
region_il=# select * from sl_node;
 no_id | no_active |   no_comment    | no_failed
-------+-----------+-----------------------------------+-----------
| t   | Node 1 - region_il@10.189.102.118 | f
| t   | Node 2 - region_il@10.189.100.195 | f
(2 rows)

          4.4. 停止被关闭节点的slon进程

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ ps -ef | grep slon
postgres 6033  1 0 Jun02 ?  00:00:05 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=ec2t-pgtest-03 dbname=region_il user=slonyadmin port=5432 password=mypna123
postgres 6050  1 0 Jun02 ?  00:00:03 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node2 60
postgres 12596 6033 0 05:00 ?  00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=ec2t-pgtest-03 dbname=region_il user=slonyadmin port=5432 password=mypna123
postgres 12598 19212 0 05:00 ?  00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3.pid -s 1000 -d0 region_il host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888
postgres 12599 15237 0 05:00 ?  00:00:00 postgres: slonyadmin region_il 10.189.100.226(29630) idle
postgres 12601 5757 0 05:00 pts/3 00:00:00 grep slon
postgres 19212  1 0 Jun14 ?  00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3.pid -s 1000 -d0 region_il host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888
postgres 19232  1 0 Jun14 ?  00:00:00 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node3 60
$ slon_status 3
Slon is running for the 'region_il' cluster on node3.
$ slon_kill
slon_kill.pl... Killing all slon and slon_watchdog instances for the cluster region_il
1. Kill slon watchdogs
slon_watchdog for cluster region_il node 2 killed - PID [6050]
slon_watchdog for cluster region_il node 3 killed - PID [19232]
$ ps -ef | grep slon
postgres 12820 5757 0 05:05 pts/3 00:00:00 grep slon

 5. 向现有集群中增加一个节点

          5.1. 在新节点安装配置Slony-I(参考之前步骤)

          5.2. 查看当街Slony-I系统表节点信息

?
1
2
3
4
5
6
region_il=# select * from sl_node;
 no_id | no_active |   no_comment    | no_failed
-------+-----------+-----------------------------------+-----------
| t   | Node 1 - region_il@10.189.102.118 | f
| t   | Node 2 - region_il@10.189.100.195 | f
(2 rows)

          5.3. 向Slony-I配置文件slon_tools.conf 增加新节点信息,并将新配置文件更新到所有节点

?
1
2
3
4
5
6
7
8
9
10
$ cat /usr/local/pgsql/etc/slon_tools.conf
...........................................
 add_node(node  => 3,
    parent => 1,
    host  => '10.189.100.226',
    dbname => 'region_il',
    port  => 5432,
    user  => 'slonyadmin',
    password => '888888',);
............................................

          5.4. 在主节点上增加向集群新节点信息

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ /usr/local/pgsql/bin/slonik_add_node --config /usr/local/pgsql/etc/slon_tools.conf 3 1
 
# ADD NODE
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 try {
  store node (id = 3, event node = 1, comment = 'Node 3 - region_il@10.189.100.226');
 } on error {
  echo 'Failed to add node to cluster';
  exit 1;
 }
 
# STORE PATHS
 store path (server = 3, client = 1, conninfo = 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888');
 store path (server = 1, client = 3, conninfo = 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888');
 store path (server = 3, client = 2, conninfo = 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888');
 store path (server = 2, client = 3, conninfo = 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888');
 echo 'added node 3 to cluster';
 echo 'Please start a slon replication daemon for node 3';
$ /usr/local/pgsql/bin/slonik_add_node --config /usr/local/pgsql/etc/slon_tools.conf 3 1 | slonik
<stdin>:7: Possible unsupported PostgreSQL version (90505) 9.5, defaulting to 8.4 support
<stdin>:19: added node 3 to cluster
<stdin>:20: Please start a slon replication daemon for node 3

           5.5. 在新节点启动slon进程

?
1
2
3
4
5
$ slon_start 3
Invoke slon for node 3 - /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3.pid -s 1000 -d0 region_il 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' > /usr/local/pgsql/data/pg_log/node3/region_il-2017-06-17.log 2>&1 &
Slon successfully started for cluster region_il, node node3
PID [13136]
Start the watchdog process as well...

           5.6. 在主节点添加新节点的复制订阅信息

?
1
2
3
4
5
6
7
8
9
10
11
12
$ /usr/local/pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/etc/slon_tools.conf 1 3
Can't locate /usr/local/pgsql/etc/etc/slon_tools.conf in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/local/pgsql/bin/slonik_subscribe_set line 29.
[postgres@ec2t-pgtest-01 bin]$ /usr/local/pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 1 3
cluster name = region_il;
 node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888';
 node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';
 
 subscribe set (id = 1, provider = 1, receiver = 3, forward = yes);
 echo 'Subscribed nodes to set 1';
[postgres@ec2t-pgtest-01 bin]$ /usr/local/pgsql/bin/slonik_subscribe_set --config /usr/local/pgsql/etc/slon_tools.conf 1 3 | slonik
<stdin>:7: Subscribed nodes to set 1

5.7. 验证数据同步(略)

总结

以上就是这篇文章的全部内容了，希望本文的内容对大家的学习或者工作具有一定的参考学习价值，如果有疑问大家可以留言交流，谢谢大家对服务器之家的支持。

原文链接：https://www.cnblogs.com/ilifeilong/p/7009322.html

标签：
PostgreSQL Slony-I 同步复制 部署教程 

相关文章

热门资讯

沙雕群名称大全2019精选 今年最火的微信群名沙雕有创意
沙雕群名称大全2019精选 今年最火的微信群名沙雕有创意 2019-07-07
玄元剑仙肉身有什么用 玄元剑仙肉身境界等级划分
玄元剑仙肉身有什么用 玄元剑仙肉身境界等级划分 2019-06-21
男生常说24816是什么意思？女生说13579是什么意思？
男生常说24816是什么意思？女生说13579是什么意思？ 2019-09-17
超A是什么意思 你好a表达的是什么
超A是什么意思 你好a表达的是什么 2019-06-06
和平精英超高音质怎么修改 和平精英超高音质重启设置方法
和平精英超高音质怎么修改 和平精英超高音质重启设置方法 2019-06-26
返回顶部