Flink(一)：编译源码

前言

以前用过Flink的1.7版本，最近要做实时数仓，于是下载flink源码编译一下，准备精读flink源码

Flink版本为：1.11

Git 配置代理

先科学上网，不然 Git clone flink源码的速度很慢、还会中断

1
2
3
4
5
6
# 设置代理
git config --global http.proxy 'socks5://127.0.0.1:1080'
git config --global https.proxy 'socks5://127.0.0.1:1080'
# 取消代理
git config --global --unset http.proxy
git config --global --unset https.proxy

下载源码

最好把源码fork到自己的仓库，切换dev分支以方便自己后续做源码注释

1
git clone https://github.com/heyzeng/flink.git

编译源码

编译源码可能会出现很多状况，根据实际问题多排查几次。

我遇到的问题是flink-runtime-web模块一直编译失败，报错信息maven仓库内的node有问题，解决方法：maven仓库中删除下载好的node包即可

1
mvn clean package -DskipTests # 编译命令

编译成功

官网说编写需要10分钟左右，我是用MacbookPro编译的，大概时间在23分钟左右。(具体编译时间根据配置吧)。下面即为编译成功

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
[INFO] Reactor Summary:
[INFO]
[INFO] force-shading ...................................... SUCCESS [  0.940 s]
[INFO] flink .............................................. SUCCESS [  1.380 s]
[INFO] flink-annotations .................................. SUCCESS [  1.014 s]
[INFO] flink-test-utils-parent ............................ SUCCESS [  0.118 s]
[INFO] flink-test-utils-junit ............................. SUCCESS [  1.030 s]
[INFO] flink-metrics ...................................... SUCCESS [  0.104 s]
[INFO] flink-metrics-core ................................. SUCCESS [  0.594 s]
[INFO] flink-core ......................................... SUCCESS [ 28.202 s]
[INFO] flink-java ......................................... SUCCESS [  4.413 s]
[INFO] flink-queryable-state .............................. SUCCESS [  0.083 s]
[INFO] flink-queryable-state-client-java .................. SUCCESS [  0.926 s]
[INFO] flink-filesystems .................................. SUCCESS [  0.093 s]
[INFO] flink-hadoop-fs .................................... SUCCESS [  1.682 s]
[INFO] flink-runtime ...................................... SUCCESS [01:22 min]
[INFO] flink-scala ........................................ SUCCESS [ 30.323 s]
[INFO] flink-mapr-fs ...................................... SUCCESS [  0.622 s]
[INFO] flink-filesystems :: flink-fs-hadoop-shaded ........ SUCCESS [  3.503 s]
[INFO] flink-s3-fs-base ................................... SUCCESS [  1.036 s]
[INFO] flink-s3-fs-hadoop ................................. SUCCESS [  4.255 s]
[INFO] flink-s3-fs-presto ................................. SUCCESS [  6.010 s]
[INFO] flink-swift-fs-hadoop .............................. SUCCESS [ 14.518 s]
[INFO] flink-oss-fs-hadoop ................................ SUCCESS [  4.389 s]
[INFO] flink-azure-fs-hadoop .............................. SUCCESS [  5.864 s]
[INFO] flink-optimizer .................................... SUCCESS [ 10.958 s]
[INFO] flink-streaming-java ............................... SUCCESS [ 10.753 s]
[INFO] flink-clients ...................................... SUCCESS [  1.620 s]
[INFO] flink-test-utils ................................... SUCCESS [  1.161 s]
[INFO] flink-runtime-web .................................. SUCCESS [03:14 min]
[INFO] flink-examples ..................................... SUCCESS [  0.380 s]
[INFO] flink-examples-batch ............................... SUCCESS [ 12.707 s]
[INFO] flink-connectors ................................... SUCCESS [  0.138 s]
[INFO] flink-hadoop-compatibility ......................... SUCCESS [  4.821 s]
[INFO] flink-state-backends ............................... SUCCESS [  0.080 s]
[INFO] flink-statebackend-rocksdb ......................... SUCCESS [  1.981 s]
[INFO] flink-tests ........................................ SUCCESS [ 35.178 s]
[INFO] flink-streaming-scala .............................. SUCCESS [ 27.274 s]
[INFO] flink-table ........................................ SUCCESS [  0.066 s]
[INFO] flink-table-common ................................. SUCCESS [  3.623 s]
[INFO] flink-table-api-java ............................... SUCCESS [  2.447 s]
[INFO] flink-table-api-java-bridge ........................ SUCCESS [  1.020 s]
[INFO] flink-table-api-scala .............................. SUCCESS [  9.194 s]
[INFO] flink-table-api-scala-bridge ....................... SUCCESS [  7.430 s]
[INFO] flink-sql-parser ................................... SUCCESS [  3.624 s]
[INFO] flink-libraries .................................... SUCCESS [  0.067 s]
[INFO] flink-cep .......................................... SUCCESS [  2.877 s]
[INFO] flink-table-planner ................................ SUCCESS [01:31 min]
[INFO] flink-table-runtime-blink .......................... SUCCESS [  4.578 s]
[INFO] flink-table-planner-blink .......................... SUCCESS [01:53 min]
[INFO] flink-jdbc ......................................... SUCCESS [  1.175 s]
[INFO] flink-hbase ........................................ SUCCESS [  4.037 s]
[INFO] flink-hcatalog ..................................... SUCCESS [  3.935 s]
[INFO] flink-metrics-jmx .................................. SUCCESS [  0.480 s]
[INFO] flink-formats ...................................... SUCCESS [  0.061 s]
[INFO] flink-json ......................................... SUCCESS [  0.578 s]
[INFO] flink-connector-kafka-base ......................... SUCCESS [  2.117 s]
[INFO] flink-connector-kafka-0.10 ......................... SUCCESS [  1.145 s]
[INFO] flink-connector-kafka-0.11 ......................... SUCCESS [  1.073 s]
[INFO] flink-connector-elasticsearch-base ................. SUCCESS [  1.782 s]
[INFO] flink-connector-elasticsearch5 ..................... SUCCESS [ 10.934 s]
[INFO] flink-connector-elasticsearch6 ..................... SUCCESS [  1.396 s]
[INFO] flink-connector-elasticsearch7 ..................... SUCCESS [  1.511 s]
[INFO] flink-orc .......................................... SUCCESS [  0.855 s]
[INFO] flink-orc-nohive ................................... SUCCESS [  0.370 s]
[INFO] flink-avro ......................................... SUCCESS [  2.053 s]
[INFO] flink-parquet ...................................... SUCCESS [  1.513 s]
[INFO] flink-csv .......................................... SUCCESS [  0.459 s]
[INFO] flink-connector-hive ............................... SUCCESS [  5.203 s]
[INFO] flink-connector-rabbitmq ........................... SUCCESS [  0.469 s]
[INFO] flink-connector-twitter ............................ SUCCESS [  1.704 s]
[INFO] flink-connector-nifi ............................... SUCCESS [  0.649 s]
[INFO] flink-connector-cassandra .......................... SUCCESS [  2.709 s]
[INFO] flink-connector-filesystem ......................... SUCCESS [  0.927 s]
[INFO] flink-connector-kafka .............................. SUCCESS [  1.188 s]
[INFO] flink-connector-gcp-pubsub ......................... SUCCESS [ 12.080 s]
[INFO] flink-connector-kinesis ............................ SUCCESS [  8.404 s]
[INFO] flink-sql-connector-elasticsearch7 ................. SUCCESS [  7.230 s]
[INFO] flink-sql-connector-elasticsearch6 ................. SUCCESS [  4.693 s]
[INFO] flink-sql-connector-kafka-0.10 ..................... SUCCESS [  0.394 s]
[INFO] flink-sql-connector-kafka-0.11 ..................... SUCCESS [  0.473 s]
[INFO] flink-sql-connector-kafka .......................... SUCCESS [  0.737 s]
[INFO] flink-sql-connector-hive-1.2.2 ..................... SUCCESS [ 39.934 s]
[INFO] flink-sql-connector-hive-2.2.0 ..................... SUCCESS [02:01 min]
[INFO] flink-sql-connector-hive-2.3.6 ..................... SUCCESS [ 14.528 s]
[INFO] flink-sql-connector-hive-3.1.2 ..................... SUCCESS [02:43 min]
[INFO] flink-avro-confluent-registry ...................... SUCCESS [  1.466 s]
[INFO] flink-sequence-file ................................ SUCCESS [  0.408 s]
[INFO] flink-compress ..................................... SUCCESS [  0.335 s]
[INFO] flink-examples-streaming ........................... SUCCESS [ 10.036 s]
[INFO] flink-examples-table ............................... SUCCESS [  5.698 s]
[INFO] flink-examples-build-helper ........................ SUCCESS [  0.125 s]
[INFO] flink-examples-streaming-twitter ................... SUCCESS [  0.653 s]
[INFO] flink-examples-streaming-state-machine ............. SUCCESS [  0.559 s]
[INFO] flink-examples-streaming-gcp-pubsub ................ SUCCESS [  3.278 s]
[INFO] flink-container .................................... SUCCESS [  0.458 s]
[INFO] flink-queryable-state-runtime ...................... SUCCESS [  0.728 s]
[INFO] flink-end-to-end-tests ............................. SUCCESS [  1.026 s]
[INFO] flink-cli-test ..................................... SUCCESS [  0.260 s]
[INFO] flink-parent-child-classloading-test-program ....... SUCCESS [  0.239 s]
[INFO] flink-parent-child-classloading-test-lib-package ... SUCCESS [  0.185 s]
[INFO] flink-dataset-allround-test ........................ SUCCESS [  0.243 s]
[INFO] flink-dataset-fine-grained-recovery-test ........... SUCCESS [  0.272 s]
[INFO] flink-datastream-allround-test ..................... SUCCESS [  1.361 s]
[INFO] flink-batch-sql-test ............................... SUCCESS [  0.241 s]
[INFO] flink-stream-sql-test .............................. SUCCESS [  0.256 s]
[INFO] flink-bucketing-sink-test .......................... SUCCESS [  0.354 s]
[INFO] flink-distributed-cache-via-blob ................... SUCCESS [  0.250 s]
[INFO] flink-high-parallelism-iterations-test ............. SUCCESS [  5.731 s]
[INFO] flink-stream-stateful-job-upgrade-test ............. SUCCESS [  0.759 s]
[INFO] flink-queryable-state-test ......................... SUCCESS [  1.371 s]
[INFO] flink-local-recovery-and-allocation-test ........... SUCCESS [  0.384 s]
[INFO] flink-elasticsearch5-test .......................... SUCCESS [  3.800 s]
[INFO] flink-elasticsearch6-test .......................... SUCCESS [  2.483 s]
[INFO] flink-quickstart ................................... SUCCESS [  0.687 s]
[INFO] flink-quickstart-java .............................. SUCCESS [  0.461 s]
[INFO] flink-quickstart-scala ............................. SUCCESS [  0.209 s]
[INFO] flink-quickstart-test .............................. SUCCESS [  0.370 s]
[INFO] flink-confluent-schema-registry .................... SUCCESS [  1.466 s]
[INFO] flink-stream-state-ttl-test ........................ SUCCESS [  2.814 s]
[INFO] flink-sql-client-test .............................. SUCCESS [  0.521 s]
[INFO] flink-streaming-file-sink-test ..................... SUCCESS [  0.233 s]
[INFO] flink-state-evolution-test ......................... SUCCESS [  0.798 s]
[INFO] flink-rocksdb-state-memory-control-test ............ SUCCESS [  0.819 s]
[INFO] flink-mesos ........................................ SUCCESS [ 16.487 s]
[INFO] flink-kubernetes ................................... SUCCESS [  3.886 s]
[INFO] flink-yarn ......................................... SUCCESS [  1.088 s]
[INFO] flink-gelly ........................................ SUCCESS [  3.125 s]
[INFO] flink-gelly-scala .................................. SUCCESS [ 11.907 s]
[INFO] flink-gelly-examples ............................... SUCCESS [  6.967 s]
[INFO] flink-metrics-dropwizard ........................... SUCCESS [  0.376 s]
[INFO] flink-metrics-graphite ............................. SUCCESS [  0.240 s]
[INFO] flink-metrics-influxdb ............................. SUCCESS [  0.831 s]
[INFO] flink-metrics-prometheus ........................... SUCCESS [  0.496 s]
[INFO] flink-metrics-statsd ............................... SUCCESS [  0.274 s]
[INFO] flink-metrics-datadog .............................. SUCCESS [  0.308 s]
[INFO] flink-metrics-slf4j ................................ SUCCESS [  0.243 s]
[INFO] flink-cep-scala .................................... SUCCESS [  8.329 s]
[INFO] flink-table-uber ................................... SUCCESS [  1.945 s]
[INFO] flink-table-uber-blink ............................. SUCCESS [  2.114 s]
[INFO] flink-sql-client ................................... SUCCESS [  3.069 s]
[INFO] flink-state-processor-api .......................... SUCCESS [  0.896 s]
[INFO] flink-python ....................................... SUCCESS [  9.620 s]
[INFO] flink-ml-parent .................................... SUCCESS [  0.059 s]
[INFO] flink-ml-api ....................................... SUCCESS [  0.377 s]
[INFO] flink-ml-lib ....................................... SUCCESS [  0.858 s]
[INFO] flink-ml-uber ...................................... SUCCESS [  0.214 s]
[INFO] flink-scala-shell .................................. SUCCESS [  8.177 s]
[INFO] flink-dist ......................................... SUCCESS [ 43.227 s]
[INFO] flink-end-to-end-tests-common ...................... SUCCESS [  0.828 s]
[INFO] flink-metrics-availability-test .................... SUCCESS [  0.266 s]
[INFO] flink-metrics-reporter-prometheus-test ............. SUCCESS [  0.291 s]
[INFO] flink-heavy-deployment-stress-test ................. SUCCESS [  5.572 s]
[INFO] flink-connector-gcp-pubsub-emulator-tests .......... SUCCESS [  1.116 s]
[INFO] flink-streaming-kafka-test-base .................... SUCCESS [  0.336 s]
[INFO] flink-streaming-kafka-test ......................... SUCCESS [  5.050 s]
[INFO] flink-streaming-kafka011-test ...................... SUCCESS [  4.449 s]
[INFO] flink-streaming-kafka010-test ...................... SUCCESS [  4.567 s]
[INFO] flink-plugins-test ................................. SUCCESS [  0.080 s]
[INFO] dummy-fs ........................................... SUCCESS [  0.131 s]
[INFO] another-dummy-fs ................................... SUCCESS [  0.127 s]
[INFO] flink-tpch-test .................................... SUCCESS [  0.620 s]
[INFO] flink-streaming-kinesis-test ....................... SUCCESS [  9.350 s]
[INFO] flink-elasticsearch7-test .......................... SUCCESS [  2.891 s]
[INFO] flink-end-to-end-tests-common-kafka ................ SUCCESS [  0.888 s]
[INFO] flink-tpcds-test ................................... SUCCESS [  0.664 s]
[INFO] flink-netty-shuffle-memory-control-test ............ SUCCESS [  0.214 s]
[INFO] flink-statebackend-heap-spillable .................. SUCCESS [  0.614 s]
[INFO] flink-contrib ...................................... SUCCESS [  0.072 s]
[INFO] flink-connector-wikiedits .......................... SUCCESS [  0.440 s]
[INFO] flink-yarn-tests ................................... SUCCESS [ 14.048 s]
[INFO] flink-fs-tests ..................................... SUCCESS [  0.493 s]
[INFO] flink-docs ......................................... SUCCESS [  0.958 s]
[INFO] flink-walkthroughs ................................. SUCCESS [  0.074 s]
[INFO] flink-walkthrough-common ........................... SUCCESS [  0.656 s]
[INFO] flink-walkthrough-table-java ....................... SUCCESS [  0.141 s]
[INFO] flink-walkthrough-table-scala ...................... SUCCESS [  0.144 s]
[INFO] flink-walkthrough-datastream-java .................. SUCCESS [  0.201 s]
[INFO] flink-walkthrough-datastream-scala ................. SUCCESS [  0.168 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:11 min
[INFO] Finished at: 2020-06-15T15:50:14+08:00
[INFO] Final Memory: 674M/2582M
[INFO] ------------------------------------------------------------------------