tez参数优化

Tez引擎跑HQL时候一直报错

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
--------------------------------------------------------------------------------
        VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
--------------------------------------------------------------------------------
Map 1 ..........   SUCCEEDED      2          2        0        0       7       0
Reducer 2            RUNNING      1          0        1        0       0       0
--------------------------------------------------------------------------------
VERTICES: 01/02  [=================>>---------] 66%   ELAPSED TIME: 11.22 s
--------------------------------------------------------------------------------
Status: Failed
Vertex re-running, vertexName=Map 1, vertexId=vertex_1507606976106_0043_2_00
Vertex re-running, vertexName=Map 1, vertexId=vertex_1507606976106_0043_2_00
Vertex failed, vertexName=Map 1, vertexId=vertex_1507606976106_0043_2_00, diagnostics=[Vertex vertex_1507606976106_0043_2_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE, Vertex vertex_1507606976106_0043_2_00 [Map 1] failed as task task_1507606976106_0043_2_00_000001 failed after vertex succeeded.]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map 1, vertexId=vertex_1507606976106_0043_2_00Vertex re-running, vertexName=Map 1, vertexId=vertex_1507606976106_0043_2_00Vertex failed, vertexName=Map 1, vertexId=vertex_1507606976106_0043_2_00, diagnostics=[Vertex vertex_1507606976106_0043_2_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE, Vertex vertex_1507606976106_0043_2_00 [Map 1] failed as task task_1507606976106_0043_2_00_000001 failed after vertex succeeded.]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
expr: syntax error

Tez调优参数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
set hive.execution.engine=tez; 
set hive.auto.convert.join=true; 
set hive.auto.convert.join.noconditionaltask=true; 
set hive.auto.convert.join.noconditionaltask.size=405306368; 
set hive.vectorized.execution.enabled=true; 
set hive.vectorized.execution.reduce.enabled =true; 
set hive.cbo.enable=true; 
set hive.compute.query.using.stats=true; 
set hive.stats.fetch.column.stats=true; 
set hive.merge.mapfiles =true; 
set hive.merge.mapredfiles=true; 
set hive.merge.size.per.task=134217728; 
set hive.merge.smallfiles.avgsize=44739242; 
set mapreduce.job.reduce.slowstart.completedmaps=0.8;

Tez参数详解