博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
hive在命令行消除进度等错误信息
阅读量:6258 次
发布时间:2019-06-22

本文共 2757 字,大约阅读时间需要 9 分钟。

大家在使用shell脚本调用hive命令的时候,发现hive的中间过程竟然打印到错误输出流里面,这样在查看错误日志的时候,需要过滤这些没用的信息,那么可以使用如下的配置参数。

set hive.session.silent=true; (默认是false)

例如:

hive> select from_original,pv from tableName where rpt_date='2014-12-08' order by pv desc limit 4;Total MapReduce jobs = 1Launching Job 1 out of 1Number of reduce tasks determined at compile time: 1In order to change the average load for a reducer (in bytes):  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers: set hive.exec.reducers.max=
In order to set a constant number of reducers: set mapred.reduce.tasks=
Starting Job = job_1417682027300_928652, Tracking URL = http://l-hdpm4.data.cn6.qunar.com:9981/proxy/application_1417682027300_928652/Kill Command = /home/q/hadoop/hadoop-2.2.0/bin/hadoop job -kill job_1417682027300_928652Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 12015-01-09 11:28:07,561 Stage-1 map = 0%, reduce = 0%2015-01-09 11:28:12,735 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.15 sec2015-01-09 11:28:13,766 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.15 sec2015-01-09 11:28:14,796 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.15 sec2015-01-09 11:28:15,826 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.15 sec2015-01-09 11:28:16,859 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.15 sec2015-01-09 11:28:17,892 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.15 sec2015-01-09 11:28:18,925 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.73 sec2015-01-09 11:28:19,958 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.73 secMapReduce Total cumulative CPU time: 2 seconds 730 msecEnded Job = job_1417682027300_928652MapReduce Jobs Launched: Job 0: Map: 1 Reduce: 1 Cumulative CPU: 2.73 sec HDFS Read: 11815 HDFS Write: 83 SUCCESSTotal MapReduce CPU Time Spent: 2 seconds 730 msecOKsuggest 6ts_hotcity 5suggest2 4mps_remdd 3Time taken: 18.502 seconds, Fetched: 4 row(s)

不过我们需要的信息就最后面那么几行,可以如此设置

hive> set hive.session.silent=true;                                                                           hive> select from_original,pv from tableName where rpt_date='2014-12-08' order by pv desc limit 4;suggest      6ts_hotcity      5suggest2      4mps_remdd       3

大家可能会有两个疑问:

1、hive在哪&怎么将info和warning信息打印到标准错误流里面的?

那么给大家看一段代码(org.apache.hadoop.hive.cli.CliDriver):

try {      ss.out = new PrintStream(System.out, true, "UTF-8");      ss.info = new PrintStream(System.err, true, "UTF-8");      ss.err = new CachingPrintStream(System.err, true, "UTF-8");    } catch (UnsupportedEncodingException e) {      return 3;    }

可见,使用的是System.err方法。

2、为什么需要打印到错误输出流里面

我的理解是便于命令行获取结果,并经这些内容要输出,如果放到标准输出里面,和结果就混淆了

转载地址:http://dnqsa.baihongyu.com/

你可能感兴趣的文章
[zz]struct epoll_event
查看>>
[PAL编程规范]SAP HANA PAL线性回归预测分析Linear Regression编程规范LRREGRESSION(模型)...
查看>>
ORA-00001: unique constraint violated并不一定是主键冲突
查看>>
SecureCRT设置背景颜色
查看>>
window下在同一台机器上安装多个版本jdk,修改环境变量不生效问题处理办法
查看>>
How to update WPF browser application manifest and xbap file with ‘mage.exe’
查看>>
SQL Server T-SQL高级查询
查看>>
通过配置CPU参数 worker_cpu_affinity 提升nginx性能
查看>>
解决的方法:warning: Clock skew detected. Your build may be incomplete.
查看>>
技术型创业者easy遇到的三大问题
查看>>
无法打开SQL Server的连接
查看>>
java泛型中的对象
查看>>
进程和线程区别理解
查看>>
php创建token
查看>>
Android 系统API实现数据库的增删改查和SQLite3工具的使用
查看>>
95、Android下在onCreate中获取控件的宽度和高度(通过回调)
查看>>
UML在需求分析阶段的应用
查看>>
JavaScript:JavaScript事件的处理
查看>>
WEB安全测试的类型
查看>>
ES6笔记(7)-- Promise异步编程
查看>>