python操作hive

一时失言乱红尘 2021-09-20 03:22 354阅读 0赞

[2019独角兽企业重金招聘Python工程师标准>>> ][2019_Python_] ![hot3.png][]

# 前言 #

HiveServer2为客户端在远程执行hive查询提供了接口，通过Thrift RPC来实现，还提供了多用户并发和认证功能。目前使用python的用户可以通过pyhs2这个模块来连接HiveServer2，实现查询和取回结果的操作。python客户端采用pyhs2模块

# 安装python工具模块 #

1.  安转pip [https://pip.pypa.io/en/stable/installing/][https_pip.pypa.io_en_stable_installing]
2.  安装依赖模块
    
     *  yum install cyrus-sasl-plain
     *  yum install cyrus-sasl-devel
     *  yum install python-devel
3.  pip install pyhs2

# python客户端代码 #

#!/usr/bin/env python
    # -*- coding: utf-8 -*-
    # hive util with hive server2
    """
    [@author](https://my.oschina.net/arthor):wyf
    [@create](https://my.oschina.net/u/192469):2016-06-29 16:55
    """
    __author__ = 'wyf'
    __version__ = '0.1'
    
    import pyhs2
    import sys
    
    default_encoding = 'utf-8'
    if sys.getdefaultencoding() != default_encoding:
        reload(sys)
        sys.setdefaultencoding(default_encoding)
    
    class HiveClient:
        def __init__(self, db_host, user, password, database, port=10000, authMechanism="PLAIN"):
            """
            create connection to hive server2
            """
            self.conn = pyhs2.connect(host=db_host,
                                      port=port,
                                      authMechanism=authMechanism,
                                      user=user,
                                      password=password,
                                      database=database,
                                      )
    
        def query(self, sql):
    
            """
            query
            """
            with self.conn.cursor() as cursor:
                cursor.execute(sql)
                return cursor.fetch()
    
        def close(self):
            """
            close connection
            """
            self.conn.close()
    
    
    def main():
        """
        main process
        """
        try:
            hive_client = HiveClient(db_host='192.168.1.13', port=10000, user='hive', password='hive',
    
                                 database='default', authMechanism='PLAIN')
    
            sql = 'select * from record limit 10'#实例sql语句
            result = hive_client.query(sql)
            hive_client.close()
        except pyhs2.error, tx:
            print '%s' % (tx.message)
            sys.exit(1)
        writeXlwt('test.xls',result)
    
    if __name__ == '__main__':  
        main()

![python操作hive结果][python_hive]

转载于:https://my.oschina.net/u/1396185/blog/1510505

[2019_Python_]: https://my.oschina.net/u/2663968/blog/3061697
[hot3.png]: /images/20210726/7327671c23ce4eaaa6e3486df8fd7ba4.png
[https_pip.pypa.io_en_stable_installing]: https://pip.pypa.io/en/stable/installing/
[python_hive]: https://static.oschina.net/uploads/img/201708/15160131_TIWy.png