feat: add JWT support to PyHive (#1)

* feat: add HTTP and HTTPS to hive (dropbox#385)

* feat: add https protocol

* support HTTP

* fix: make hive https py2 compat (dropbox#389)

* fix: make hive https py2 compat

* fix lint

* Update README.rst (dropbox#423)

* chore: rename Trino entry point (dropbox#428)

* Support for Presto decimals (dropbox#430)

* Support for Presto decimals

* lower

* Use str type for driver and name in HiveDialect (dropbox#450)

PyHive's HiveDialect usage of bytes for the name and driver fields is not the norm is causing issues upstream: apache/superset#22316
Even other dialects within PyHive use strings. SQLAlchemy does not strictly require a string, but all the stock dialects return a string, so I figure it is heavily implied.

I think the risk of breaking something upstream with this change is low (but it is there ofc). I figure in most cases we just make someone's `str(dialect.driver)` expression redundant.

Examples for some of the other stock sqlalchemy dialects (name and driver fields using str):
https://github.com/zzzeek/sqlalchemy/blob/main/lib/sqlalchemy/dialects/sqlite/pysqlite.py#L501
https://github.com/zzzeek/sqlalchemy/blob/main/lib/sqlalchemy/dialects/sqlite/base.py#L1891
https://github.com/zzzeek/sqlalchemy/blob/main/lib/sqlalchemy/dialects/mysql/base.py#L2383
https://github.com/zzzeek/sqlalchemy/blob/main/lib/sqlalchemy/dialects/mysql/mysqldb.py#L113
https://github.com/zzzeek/sqlalchemy/blob/main/lib/sqlalchemy/dialects/mysql/pymysql.py#L59

* Correcting Iterable import for python 3.10 (dropbox#451)

* changing drivers to support hive, presto and trino with sqlalchemy>=2.0 (dropbox#448)

* Revert "changing drivers to support hive, presto and trino with sqlalchemy>=2.0 (dropbox#448)" (dropbox#452)

This reverts commit b0206d3.

* Update __init__.py (dropbox#453)

dropbox@1c1da8b

dropbox@1f99552

* use pure-sasl with python 3.11 (dropbox#454)

* minimal changes for sqlalchemy 2.0 support (dropbox#457)

* update readme to reflect recent changes (dropbox#459)

* Update README.rst (dropbox#475)

* Update README.rst (dropbox#476)

* feat: JWT support

* Add CI to build package

---------

Co-authored-by: Daniel Vaz Gaspar <[email protected]>
Co-authored-by: Bogdan <[email protected]>
Co-authored-by: serenajiang <[email protected]>
Co-authored-by: Usiel Riedl <[email protected]>
Co-authored-by: Multazim Deshmukh <[email protected]>
Co-authored-by: nicholas-miles <[email protected]>

Author: 7 people

.gitignore

@@ -14,3 +14,4 @@ cover/
 .cache/
 *.iml
 /scripts/.thrift_gen
+.python-version

CHANGELOG.md

@@ -0,0 +1,4 @@
+0.7.0a
+======
+
+- Add support for JWT authentication.

Jenkinsfile

@@ -0,0 +1,94 @@
+LIB_NAME = 'PyHive'
+String currentVersion = ""
+
+
+podTemplate(
+    imagePullSecrets: ['preset-pull'],
+    nodeUsageMode: 'NORMAL',
+    containers: [
+        containerTemplate(
+            alwaysPullImage: true,
+            name: 'ci',
+            image: 'preset/ci:latest',
+            ttyEnabled: true,
+            command: 'cat',
+            resourceRequestCpu: '100m',
+            resourceLimitCpu: '200m',
+            resourceRequestMemory: '1000Mi',
+            resourceLimitMemory: '2000Mi',
+        ),
+        containerTemplate(
+            alwaysPullImage: true,
+            name: 'py-ci',
+            image: 'preset/python:3.8.9-ci',
+            ttyEnabled: true,
+            command: 'cat'
+        )
+    ]
+) {
+    node(POD_LABEL) {
+        container('py-ci') {
+            stage('Checkout') {
+                checkout scm
+            }
+
+            stage('Tests') {
+                sh(script: 'pip install -e . && pip install -r requirements-dev.txt', label: 'install dependencies')
+                parallel(
+                    check: {
+                        currentVersion = sh(
+                                script: "python setup.py --version",
+                                returnStdout: true,
+                                label: 'Get current version'
+                        ).trim()
+                        def retVal = sh(
+                                script: "curl -I -f https://pypi.devops.preset.zone/${LIB_NAME}/${LIB_NAME}-${currentVersion}.tar.gz",
+                                returnStatus: true,
+                                label: 'Check for existing tarball'
+                        )
+                        // If the thing exists, we should bail as we don't want to overwrite
+                        if (retVal == 0) {
+                            error("Version ${currentVersion} of ${LIB_NAME} already exists! Version bump required.")
+                        }
+                    }
+                )
+            }
+        }
+
+        container('py-ci') {
+            stage('Package Release') {
+                if (env.BRANCH_NAME.startsWith("PR-")) {
+                    def shortGitRev = sh(
+                            returnStdout: true,
+                            script: 'git rev-parse --short HEAD'
+                    ).trim()
+                    def pullRequestVersion = "${currentVersion}+${env.BRANCH_NAME}.${shortGitRev}"
+                    sh(script:"sed -i \'s/version = ${currentVersion}/version = ${pullRequestVersion}/g\' setup.cfg", label: 'Changing version for PR')
+                    sh(script:"echo PR version: ${pullRequestVersion}", label: 'PR Release candidate version')
+                }
+                sh(script: 'python setup.py sdist --formats=gztar', label: 'Bundling release')
+                sh(script: "mkdir -p dist/${LIB_NAME} && mv dist/*.gz dist/${LIB_NAME}", label: 'Setup release folder')
+            }
+        }
+
+        container('ci') {
+            stage('Upload Release') {
+                withCredentials([
+                    [
+                        $class           : 'AmazonWebServicesCredentialsBinding',
+                        credentialsId    : 'ci-user',
+                        accessKeyVariable: 'AWS_ACCESS_KEY_ID',
+                        secretKeyVariable: 'AWS_SECRET_ACCESS_KEY',
+                    ]
+                ]) {
+                    if ((env.BRANCH_NAME == 'master') || (env.BRANCH_NAME.startsWith("PR-"))) {
+                        sh(script: "aws s3 sync ./dist s3://preset-pypi", label: "Uploading to s3")
+                    }
+                    else {
+                        echo "Skipping upload as this isn't master..."
+                    }
+                }
+            }
+        }
+    }
+}

README.rst

@@ -1,24 +1,31 @@
-.. image:: https://travis-ci.org/dropbox/PyHive.svg?branch=master
-    :target: https://travis-ci.org/dropbox/PyHive
-.. image:: https://img.shields.io/codecov/c/github/dropbox/PyHive.svg
+========================================================
+PyHive project has been donated to  Apache Kyuubi
+========================================================
+
+You can follow it's development and report any issues you are experiencing here: https://github.com/apache/kyuubi/tree/master/python/pyhive 
+
+
+
+Legacy notes / instructions
+===========================
 
-======
 PyHive
-======
+**********
+
 
 PyHive is a collection of Python `DB-API <http://www.python.org/dev/peps/pep-0249/>`_ and
-`SQLAlchemy <http://www.sqlalchemy.org/>`_ interfaces for `Presto <http://prestodb.io/>`_ and
-`Hive <http://hive.apache.org/>`_.
+`SQLAlchemy <http://www.sqlalchemy.org/>`_ interfaces for `Presto <http://prestodb.io/>`_ ,
+`Hive <http://hive.apache.org/>`_ and `Trino <https://trino.io/>`_.
 
 Usage
-=====
+**********
 
 DB-API
 ------
 .. code-block:: python
 
     from pyhive import presto  # or import hive or import trino
-    cursor = presto.connect('localhost').cursor()
+    cursor = presto.connect('localhost').cursor()  # or use hive.connect or use trino.connect
     cursor.execute('SELECT * FROM my_awesome_data LIMIT 10')
     print cursor.fetchone()
     print cursor.fetchall()
@@ -54,7 +61,7 @@ In Python 3.7 `async` became a keyword; you can use `async_` instead:
 
 SQLAlchemy
 ----------
-First install this package to register it with SQLAlchemy (see ``setup.py``).
+First install this package to register it with SQLAlchemy, see ``entry_points`` in ``setup.py``.
 
 .. code-block:: python
 
@@ -64,12 +71,33 @@ First install this package to register it with SQLAlchemy (see ``setup.py``).
     # Presto
     engine = create_engine('presto://localhost:8080/hive/default')
     # Trino
-    engine = create_engine('trino://localhost:8080/hive/default')
+    engine = create_engine('trino+pyhive://localhost:8080/hive/default')
     # Hive
     engine = create_engine('hive://localhost:10000/default')
+
+    # SQLAlchemy < 2.0
     logs = Table('my_awesome_data', MetaData(bind=engine), autoload=True)
     print select([func.count('*')], from_obj=logs).scalar()
 
+    # Hive + HTTPS + LDAP or basic Auth
+    engine = create_engine('hive+https://username:password@localhost:10000/')
+    logs = Table('my_awesome_data', MetaData(bind=engine), autoload=True)
+    print select([func.count('*')], from_obj=logs).scalar()
+
+    # SQLAlchemy >= 2.0
+    metadata_obj = MetaData()
+    books = Table("books", metadata_obj, Column("id", Integer), Column("title", String), Column("primary_author", String))
+    metadata_obj.create_all(engine)
+    inspector = inspect(engine)
+    inspector.get_columns('books')
+
+    with engine.connect() as con:
+        data = [{ "id": 1, "title": "The Hobbit", "primary_author": "Tolkien" }, 
+                { "id": 2, "title": "The Silmarillion", "primary_author": "Tolkien" }]
+        con.execute(books.insert(), data[0])
+        result = con.execute(text("select * from books"))
+        print(result.fetchall())
+
 Note: query generation functionality is not exhaustive or fully tested, but there should be no
 problem with raw SQL.
 
@@ -89,7 +117,7 @@ Passing session configuration
                       'session_props': {'query_max_run_time': '1234m'}}
     )
     create_engine(
-        'trino://user@host:443/hive',
+        'trino+pyhive://user@host:443/hive',
         connect_args={'protocol': 'https',
                       'session_props': {'query_max_run_time': '1234m'}}
     )
@@ -104,27 +132,30 @@ Passing session configuration
     )
 
 Requirements
-============
+************
 
 Install using
 
-- ``pip install 'pyhive[hive]'`` for the Hive interface and
-- ``pip install 'pyhive[presto]'`` for the Presto interface.
+- ``pip install 'pyhive[hive]'`` or ``pip install 'pyhive[hive_pure_sasl]'`` for the Hive interface
+- ``pip install 'pyhive[presto]'`` for the Presto interface
 - ``pip install 'pyhive[trino]'`` for the Trino interface
 
+Note: ``'pyhive[hive]'`` extras uses `sasl <https://pypi.org/project/sasl/>`_ that doesn't support Python 3.11, See `github issue <https://github.com/cloudera/python-sasl/issues/30>`_.
+Hence PyHive also supports `pure-sasl <https://pypi.org/project/pure-sasl/>`_ via additional extras ``'pyhive[hive_pure_sasl]'`` which support Python 3.11.
+
 PyHive works with
 
 - Python 2.7 / Python 3
-- For Presto: Presto install
-- For Trino: Trino install
+- For Presto: `Presto installation <https://prestodb.io/docs/current/installation.html>`_
+- For Trino: `Trino installation <https://trino.io/docs/current/installation.html>`_
 - For Hive: `HiveServer2 <https://cwiki.apache.org/confluence/display/Hive/Setting+up+HiveServer2>`_ daemon
 
 Changelog
-=========
+*********
 See https://github.com/dropbox/PyHive/releases.
 
 Contributing
-============
+************
 - Please fill out the Dropbox Contributor License Agreement at https://opensource.dropbox.com/cla/ and note this in your pull request.
 - Changes must come with tests, with the exception of trivial things like fixing comments. See .travis.yml for the test environment setup.
 - Notes on project scope:
@@ -134,8 +165,28 @@ Contributing
   - We prefer having a small number of generic features over a large number of specialized, inflexible features.
     For example, the Presto code takes an arbitrary ``requests_session`` argument for customizing HTTP calls, as opposed to having a separate parameter/branch for each ``requests`` option.
 
+Tips for test environment setup
+****************************************
+You can setup test environment by following ``.travis.yaml`` in this repository. It uses `Cloudera's CDH 5 <https://docs.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_download_510.html>`_ which requires username and password for download.
+It may not be feasible for everyone to get those credentials. Hence below are alternative instructions to setup test environment.
+
+You can clone `this repository <https://github.com/big-data-europe/docker-hive/blob/master/docker-compose.yml>`_ which has Docker Compose setup for Presto and Hive.
+You can add below lines to its docker-compose.yaml to start Trino in same environment::
+ 
+    trino:
+        image: trinodb/trino:351    
+        ports:     
+            - "18080:18080"    
+        volumes:    
+            - ./trino:/etc/trino
+
+Note: ``./trino`` for docker volume defined above is `trino config from PyHive repository <https://github.com/dropbox/PyHive/tree/master/scripts/travis-conf/trino>`_
+
+Then run::
+    docker-compose up -d
+
 Testing
-=======
+*******
 .. image:: https://travis-ci.org/dropbox/PyHive.svg
     :target: https://travis-ci.org/dropbox/PyHive
 .. image:: http://codecov.io/github/dropbox/PyHive/coverage.svg?branch=master
@@ -154,7 +205,7 @@ WARNING: This drops/creates tables named ``one_row``, ``one_row_complex``, and `
 database called ``pyhive_test_database``.
 
 Updating TCLIService
-====================
+********************
 
 The TCLIService module is autogenerated using a ``TCLIService.thrift`` file. To update it, the
 ``generate.py`` file can be used: ``python generate.py <TCLIServiceURL>``. When left blank, the

dev_requirements.txt

@@ -12,6 +12,8 @@ pytest-timeout==1.2.0
 requests>=1.0.0
 requests_kerberos>=0.12.0
 sasl>=0.2.1
+pure-sasl>=0.6.2
+kerberos>=1.3.0
 thrift>=0.10.0
 #thrift_sasl>=0.1.0
 git+https://github.com/cloudera/thrift_sasl  # Using master branch in order to get Python 3 SASL patches

pyhive/__init__.py

@@ -1,3 +1,3 @@
 from __future__ import absolute_import
 from __future__ import unicode_literals
-__version__ = '0.6.3'
+__version__ = '0.7.0a'

pyhive/common.py

@@ -18,6 +18,11 @@
 from future.utils import with_metaclass
 from itertools import islice
 
+try:
+    from collections.abc import Iterable
+except ImportError:
+    from collections import Iterable
+
 
 class DBAPICursor(with_metaclass(abc.ABCMeta, object)):
     """Base class for some common DB-API logic"""
@@ -245,7 +250,7 @@ def escape_item(self, item):
             return self.escape_number(item)
         elif isinstance(item, basestring):
             return self.escape_string(item)
-        elif isinstance(item, collections.Iterable):
+        elif isinstance(item, Iterable):
             return self.escape_sequence(item)
         elif isinstance(item, datetime.datetime):
             return self.escape_datetime(item, self._DATETIME_FORMAT)