Merging release 1.0.0a22 (#164)

* Use SqlToolsService built on .NET Core 2.0 and a build script updates (#131) * Bump version to 1.0.0a19 * Use .NET Core 2.0 RTM built sqltoolsservice * Add build script to upload to azure blob storage * Upgrade to VS 2017 * Remove 3.3 as supported Python version * Fix perf issue where main event loop takes 100% of CPU (#132) Fix perf issue where main event loop takes 100% of CPU We have a 2 threads: Thread #1 runs in a loop polling the response queue Thread #2 runs in a loop decoding responses from the sqltoolsservice over stdout and posting them to the response queue Since thread #1 doesn't sleep, it's takes 100% CPU. In addition, running python 2.7 on windows, #2 doesn’t preempt the CPU due to #1 taking all of the CPU cycles, so no response is processed. Fix is simple – thread #1 needs to sleep so thread #2 can get scheduled and get it’s work done. * Refine event loop perf fix in main.py Refine event loop perf fix in main.py * Fixing regular expression Previous regex would result in release:a1 and release_version: 12. Modified the regex for part Release to only pick up lower case letters. * Adding missing forward slash on test pypi url * fixing typos/grammar (#138) fixing typos/grammar. * Updating to release version 1.0.0a20. * Create doc for official msft docs page * Updated documentation page with usage_guide * Added link to download adventureworks * Updated with sqlcmd usage * Added in run and cloud shell support * Update/consolidate linux install (#153) * universal linux wheel gen and setup update. * Updating version cfg. * Updating sqltoolsservice container. * Updating spacing for flake8. * Updating team email. (#154) * Fixing resource warning for sqltoolsservice (#158) * Fixing resource warning for sqltoolsservice by closing stdout after killing process. * Shortening sleep time during shutdown. * Fixing missing bracket. * Updating doc's after repro rename (#160) * Updating files after repro rename. * Fixing flake8 issues. * Ron/platform wheels support (#161) * Merge release 1.0.0a21 (#155) * Use SqlToolsService built on .NET Core 2.0 and a build script updates (#131) * Bump version to 1.0.0a19 * Use .NET Core 2.0 RTM built sqltoolsservice * Add build script to upload to azure blob storage * Upgrade to VS 2017 * Remove 3.3 as supported Python version * Fix perf issue where main event loop takes 100% of CPU (#132) Fix perf issue where main event loop takes 100% of CPU We have a 2 threads: Thread #1 runs in a loop polling the response queue Thread #2 runs in a loop decoding responses from the sqltoolsservice over stdout and posting them to the response queue Since thread #1 doesn't sleep, it's takes 100% CPU. In addition, running python 2.7 on windows, #2 doesn’t preempt the CPU due to #1 taking all of the CPU cycles, so no response is processed. Fix is simple – thread #1 needs to sleep so thread #2 can get scheduled and get it’s work done. * Refine event loop perf fix in main.py Refine event loop perf fix in main.py * Fixing regular expression Previous regex would result in release:a1 and release_version: 12. Modified the regex for part Release to only pick up lower case letters. * Adding missing forward slash on test pypi url * fixing typos/grammar (#138) fixing typos/grammar. * Updating to release version 1.0.0a20. * Create doc for official msft docs page * Updated documentation page with usage_guide * Added link to download adventureworks * Updated with sqlcmd usage * Added in run and cloud shell support * Update/consolidate linux install (#153) * universal linux wheel gen and setup update. * Updating version cfg. * Updating sqltoolsservice container. * Updating spacing for flake8. * Updating team email. (#154) * Updating mssqltoolsservice to be integrated as a package of mssqlscripter. * Updating sqltoolsservice to be loaded from the repro instead of storage account. * Fix index file generation for daily storage account. * Fixing manylinux1 tag. * Updating platform tag for win x64. * Renaming sqltoolsservice win x64 folder. * Adding platform tags for win_amd64, manylinux1_x86_64, manylinux1_i686. * version bumping to 1.0.0a22. * Flake8 format fixes. * Erroring out when build receives invalid flag. * Fixing tag for win64 * Update libunwind8 install for CentOS * Ron/sqltoolsservice update (#163) * Updating sqltoolsservice with self contained version. * Refreshing sqltoolsservice again. * Making mssql-scripter executable and adding null checks in main.py * Adding clean up step to remove build directory after each build. * Fixing path for build directory. * Removing 'pypi' from upload step. * Flake 8 extra line fix.
microsoft · Nov 17, 2017 · be55629 · be55629
1 parent 606053a
commit be55629
Show file tree

Hide file tree

Showing 33 changed files with 295 additions and 541 deletions.
diff --git a/.bumpversion.cfg b/.bumpversion.cfg
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 1.0.0a21
+current_version = 1.0.0a22
 
 parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)((?P<release>[a-z]+))(?P<release_version>\d+)
 serialize = 
@@ -14,9 +14,5 @@ values =
 
 [bumpversion:file:setup.py]
 
-[bumpversion:file:mssqltoolsservice/setup.py]
-
 [bumpversion:file:mssqlscripter/__init__.py]
 
-[bumpversion:file:mssqltoolsservice/mssqltoolsservice/__init__.py]
-
diff --git a/.gitignore b/.gitignore
@@ -93,7 +93,7 @@ ENV/
 
 # sqltoolsservice binaries
 /mssqlscripter/sqltoolsservice/*
-/mssqltoolsservice/mssqltoolsservice/bin/*
+/mssqlscripter/mssqltoolsservice/bin/*
 
 
 # VSCode configuration

diff --git a/MANIFEST.in b/MANIFEST.in
@@ -1,2 +1,4 @@
 include LICENSE.txt
 include README.rst
+include mssqlscripter/mssqltoolsservice/bin/*
+include mssqlscripter/mssqltoolsservice/bin/*/*
diff --git a/README.rst b/README.rst
@@ -1,17 +1,17 @@
-.. image:: https://travis-ci.org/Microsoft/sql-xplat-cli.svg?branch=dev
-    :target: https://travis-ci.org/Microsoft/sql-xplat-cli
+.. image:: https://travis-ci.org/Microsoft/mssql-scripter.svg?branch=dev
+    :target: https://travis-ci.org/Microsoft/mssql-scripter
 
 .. image:: https://ci.appveyor.com/api/projects/status/vpm8p71265ijytqs/branch/dev?svg=true
-    :target: https://ci.appveyor.com/project/MrMeemus/sql-xplat-cli
+    :target: https://ci.appveyor.com/project/MrMeemus/mssql-scripter
 
-.. image:: https://codecov.io/gh/Microsoft/sql-xplat-cli/branch/dev/graph/badge.svg?token=M96uWrHOIu
-    :target: https://codecov.io/gh/Microsoft/sql-xplat-cli/branch/dev
+.. image:: https://codecov.io/gh/Microsoft/mssql-scripter/branch/dev/graph/badge.svg?token=M96uWrHOIu
+    :target: https://codecov.io/gh/Microsoft/mssql-scripter/branch/dev
 
 .. image:: https://badge.fury.io/py/mssql-scripter.svg
     :target: https://pypi.python.org/pypi/mssql-scripter
 
 .. image:: https://img.shields.io/pypi/pyversions/mssql-scripter.svg
-    :target: https://travis-ci.org/Microsoft/sql-xplat-cli
+    :target: https://travis-ci.org/Microsoft/mssql-scripter
 
 mssql-scripter
 ===============
@@ -68,7 +68,7 @@ mssql-scritper is licensed under the `MIT license`_.
 .. _installation guide: doc/installation_guide.md
 .. _development guide: doc/development_guide.md
 .. _usage guide: doc/usage_guide.md
-.. _Issues: https://github.com/Microsoft/sql-xplat-cli/issues
+.. _Issues: https://github.com/Microsoft/mssql-scripter/issues
 .. _Microsoft Open Source Code of Conduct: https://opensource.microsoft.com/codeofconduct/
 .. _Code of Conduct FAQ: https://opensource.microsoft.com/codeofconduct/faq/
-.. _MIT license: https://github.com/Microsoft/sql-xplat-cli/blob/dev/LI
+.. _MIT license: https://github.com/Microsoft/mssql-scripter/blob/dev/LI
diff --git a/build.py b/build.py
@@ -6,23 +6,55 @@
 # --------------------------------------------------------------------------------------------
 
 from __future__ import print_function
+from azure.storage.blob import BlockBlobService, ContentSettings
 import os
-import re
 import sys
-import tempfile
 import utility
-from azure.storage.blob import BlockBlobService, ContentSettings
+import mssqlscripter.mssqltoolsservice.external as mssqltoolsservice
 
 AZURE_STORAGE_CONNECTION_STRING = os.environ.get('AZURE_STORAGE_CONNECTION_STRING')
 BLOB_CONTAINER_NAME = 'simple'
-UPLOADED_PACKAGE_LINKS = [] 
+UPLOADED_PACKAGE_LINKS = []
 
 
 def print_heading(heading, f=None):
     print('{0}\n{1}\n{0}'.format('=' * len(heading), heading), file=f)
 
 
-def upload_index_file(service, blob_name, title, links):
+def build(platform_names):
+    """
+        Builds mssql-scripter package.
+    """
+    print_heading('Cleanup')
+
+    # clean
+    utility.clean_up(utility.MSSQLSCRIPTER_DIST_DIRECTORY)
+
+    print_heading('Running setup')
+
+    # install general requirements.
+    utility.exec_command('pip install -r dev_requirements.txt', utility.ROOT_DIR)
+
+    # convert windows line endings to unix for mssql-cli bash script
+    utility.exec_command('python dos2unix.py mssql-scripter mssql-scripter', utility.ROOT_DIR)
+
+    for platform in platform_names:
+        utility.clean_up(utility.MSSQLSCRIPTER_BUILD_DIRECTORY)
+        utility.cleaun_up_egg_info_sub_directories(utility.ROOT_DIR)
+
+        mssqltoolsservice.copy_sqltoolsservice(platform)
+
+        print_heading('Building mssql-scripter {} wheel package package'.format(platform))
+        utility.exec_command('python --version', utility.ROOT_DIR)
+        utility.exec_command(
+            'python setup.py check -r -s bdist_wheel --plat-name {}'.format(platform),
+            utility.ROOT_DIR,
+            continue_on_error=False)
+
+        mssqltoolsservice.clean_up_sqltoolsservice()
+
+
+def _upload_index_file(service, blob_name, title, links):
     print('Uploading index file {}'.format(blob_name))
     service.create_blob_from_text(
         container_name=BLOB_CONTAINER_NAME,
@@ -37,23 +69,23 @@ def upload_index_file(service, blob_name, title, links):
             content_language=None,
             content_md5=None,
             cache_control=None
-        )
+            )
     )
 
 
-def gen_pkg_index_html(service, pkg_name):
+def _gen_pkg_index_html(service, pkg_name):
     links = []
     index_file_name = pkg_name+'/'
     for blob in list(service.list_blobs(BLOB_CONTAINER_NAME, prefix=index_file_name)):
         if blob.name == index_file_name:
             # Exclude the index file from being added to the list
             continue
         links.append(blob.name.replace(index_file_name, ''))
-    upload_index_file(service, index_file_name, 'Links for {}'.format(pkg_name), links)
+    _upload_index_file(service, index_file_name, 'Links for {}'.format(pkg_name), links)
     UPLOADED_PACKAGE_LINKS.append(index_file_name)
 
 
-def upload_package(service, file_path, pkg_name):
+def _upload_package(service, file_path, pkg_name):
     print('Uploading {}'.format(file_path))
     file_name = os.path.basename(file_path)
     blob_name = '{}/{}'.format(pkg_name, file_name)
@@ -62,63 +94,84 @@ def upload_package(service, file_path, pkg_name):
         blob_name=blob_name,
         file_path=file_path
     )
-    gen_pkg_index_html(service, pkg_name)
-
 
-def build(options):
 
-    supported_actions = ['nightly']
-    action = None
+def validate_package(platform_names):
+    """
+        Install mssql-scripter wheel package locally.
+    """
+    root_dir = os.path.abspath(os.path.join(os.path.abspath(__file__), '..'))
+    # Local install of mssql-scripter.
+    mssqlscripter_wheel_dir = os.listdir(utility.MSSQLSCRIPTER_DIST_DIRECTORY)
+    current_platform = utility.get_current_platform()
 
-    if len(options) >= 1:
-        if options[0] not in supported_actions:
-            print('Please provide a supported action {}.'.format(supported_actions))
-            return
-        action = options[0]
+    mssqlscripter_wheel_name = [pkge for pkge in mssqlscripter_wheel_dir if current_platform in pkge]
 
-    if action == 'nightly':
-        assert AZURE_STORAGE_CONNECTION_STRING, 'Set AZURE_STORAGE_CONNECTION_STRING environment variable'
-
-    print_heading('Cleanup')
-
-    # clean
-    utility.clean_up(utility.MSSQLSCRIPTER_DIST_DIRECTORY)
-    utility.clean_up(utility.MSSQLTOOLSSERVICE_DIST_DIRECTORY)
-    utility.cleaun_up_egg_info_sub_directories(utility.ROOT_DIR)
-    utility.cleaun_up_egg_info_sub_directories(utility.MSSQLTOOLSSERVICE_DIRECTORY)
+    # To ensure we have a clean install, we disable the cache as to prevent cache overshadowing actual changes made.
+    utility.exec_command(
+       'pip install --no-cache-dir --no-index ./dist/{}'.format(mssqlscripter_wheel_name),
+       root_dir, continue_on_error=False)
 
-    print_heading('Running setup')
-
-    # install general requirements.
-    utility.exec_command('pip install -r dev_requirements.txt', utility.ROOT_DIR)
-
-    print_heading('Running mssql-scripter tests')
-    utility.exec_command('tox', utility.ROOT_DIR, continue_on_error = False)
-
-    print_heading('Building mssql-scripter pip package')
-    utility.exec_command('python setup.py check -r -s sdist', utility.ROOT_DIR, continue_on_error = False)
-
-    print_heading('Building mssqltoolsservice pip package')
-    utility.exec_command('python buildwheels.py', utility.MSSQLTOOLSSERVICE_DIRECTORY, continue_on_error = False)
-
-    if action == 'nightly':
-        blob_service = BlockBlobService(connection_string=AZURE_STORAGE_CONNECTION_STRING)
-
-        print_heading('Uploading packages to blob storage ')
-        for pkg in os.listdir(utility.MSSQLSCRIPTER_DIST_DIRECTORY):
-            pkg_path = os.path.join(utility.MSSQLSCRIPTER_DIST_DIRECTORY, pkg)
-            print('Uploading package {}'.format(pkg_path))
-            upload_package(blob_service, pkg_path, 'mssql-scripter')
-
-        for pkg in os.listdir(utility.MSSQLTOOLSSERVICE_DIST_DIRECTORY):
-            pkg_path = os.path.join(utility.MSSQLTOOLSSERVICE_DIST_DIRECTORY, pkg)
-            pkg_name = os.path.basename(pkg_path).split('-')[0].replace('_', '-').lower()
-            print('Uploading package {}'.format(pkg_name))
-            upload_package(blob_service, pkg_path, pkg_name)
 
-        # Upload the final index file
-        upload_index_file(blob_service, 'index.html', 'Simple Index', UPLOADED_PACKAGE_LINKS)
+def publish_daily(platforms_names):
+    """
+    Publish mssql-scripter wheel package to daily storage account.
+    """
+    print('Publishing to simple container within storage account.')
+    assert AZURE_STORAGE_CONNECTION_STRING, 'Set AZURE_STORAGE_CONNECTION_STRING environment variable'
+
+    blob_service = BlockBlobService(connection_string=AZURE_STORAGE_CONNECTION_STRING)
+
+    print_heading('Uploading packages to blob storage ')
+    for pkg in os.listdir(utility.MSSQLSCRIPTER_DIST_DIRECTORY):
+        pkg_path = os.path.join(utility.MSSQLSCRIPTER_DIST_DIRECTORY, pkg)
+        print('Uploading package {}'.format(pkg_path))
+        _upload_package(blob_service, pkg_path, 'mssql-scripter')
+
+    # Upload index files
+    _gen_pkg_index_html(blob_service, 'mssql-scripter')
+    _upload_index_file(blob_service, 'index.html', 'Simple Index', UPLOADED_PACKAGE_LINKS)
+
+
+def publish_official(platforms_names):
+    """
+    Publish mssql-scripter wheel package to PyPi.
+    """
+    mssqlscripter_wheel_dir = os.listdir(utility.MSSQLSCRIPTER_DIST_DIRECTORY)
+    # Run twine action for mssqlscripter.
+    # Only authorized users with credentials will be able to upload this package.
+    # Credentials will be stored in a .pypirc file.
+    for mssqlscripter_wheel_name in mssqlscripter_wheel_dir:
+        utility.exec_command(
+            'twine upload {}'.format(mssqlscripter_wheel_name),
+            utility.MSSQLSCRIPTER_DIST_DIRECTORY)
 
 
 if __name__ == '__main__':
-    build(sys.argv[1:])
+    action = 'build'
+    supported_platforms = [
+        'win32',
+        'win_amd64',
+        'win64',
+        'macosx_10_11_intel',
+        'manylinux1_x86_64',
+        'manylinux1_i686']
+
+    targets = {
+        'build': build,
+        'validate_package': validate_package,
+        'publish_daily': publish_daily,
+        'publish_official': publish_official
+    }
+
+    if len(sys.argv) > 1:
+        action = sys.argv[1]
+
+    if len(sys.argv) > 2:
+        supported_platforms = [sys.argv[2]]
+
+    if action in targets:
+        targets[action](supported_platforms)
+    else:
+        print('{} is not a supported action'.format(action))
+        print('Supported actions are {}'.format(list(targets.keys())))
diff --git a/dev_setup.py b/dev_setup.py
@@ -9,19 +9,21 @@
 from __future__ import print_function
 
 import os
-import setup
+import platform
 import utility
+import mssqlscripter.mssqltoolsservice.external as mssqltoolsservice
 
 print('Running dev setup...')
 print('Root directory \'{}\'\n'.format(utility.ROOT_DIR))
 
 # install general requirements.
 utility.exec_command('pip install -r dev_requirements.txt', utility.ROOT_DIR)
+run_time_id = utility.get_current_platform()
+
+if run_time_id:
+    mssqltoolsservice.copy_sqltoolsservice(run_time_id)
+else:
+    print("This platform does not support mssqltoolsservice.")
 
-# install mssqltoolsservice if this platform supports it.
-mssqltoolsservice_package_name = os.environ['MSSQLTOOLSSERVICE_PACKAGE_NAME']
-print('Installing {}...'.format(mssqltoolsservice_package_name))
-# mssqltoolsservice package name is retrieved from environment variable set by setup.py.
-utility.exec_command('pip install {}'.format(mssqltoolsservice_package_name), utility.ROOT_DIR)
 
 print('Finished dev setup.')
diff --git a/doc/development_guide.md b/doc/development_guide.md
@@ -11,18 +11,18 @@ Development Guide
 
 See the [Architecture Guide](architecture_guide.md) for details on how mssql-scripter is designed and implemented.
 
-mssql-scripter sources are located on GitHub (https://github.com/Microsoft/sql-xplat-cli). In order to contribute to the project, you are expected to: 
+mssql-scripter sources are located on GitHub (https://github.com/Microsoft/mssql-scripter). In order to contribute to the project, you are expected to: 
 -	Have a GitHub account. For Microsoft contributors, follow the guidelines on https://opensourcehub.microsoft.com/ to create, configure and link your account
--	Fork the  https://github.com/Microsoft/sql-xplat-clirepository into your private GitHub account
--	Create pull requests against the https://github.com/Microsoft/sql-xplat-cli repository to get your code changes merged into the project repository.
+-	Fork the  https://github.com/Microsoft/mssql-scripterrepository into your private GitHub account
+-	Create pull requests against the https://github.com/Microsoft/mssql-scripter repository to get your code changes merged into the project repository.
 
 ## <a name="Preparing_Machine"></a>1. Preparing your machine
 1.	Install latest Python from http://python.org. Please note that the version of Python that comes preinstalled on OSX is 2.7. It is recommended to install both Python 2.7 and Python3.6 to ensure backwards compatibility for testing.
     #### Windows
     - The latest Python installation package can be downloaded from [here](https://www.python.org/downloads/).  
     - During installation, select the 'Add Python to PATH' option.  Python must be in the PATH environment variable.
 
-2. Clone the repo from [https://github.com/Microsoft/sql-xplat-cli](https://github.com/Microsoft/sql-xplat-cli)
+2. Clone the repo from [https://github.com/Microsoft/mssql-scripter](https://github.com/Microsoft/mssql-scripter)
 
 ## <a name="Environment_Setup"></a>2. Environment Setup
 When developing on a Python project, it is recommended to do so in a virtual environment. A virtual environment is a sandbox that maintains a copy of all libraries necessary to run python in a isolated environment without interfering with the system or global python. For more information on virtual environments, go to [Virtual Environment Info](docs/virtual_environment_info.md).
@@ -78,7 +78,7 @@ General development steps that apply to both a virtual environment or a global e
 ## <a name="Configure_IDE"></a>3. Configuring your IDE
 #### Visual Studio (Windows only)
 1.	Install [Python Tools for Visual Studio](https://github.com/Microsoft/PTVS)
-2.	Open the `<clone_root>\sql-xplat-cli.pyproj` project in Visual Studio
+2.	Open the `<clone_root>\mssql-scripter.pyproj` project in Visual Studio
 
 
 #### Visual Studio Code (Any platform)

diff --git a/doc/installation_guide.md b/doc/installation_guide.md
@@ -90,7 +90,7 @@ C:\> pip install mssql-scripter
 
 # Troubleshooting
 
-If you're having installation issues, please check the below known issues and workarounds.  If you're having a different issue, please check the [issues](https://github.com/Microsoft/sql-xplat-cli/issues) page to see if the issue has already been reported.  If you don't see your issue there, filing a new issue would be appreciated.
+If you're having installation issues, please check the below known issues and workarounds.  If you're having a different issue, please check the [issues](https://github.com/Microsoft/mssql-scripter/issues) page to see if the issue has already been reported.  If you don't see your issue there, filing a new issue would be appreciated.
 
 ## Error: No module named mssqlscripter
 If the installation was successful and this error message is encountered, this may be caused by different versions of python in the environment.
@@ -151,10 +151,19 @@ $ sudo apt-get update
 $ sudo apt-get install libunwind8
 ```
 
+### CentOS 7
+Run commands
+```shell
+## libunwind.x86_64 is in the epel-release repository
+## (Not necessary on GNOME, minimal install does not have this installed)
+$ sudo yum -y install epel-release
+$ sudo yum -y install libunwind
+```
+
 ## Installing the icu package
 
 ### RHEL 7.3
 Run commands
 ```shell
-$ sudo sudo yum install icu
+$ sudo yum install icu
 ```