Python 3.10 / structure fixes / tox introduction

Too large changeset to be honest (for one commit), but breaking this into smaller ones is a pain, as the changes depend on each other. Anyways: - Proper restructuring so that you can just pip install . - Add `tox` with tests running in 3.10, 3.11. and 3.12. - Updated pandas and numpy to latest versions. - Change CI to also use pip install . and tox for testing - README.md update. Python 3.8 and 3.9 have now been dropped. If you need them, use an older version of this tool.
PelionIoT · Feb 10, 2025 · 2aa6247 · 2aa6247
1 parent fc242c4
commit 2aa6247
Show file tree

Hide file tree

Showing 13 changed files with 238 additions and 119 deletions.
diff --git a/.github/workflows/pr-checker.yml b/.github/workflows/pr-checker.yml
@@ -4,13 +4,22 @@ on:
   push:
   workflow_dispatch:
 jobs:
-  run-pytest-pylint:
+  run-tox:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - run: sudo apt-get update && sudo apt-get install -y build-essential python3-pandas
-      - run: pip install -r requirements.txt
-      - run: pip install -r dev-requirements.txt
-      - run: pip install --editable .
-      - run: pytest -v -o junit_family=xunit1 --cov=. --cov-report xml:coverage.xml --cov-report html:test-results/cov_html --junitxml=xunit-reports/xunit.xml
-      - run: pylint licensetool.py tests/*.py
+      - uses: actions/setup-python@v5 
+        with:
+         python-version: '3.10'
+      - uses: actions/setup-python@v5 
+        with:
+          python-version: '3.11'
+      - uses: actions/setup-python@v5 
+        with:
+          python-version: '3.12'
+      - run: python -m pip install --upgrade pip
+      - name: Install tox
+        run: | 
+          pipx install tox
+      - run: ./dev-init.sh
+      - run: tox
diff --git a/README.md b/README.md
@@ -11,11 +11,11 @@ LicenseTool is a Python program that has two primary functions:
 
 ## Installation
 
-You need to have Python version 3.6 (or newer) installed in your system. This has only been tested in Linux, we do not plan to test or support Windows or Mac platforms in any way or manner as Yocto builds are done on Linux anyway.
+You need to have Python version 3.10 (or newer) installed in your system. This has only been tested in Linux, we do not plan to test or support Windows or Mac platforms in any way or manner as Yocto builds are done on Linux anyway.
 
 We highly recommend using a [Python virtual environment](https://docs.python.org/3/tutorial/venv.html).
 
-Python 3.10 will give you unfortunately a lot of warnings. They just love deprecating APIs in the Python world, don't they?
+If you need to use older versions of Python (3.6 - 3.9), use `v1.0` of this tool.
 
 ## User installation
 
@@ -24,7 +24,7 @@ git clone [email protected]:PelionIoT/licensetool.git
 cd licensetool
 python3 -m venv venv
 source venv/bin/activate
-pip install -r requirements.txt
+pip install .
 ```
 
 ## Developer installation
@@ -42,7 +42,7 @@ The license file is generated as part of the Yocto build. In Foundries.io LmP-bu
 
 ### Generate CSV-formatted license file
 
-`python licensetool.py list <input manifest file> <output>`
+`licensetool list <input manifest file> <output>`
 
 This will generate two output files,
 
@@ -51,7 +51,7 @@ This will generate two output files,
 
 ### Generate license changes file
 
-`python licensetool.py changes <previous manifest file> <current manifest file> <output file>`
+`licensetool changes <previous manifest file> <current manifest file> <output file>`
 
 This will generate two output files,
 
@@ -60,22 +60,26 @@ This will generate two output files,
 
 ### Other options
 
-Run the tool with `python licensetool.py` to get information on optional parameters.
+Run the tool with `licensetool` to get information on optional parameters.
 
 ## Tests
 
 Tests and test material are located in [tests](tests)-folder.
 You can run them from the root folder:
 
+```bash
+tox
 ```
-pytest -v -o junit_family=xunit1 --cov=. --cov-report xml:coverage.xml --cov-report html:test-results/cov_html --junitxml=xunit-reports/xunit.xml
+or just directly for just one Python-version.
+```bash
+pytest -v
 ```
 
 ## Contributions
 
 All contributions must pass:
 - Clear written statement that author agrees to Apache 2.0 license and is the original author of the changes.
 - Code review, so submit a pull request (PR).
-- Run `pylint licensetool.py tests/*.py` and make sure the score does not get worse (10/10 now).
+- Run `tox` and make sure the score does not get worse (10/10 now).
 - Include necessary test case updates, so that coverage does not decrease - provide evidence in the PR.
 - Include required documentation updates.
diff --git a/__init__.py b/__init__.py
@@ -1,5 +1,6 @@
 # ----------------------------------------------------------------------------
 # Copyright 2021 Pelion
+# Copyright 2025 Izuma Networks
 #
 # SPDX-License-Identifier: Apache-2.0
 #
@@ -16,4 +17,4 @@
 # limitations under the License.
 # ----------------------------------------------------------------------------
 
-__version__ = "0.0.1"
+__version__ = "1.0.0"
diff --git a/dev-init.sh b/dev-init.sh
@@ -1,7 +1,7 @@
 #!/bin/bash -ex
 # ----------------------------------------------------------------------------
 # Copyright 2021 Pelion
-# Copyright 2021 Izuma Networks
+# Copyright 2022-2025 Izuma Networks
 #
 # SPDX-License-Identifier: Apache-2.0
 #
@@ -19,6 +19,11 @@
 # ----------------------------------------------------------------------------
 set -ex
 
+# Tox needs pipx
+if ! command -v pipx &> /dev/null; then
+    pip install pipx
+fi
+
 # Create virtual environment if one is not in place
 if [[ ! -d venv ]]; then
     python3 -m venv venv
@@ -30,7 +35,9 @@ source venv/bin/activate
 if [[ $(pip show licensetool) ]]; then
     pip uninstall licensetool --yes
 fi
-pip install --editable .
 
-pip install -r requirements.txt
-pip install -r dev-requirements.txt
+if ! command -v tox &> /dev/null; then
+    pipx install tox
+fi
+pip install --editable ".[dev]"
+
diff --git a/dev-requirements.txt b/dev-requirements.txt
@@ -1,9 +1 @@
-pylint==2.11.0
-pytest
-pytest-mock
-pytest-cov
-tox
-pycodestyle
-coverage
-wheel
-bandit
+-e .[dev]
diff --git a/licensetool/licensetool/__init__.py b/licensetool/licensetool/__init__.py
@@ -0,0 +1,46 @@
+#!/usr/bin/env python3
+#
+# Copyright (c) 2024 Izuma Networks
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+A tool for dealing with Yocto license manifest files.
+
+This tool can converting them to CSV/Excel-format,
+optionally with change information. 
+
+This file has the (global) imports required by the tests.
+Actual values defiend in main.py
+"""
+
+
+
+from .main import (
+    read_manifest_file,
+    gen_list,
+    gen_changes,
+    _DATA_SHEET_NAME,
+    main,
+)
+
+
+__all__ = [
+    'read_manifest_file',
+    'gen_list',
+    'gen_changes',
+    '_DATA_SHEET_NAME',
+    'main',
+]
diff --git a/licensetool.py → licensetool/licensetool/main.py b/licensetool.py → licensetool/licensetool/main.py
@@ -32,9 +32,17 @@
 import argparse
 import re
 import pandas as pd
-from openpyxl.utils import get_column_letter
 from openpyxl import load_workbook
 
+# Public functions/constants so that tests find
+__all__ = [
+    'read_manifest_file',
+    '_DATA_SHEET_NAME',
+    'gen_list',
+    'gen_changes',
+    # Add other public functions/constants here
+]
+
 # Lots of literals
 _CSV = ".csv"
 _XLS = ".xlsx"
@@ -130,9 +138,8 @@ def read_manifest_file(input_file):
             ]], columns=column_names)
             d_f = pd.concat([d_f, new_row], ignore_index=True)
 
-        if (
-            package_count == 0
-        ):  # needs to have at least one package or it is an error
+        # Needs to have at least one package or it is an error
+        if package_count == 0:
             print("Package count is zero")
             errors = True
 
@@ -324,8 +331,9 @@ def gen_changes(previous, current, output):
             package_change = True
             change_summary[_PKG_ADD] += 1
         # Package removed
-        if package_change is False and pd.isna(
-            d_f_combo.at[i, _CURR_REC]
+        if (
+                package_change is False and
+                pd.isna(d_f_combo.at[i, _CURR_REC])
         ):  # NaN
             d_f_combo.at[i, _CHG] = _MARK_CHG
             d_f_combo.at[i, _PKG_REM] = _MARK_CHG
@@ -361,8 +369,8 @@ def gen_changes(previous, current, output):
             change_summary[_PKG_REM] += 1
         # Version change
         if (
-            package_change is False
-            and d_f_combo.at[i, _PREV_VER] != d_f_combo.at[i, _CURR_VER]
+                package_change is False
+                and d_f_combo.at[i, _PREV_VER] != d_f_combo.at[i, _CURR_VER]
         ):
             d_f_combo.at[i, _CHG] = _MARK_CHG
             d_f_combo.at[i, _VER_CHG] = _MARK_CHG
@@ -383,8 +391,8 @@ def gen_changes(previous, current, output):
             change_summary[_VER_CHG] += 1
         # License change
         if (
-            package_change is False
-            and d_f_combo.at[i, _PREV_LIC] != d_f_combo.at[i, _CURR_LIC]
+                package_change is False
+                and d_f_combo.at[i, _PREV_LIC] != d_f_combo.at[i, _CURR_LIC]
         ):
             d_f_combo.at[i, _CHG] = _MARK_CHG
             d_f_combo.at[i, _LIC_CHG] = _MARK_CHG
@@ -423,43 +431,46 @@ def gen_changes(previous, current, output):
 #                  styled = styled Pandas dataframe
 #
 def generate_excel(output, styled, template_file=None):
-    """Generate Excel-file (output) from styled Panda's dataframe."""
-    # Add autofilters to Excel sheet
-    # pylint: disable=abstract-class-instantiated
-    writer = pd.ExcelWriter(output, engine="openpyxl")
-
-    if template_file:
+    """Generate Excel file from dataframe."""
+    if template_file and os.path.exists(template_file):
+        # Load template if it exists
         template_book = load_workbook(template_file)
-        writer.book = template_book
-
-    styled.to_excel(writer, sheet_name=_DATA_SHEET_NAME, index=False)
-
-    # Get the xlsxwriter workbook and worksheet objects.
-    # pylint: disable=E1101
-    workbook = writer.book
-    worksheet = workbook[_DATA_SHEET_NAME]
 
-    # put the datasheet first, _sheets is protected
-    # pylint: disable=W0212
-    oldindex = workbook._sheets.index(worksheet)
-    # pylint: disable=W0212
-    workbook._sheets.pop(oldindex)
-    # pylint: disable=W0212
-    workbook._sheets.insert(0, worksheet)
-
-    worksheet.auto_filter.ref = worksheet.dimensions
-
-    colum_names = []
-    for cell in worksheet[1]:
-        colum_names.append(str(cell.value))
-
-    # set default width of colums to match the title
-    for col in range(worksheet.min_column, worksheet.max_column + 1):
-        worksheet.column_dimensions[get_column_letter(col)].width = (
-            len(colum_names[col - 1]) + 5
+        # Write the main data sheet using pandas to_excel directly
+        styled.to_excel(
+            output,
+            sheet_name=_DATA_SHEET_NAME,
+            engine='openpyxl',
+            index=False
         )
 
-    writer.save()
+        # Load the written file to add template sheets
+        workbook = load_workbook(output)
+
+        # Copy sheets from template
+        for sheet_name in template_book.sheetnames:
+            if sheet_name != _DATA_SHEET_NAME:
+                # Copy sheet from template
+                template_sheet = template_book[sheet_name]
+                if sheet_name not in workbook.sheetnames:
+                    workbook.create_sheet(sheet_name)
+                dest_sheet = workbook[sheet_name]
+
+                # Copy content
+                for row in template_sheet.rows:
+                    for cell in row:
+                        dest_sheet[cell.coordinate] = cell.value
+                dest_sheet.sheet_state = 'visible'
+        # Save the workbook
+        workbook.save(output)
+    else:
+        # Just write the dataframe if no template
+        styled.to_excel(
+            output,
+            sheet_name=_DATA_SHEET_NAME,
+            engine='openpyxl',
+            index=False
+        )
 
 
 #
@@ -550,8 +561,9 @@ def parse_list(args):
     if not os.path.isfile(args.inputfile):
         print("ERROR - input file: '" + args.inputfile + "' does not exist.")
         sys.exit(2)  # ENOENT
-    if os.path.isfile(args.listfile + _CSV) or os.path.isfile(
-        args.listfile + _XLS
+    if (
+            os.path.isfile(args.listfile + _CSV) or
+            os.path.isfile(args.listfile + _XLS)
     ):
         if not args.force:
             print("ERROR - output file: '" + args.listfile + _EXISTS)
@@ -576,8 +588,9 @@ def parse_changes(args):
     if not os.path.isfile(args.current):
         print("ERROR - current license file: '" + args.current + _NOT_EXIST)
         sys.exit(2)  # ENOENT
-    if os.path.isfile(args.changefile + _CSV) or os.path.isfile(
-        args.changefile + _XLS
+    if (
+            os.path.isfile(args.changefile + _CSV) or
+            os.path.isfile(args.changefile + _XLS)
     ):
         if not args.force:
             print("ERROR - output file: '" + args.changefile + _EXISTS)

diff --git a/requirements.txt b/requirements.txt
@@ -1,3 +1 @@
-pandas==1.5.3
-jinja2
-openpyxl
+-e .