Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow reading of EDAX TSL .ang files with ten columns #416

Merged
merged 10 commits into from
Jan 31, 2023
+215 −164
Merged

Allow reading of EDAX TSL .ang files with ten columns #416

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
f1739a7
Complete coverage by removing unnecessary figure closing
hakonanes Dec 7, 2022
928b7db
Allow 10 column EDAX TSL .ang file
hakonanes Dec 7, 2022
44135b5
Update .ang file reader tests
hakonanes Jan 30, 2023
e601275
New 0.10.3 patch release entry in changelog
hakonanes Dec 7, 2022
98072d5
Bump version to 0.10.3
hakonanes Jan 30, 2023
6664e71
Mention change in IO scan_unit nm/um behavior
hakonanes Dec 7, 2022
286a68a
Rename 'unknown1' column to 'detector_signal'
hakonanes Dec 8, 2022
7faaacc
Run tests in parallel on GitHub Actions
hakonanes Jan 30, 2023
88904f7
Remove planned 0.10.3 patch release and instead go for 0.11.0
hakonanes Jan 31, 2023
c20bbf7
Ease failing Symmetry.plot() tests due to unexpected no. collections
hakonanes Jan 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 42 additions & 22 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,48 +1,64 @@
name: build

on: [push, pull_request]
on:
push:
branches:
- '*'
pull_request:
branches:
- '*'
workflow_dispatch:
workflow: '*'

env:
MPLBACKEND: agg

jobs:
code:
name: code style
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3

- uses: psf/black@stable
- uses: actions/setup-python@v2

- uses: actions/setup-python@v4
with:
python-version: 3.9

- uses: isort/isort-action@master
with:
configuration: --profile black --filter-files --force-sort-within-sections --check-only --diff

- name: Install Black with Jupyter extension
run: pip install black[jupyter]
run: |
pip install black[jupyter]

- name: Check code style of Jupyter notebooks
run: black doc/**/*.ipynb --check
run: |
black doc/**/*.ipynb --check

# Make sure all necessary files will be included in a release
manifest:
name: check manifest
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3

- uses: actions/setup-python@v2
- uses: actions/setup-python@v4

- name: Install dependencies
run: pip install manifix
run: |
pip install manifix

- name: Check MANIFEST.in file
run: python setup.py manifix
run: |
python setup.py manifix

build-with-pip:
name: ${{ matrix.os }}-py${{ matrix.python-version }}${{ matrix.LABEL }}
runs-on: ${{ matrix.os }}
timeout-minutes: 15
env:
MPLBACKEND: agg
strategy:
fail-fast: false
matrix:
Expand All @@ -55,26 +71,28 @@ jobs:
DEPENDENCIES: diffpy.structure==3 matplotlib==3.3
LABEL: -oldest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}

- name: Display Python and pip versions
run: python -V; pip -V

- name: Install depedencies and package
shell: bash
run: pip install -U -e .'[doc, tests]'
run: |
pip install -U -e .'[doc, tests]'

- name: Install oldest supported version
if: ${{ matrix.OLDEST_SUPPORTED_VERSION }}
run: pip install ${{ matrix.DEPENDENCIES }}
run: |
pip install ${{ matrix.DEPENDENCIES }}

- name: Display package versions
run: pip list
- name: Display Python, pip and package versions
run: |
python -V
pip -V
pip list

- name: Run docstring tests
if: ${{ matrix.os == 'ubuntu-latest' }}
Expand All @@ -83,11 +101,13 @@ jobs:
pytest --doctest-modules --ignore-glob=orix/tests orix/*.py

- name: Run tests
run: pytest --cov=orix --pyargs orix
run: |
pytest -n 2 --cov=orix --pyargs orix

- name: Generate line coverage
if: ${{ matrix.os == 'ubuntu-latest' }}
run: coverage report --show-missing
run: |
coverage report --show-missing

- name: Upload coverage to Coveralls
if: ${{ matrix.os == 'ubuntu-latest' }}
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ Added
Changed
-------
- Bumped minimal version of ``diffpy.structure >= 3.0.2``.
- Only ASTAR .ang files return crystal maps with ``"nm"`` as scan unit.

Deprecated
----------
Expand All @@ -37,6 +38,7 @@ Removed

Fixed
-----
- Reading of EDAX TSL .ang files with ten columns should now work.

Security
--------
Expand Down
187 changes: 106 additions & 81 deletions orix/io/plugins/ang.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,6 @@

__all__ = ["file_reader", "file_writer"]

# MTEX has this format sorted out, check out their readers when fixing
# issues and adapting to other versions of this file format in the future:
# https://github.com/mtex-toolbox/mtex/blob/develop/interfaces/loadEBSD_ang.m
# https://github.com/mtex-toolbox/mtex/blob/develop/interfaces/loadEBSD_ACOM.m

# Plugin description
format_name = "ang"
file_extensions = ["ang"]
Expand Down Expand Up @@ -117,15 +112,16 @@ def file_reader(filename: str) -> CrystalMap:
)

# Set which data points are not indexed
if vendor in ["orix", "tsl"]:
data_dict["phase_id"][np.where(data_dict["prop"]["ci"] == -1)] = -1
# TODO: Add not-indexed convention for INDEX ASTAR
if vendor in ["orix", "tsl"]:
not_indexed = data_dict["prop"]["ci"] == -1
data_dict["phase_id"][not_indexed] = -1

# Set scan unit
if vendor in ["tsl", "emsoft"]:
scan_unit = "um"
else: # NanoMegas
if vendor == "astar":
scan_unit = "nm"
else:
scan_unit = "um"
data_dict["scan_unit"] = scan_unit

# Create rotations
Expand Down Expand Up @@ -173,7 +169,8 @@ def _get_vendor_columns(header: List[str], n_cols_file: int) -> Tuple[str, List[
Returns
-------
vendor
Determined vendor (``"tsl"``, ``"astar"``, or ``"emsoft"``).
Determined vendor (``"tsl"``, ``"astar"``, ``"emsoft"`` or
``"orix"``).
column_names
List of column names.
"""
Expand All @@ -194,77 +191,101 @@ def _get_vendor_columns(header: List[str], n_cols_file: int) -> Tuple[str, List[
footprint_line = line
break

# Vendor column names
# Variants of vendor column names encountered in real data sets
column_names = {
"unknown": [
"euler1",
"euler2",
"euler3",
"x",
"y",
"unknown1",
"unknown2",
"phase_id",
],
"tsl": [
"euler1",
"euler2",
"euler3",
"x",
"y",
"iq", # Image quality from Hough transform
"ci", # Confidence index
"phase_id",
"unknown1",
"fit", # Pattern fit
"unknown2",
"unknown3",
"unknown4",
"unknown5",
],
"emsoft": [
"euler1",
"euler2",
"euler3",
"x",
"y",
"iq", # Image quality from Krieger Lassen's method
"dp", # Dot product
"phase_id",
],
"astar": [
"euler1",
"euler2",
"euler3",
"x",
"y",
"ind", # Correlation index
"rel", # Reliability
"phase_id",
"relx100", # Reliability x 100
],
"orix": [
"euler1",
"euler2",
"euler3",
"x",
"y",
"iq",
"ci",
"phase_id",
"detector_signal",
"fit",
],
"tsl": {
0: [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe changing unknown1 to sem is the correct way to read TSL files. example below:

            0: [
                "euler1",
                "euler2",
                "euler3",
                "x",
                "y",
                "iq",  # Image quality from Hough transform
                "ci",  # Confidence index
                "phase_id",
                "sem",
                "fit",  # Pattern fit
                "unknown1",
                "unknown2",
                "unknown3",
                "unknown4",
            ],
            1: [
                "euler1",
                "euler2",
                "euler3",
                "x",
                "y",
                "iq",
                "ci",
                "phase_id",
                "sem",
                "fit",
            ],

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, thanks! I've renamed it to "detector_signal", as I find this more descriptive.

"euler1",
"euler2",
"euler3",
"x",
"y",
"iq", # Image quality from Hough transform
"ci", # Confidence index
"phase_id",
"detector_signal",
"fit", # Pattern fit
"unknown1",
"unknown2",
"unknown3",
"unknown4",
],
1: [
"euler1",
"euler2",
"euler3",
"x",
"y",
"iq",
"ci",
"phase_id",
"detector_signal",
"fit",
],
},
"emsoft": {
0: [
"euler1",
"euler2",
"euler3",
"x",
"y",
"iq", # Image quality from Krieger Lassen's method
"dp", # Dot product
"phase_id",
]
},
"astar": {
0: [
"euler1",
"euler2",
"euler3",
"x",
"y",
"ind", # Correlation index
"rel", # Reliability
"phase_id",
"relx100", # Reliability x 100
],
},
"orix": {
0: [
"euler1",
"euler2",
"euler3",
"x",
"y",
"iq",
"ci",
"phase_id",
"detector_signal",
"fit",
],
},
"unknown": {
0: [
"euler1",
"euler2",
"euler3",
"x",
"y",
"unknown1",
"unknown2",
"phase_id",
]
},
}

n_cols_expected = len(column_names[vendor])
n_variants = len(column_names[vendor])
n_cols_expected = [len(column_names[vendor][k]) for k in range(n_variants)]
if vendor == "orix" and "Column names" in footprint_line:
# Append names of extra properties found, if any, in the orix
# .ang file header
n_cols = len(column_names[vendor])
vendor_column_names = column_names[vendor][0]
n_cols = n_cols_expected[0]
extra_props = footprint_line.split(":")[1].split(",")[n_cols:]
column_names[vendor] += [i.lstrip(" ").replace(" ", "_") for i in extra_props]
elif n_cols_file != n_cols_expected:
vendor_column_names += [i.lstrip(" ").replace(" ", "_") for i in extra_props]
elif n_cols_file not in n_cols_expected:
warnings.warn(
f"Number of columns, {n_cols_file}, in the file is not equal to "
f"the expected number of columns, {n_cols_expected}, for the \n"
Expand All @@ -273,13 +294,17 @@ def _get_vendor_columns(header: List[str], n_cols_file: int) -> Tuple[str, List[
"phase_id, unknown3, unknown4, etc."
)
vendor = "unknown"
n_cols_unknown = len(column_names["unknown"])
if n_cols_file > n_cols_unknown:
# Add potential extra columns to properties
for i in range(n_cols_file - n_cols_unknown):
column_names["unknown"].append("unknown" + str(i + 3))
vendor_column_names = column_names[vendor][0]
n_cols = len(vendor_column_names)
if n_cols_file > n_cols:
# Add any extra columns as properties
for i in range(n_cols_file - n_cols):
vendor_column_names.append("unknown" + str(i + 3))
else:
idx = np.where(np.equal(n_cols_file, n_cols_expected))[0][0]
vendor_column_names = column_names[vendor][idx]

return vendor, column_names[vendor]
return vendor, vendor_column_names


def _get_phases_from_header(
Expand Down
2 changes: 1 addition & 1 deletion orix/quaternion/symmetry.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -476,7 +476,7 @@ def plot(
default marker style for reprojected vectors is "+". Values
used for vector(s) on the visible hemisphere are used unless
another value is passed here.
kwargs
**kwargs
Keyword arguments passed to
:meth:`~orix.plot.StereographicPlot.scatter`, which passes
these on to :meth:`matplotlib.axes.Axes.scatter`.
Expand Down
Loading