Generate schema in CI #197

sujuka99 · 2023-04-21T11:07:54Z

closes #71

Reasoning: Ease testing, provide more flexibility

Do not make snapshots of the real schemas, this is handled in the CI Focus on making sure that the functions work as intended. No need to test output correctness of`gen_config_schema()`, it is essentially an alias for the pydantic schema_json_of() command

.github/workflows/ci.yaml

.github/scripts/run_mypy.sh

docs/docs/user/references/cli-commands.md

hooks/run_mypy.sh

docs/docs/schema/config.json

kpops/utils/gen_schema.py

hooks/gen_schema.sh

kpops/utils/gen_schema.py

tests/cli/test_schema_generation.py

disrupted · 2023-04-26T11:21:19Z

the gen-docs hook is failing on my local machine

gen-docs.................................................................Failed
- hook id: gen-docs
- exit code: 1
- files were modified by this hook

Docs saved to: /Users/disrupted/bakdata/kpops/docs/docs/user/references/cli-commands.md
sed: 1: "/Users/disrupted/bakdat ...": extra characters at the end of d command

might be due to different version of sed?!

tests/cli/test_schema_generation.py

sujuka99 · 2023-04-26T11:36:11Z

the gen-docs hook is failing on my local machine

gen-docs.................................................................Failed
- hook id: gen-docs
- exit code: 1
- files were modified by this hook

Docs saved to: /Users/disrupted/bakdata/kpops/docs/docs/user/references/cli-commands.md
sed: 1: "/Users/disrupted/bakdat ...": extra characters at the end of d command

might be due to different version of sed?!

Yea, it behaves differently on mac, I will fix it

I guess it didn't work at all until now if it never failed for you or @raminqaf (He also uses macOS, right?)

tests/cli/test_schema_generation.py

stale

tests/cli/test_schema_generation.py

disrupted · 2023-04-26T15:19:48Z

might be due to different version of sed?!

Yea, it behaves differently on mac, I will fix it

now I am getting

gen-docs.................................................................Failed
- hook id: gen-docs
- exit code: 1
- files were modified by this hook

Docs saved to: /Users/disrupted/bakdata/kpops/docs/docs/user/references/cli-commands.md
sed: illegal option -- -
usage: sed script [-Ealnru] [-i extension] [file ...]
        sed [-Ealnu] [-i extension] [-e script] ... [-f script_file] ... [file ...]

edit: I believe sed -i '' ... works

sujuka99 · 2023-04-26T17:06:29Z

might be due to different version of sed?!

Yea, it behaves differently on mac, I will fix it

now I am getting

gen-docs.................................................................Failed
- hook id: gen-docs
- exit code: 1
- files were modified by this hook

Docs saved to: /Users/disrupted/bakdata/kpops/docs/docs/user/references/cli-commands.md
sed: illegal option -- -
usage: sed script [-Ealnru] [-i extension] [file ...]
        sed [-Ealnu] [-i extension] [-e script] ... [-f script_file] ... [file ...]

edit: I believe sed -i '' ... works

How about now? Apparently there is no way to make the -i flag work on both Zsh and bash with the same syntax..

I added a check in the CI to ensure that the scripts work in Zsh in this PR

tests/cli/test_schema_generation.py

disrupted

last one, otherwise LGTM

tests/cli/test_schema_generation.py

sujuka99 added 7 commits April 21, 2023 10:07

feat: Add flag that toggles inclusion of stock types

a64295b

Reasoning: Ease testing, provide more flexibility

fix: pre-commit hooks rely on typer-cli

a8ab09e

fix: components undefined

aa66762

docs: update

e0d2905

fix: assignment type inconsistencies, update doc

08f2d26

feat: Add flag to kpops schema pipeline

7b57574

sujuka99 self-assigned this Apr 21, 2023

sujuka99 added 3 commits April 21, 2023 15:13

ci: generate schema in pre-commit hooks

2b4bbe4

Merge remote-tracking branch 'origin/main' into ci/schema

d11f09f

fix: Generated schemas have their keys sorted

ea3b4b4

sujuka99 marked this pull request as ready for review April 21, 2023 13:55

sujuka99 requested review from raminqaf, philipp94831 and disrupted as code owners April 21, 2023 13:55

disrupted reviewed Apr 24, 2023

View reviewed changes

.github/workflows/ci.yaml Show resolved Hide resolved

.github/scripts/run_mypy.sh Outdated Show resolved Hide resolved

docs/docs/user/references/cli-commands.md Show resolved Hide resolved

refactor: move scripts to a more suitable dir

2e19ce4

disrupted reviewed Apr 25, 2023

View reviewed changes

hooks/run_mypy.sh Outdated Show resolved Hide resolved

refactor: fix variable names bash

2b0dc3b

disrupted reviewed Apr 25, 2023

View reviewed changes

docs/docs/schema/config.json Outdated Show resolved Hide resolved

disrupted requested changes Apr 25, 2023

View reviewed changes

kpops/utils/gen_schema.py Outdated Show resolved Hide resolved

kpops/utils/gen_schema.py Outdated Show resolved Hide resolved

refactor: fix anti-pattern

4c5c0d4

disrupted reviewed Apr 25, 2023

View reviewed changes

kpops/utils/gen_schema.py Outdated Show resolved Hide resolved

fix: pre-commit hooks

05eb319

disrupted reviewed Apr 25, 2023

View reviewed changes

hooks/gen_schema.sh Outdated Show resolved Hide resolved

sujuka99 added 2 commits April 25, 2023 18:03

fix: remove unnecessary flag

83d4661

fix: silence mypy tuple-related errors

f05f4f2

disrupted previously requested changes Apr 26, 2023

View reviewed changes

kpops/utils/gen_schema.py Show resolved Hide resolved

disrupted reviewed Apr 26, 2023

View reviewed changes

tests/cli/test_schema_generation.py Outdated Show resolved Hide resolved

disrupted reviewed Apr 26, 2023

View reviewed changes

tests/cli/test_schema_generation.py Outdated Show resolved Hide resolved

tests/cli/test_schema_generation.py Outdated Show resolved Hide resolved

disrupted reviewed Apr 26, 2023

View reviewed changes

tests/cli/test_schema_generation.py Outdated Show resolved Hide resolved

sujuka99 added 2 commits April 26, 2023 14:45

fix: sed works differently on mac

aa3791f

refactor: pythonize tests

a3aea09

disrupted reviewed Apr 26, 2023

View reviewed changes

tests/cli/test_schema_generation.py Outdated Show resolved Hide resolved

disrupted reviewed Apr 26, 2023

View reviewed changes

tests/cli/test_schema_generation.py Outdated Show resolved Hide resolved

fix: test

6aa117a

sujuka99 mentioned this pull request Apr 26, 2023

Test shell scripts on zsh in the CI #205

Closed

sujuka99 added 3 commits April 26, 2023 18:24

Merge remote-tracking branch 'origin/main' into ci/schema

c83183d

fix: pass tests

3b0b956

fix: gen-docs works on zsh now

9f534cb

disrupted reviewed Apr 27, 2023

View reviewed changes

tests/cli/test_schema_generation.py Outdated Show resolved Hide resolved

sujuka99 added 2 commits April 27, 2023 13:11

refactor: add type annotation for fixture

6c99e2c

chore: update snapshot

19d4aab

disrupted requested changes Apr 27, 2023

View reviewed changes

tests/cli/test_schema_generation.py Show resolved Hide resolved

fix: test

765c183

disrupted approved these changes Apr 27, 2023

View reviewed changes

sujuka99 merged commit bf7ab44 into main Apr 27, 2023

sujuka99 deleted the ci/schema branch April 27, 2023 11:45

disrupted mentioned this pull request Jun 14, 2023

Refactor input/output types #232

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate schema in CI #197

Generate schema in CI #197

sujuka99 commented Apr 21, 2023

disrupted commented Apr 26, 2023

sujuka99 commented Apr 26, 2023 •

edited

Loading

disrupted commented Apr 26, 2023 •

edited

Loading

sujuka99 commented Apr 26, 2023

disrupted left a comment

Generate schema in CI #197

Generate schema in CI #197

Conversation

sujuka99 commented Apr 21, 2023

disrupted commented Apr 26, 2023

sujuka99 commented Apr 26, 2023 • edited Loading

disrupted commented Apr 26, 2023 • edited Loading

sujuka99 commented Apr 26, 2023

disrupted left a comment

Choose a reason for hiding this comment

sujuka99 commented Apr 26, 2023 •

edited

Loading

disrupted commented Apr 26, 2023 •

edited

Loading