docs: how to create invalid cases in the test-suite?

[jkowalleck]

test suite's purpose is defined as

purl-spec/PURL-SPECIFICATION.rst

Lines 470 to 485 in 65d98c4

	- **is_invalid**: a boolean flag set to true if the test should report an
	error
	To test ``purl`` parsing and building, a tool can use this test suite and for
	every listed test object, run these tests:
	- parsing the test canonical ``purl`` then re-building a ``purl`` from these parsed
	components should return the test canonical ``purl``
	- parsing the test ``purl`` should return the components parsed from the test
	canonical ``purl``
	- parsing the test ``purl`` then re-building a ``purl`` from these parsed components
	should return the test canonical ``purl``
	- building a ``purl`` from the test components should return the test canonical ``purl``

@johnmhoran via #368 (comment):

thinking of some other test objects I've seen, must a canonical_purl be either null or a valid purl? Or another way to think about it: when is a canonical_purl null, and can it ever be an invalid purl (like https://github.com/package-url/purl-spec/blob/main/test-suite-data.json#L197)?

we are not certain how an invalid test must be structured.
sure, is_invalid must be true, but how about the other parts?

And we should find a consensus, and write guidelines for that.

[johnmhoran]

Thank you for opening this issue @jkowalleck. The existing test directions are a good start but are incomplete in a number of respects, including (but not limited to) the structure and content of an invalid test.

@pombredanne Can you please provide guidance on this important question? (This overlaps inter alia with the ongoing, unresolved discussion about when percent-encoding is required, permitted, prohibited etc. -- focused atm on the qualifiers core spec updating (see #398 (comment)).)

Perhaps we can discuss at tomorrow's biweekly purl community meeting.

• Dates and times:
  • 17:00 to 17:30 UTC
  • 18:00 to 18:30 CEST (Europe/Brussels)
  • 12:00 to 12:30 EST (America/New_York)
  • 09:00 to 09:30 PST (America/Los Angeles)
  • 02:00 to 02:30 JST (Tokyo, Japan)
• Attendee information:
  • https://meet.google.com/ydj-qwbs-iiv

[pombredanne]

I am working on a JSON schema for tests FWIW.

[johnmhoran]

That ^ is welcome news. 👍