Use XLSForm injection during project creation

[spwoodcock]

What type of PR is this? (check all applicable)

• 🍕 Feature
• 🐛 Bug Fix
• 📝 Documentation
• 🧑‍💻 Refactor
• ✅ Test
• 🤖 Build or CI
• ❓ Other (please specify)

Related Issue

Fixes #1722

Describe this PR

• PR Update form by injecting mandatory fields and validate it #1763 added the osm-fieldwork update_xlsform work to the validate-form endpoint.
• This PR also adds the functionality during the project creation.
• Users can upload a custom form full of question and have the required FMTM fields injected automatically.
• Removes the custom XML manipulation that was difficult to understand / maintain, replaced with much easier Pandas-based XLSForm concatenation.

Review Guide

• Rebuild the docker image
• Run the API
• Create a new project using a custom form
• Also test using the default / bundled forms

Checklist before requesting a review

• 📖 Read the FMTM Contributing Guide: https://github.com/hotosm/fmtm/blob/main/CONTRIBUTING.md
• 📖 Read the HOT Code of Conduct: https://docs.hotosm.org/code-of-conduct
• 👷‍♀️ Create small PRs. In most cases, this will be possible.
• ✅ Provide tests for your changes.
• 📝 Use descriptive commit messages.
• 📗 Update any related documentation and include any relevant screenshots.

[optional] What gif best describes this PR or how it makes you feel?

[spwoodcock]

@Sujanadh sorry to distract from the fmtm-splitter work, but could you please take a look at this PR tomorrow?

I was coding into the night, so no doubt have missed something. Please check everything works as intended / looks good to you 😄

This will finalise the work on injecting XLSForm fields and allow #1653 to be easily solved.

Thank you!

[Sujanadh]

sorry i didn't see the PR , i will test it

[Sujanadh]

Encountered few issues while testing will post it here briefly.

[spwoodcock]

Thanks! There are definitely some bugs to find 😆

I didn't actually get to test updating an existing form either, so that probably needs some work!

[Sujanadh]

HTTPStatus.UNPROCESSABLE_ENTITY: XLSForm is invalid: There are two sections with the name survey_questions.
yes while appending there is a problem since we need to get common fields and then append likewise

I used default form here.

[Sujanadh]

when trying to check custom form also there is issue

{
  "detail": "XLSForm is invalid: There has been a problem trying to replace ${existing} with the XPath to the survey element named 'existing'. There is no survey element with this name."
}

this was the particular reason that i had to first append mandatory fields before testing for validation.

oh actually my bad , i should debug on for custom form , it works fine on custom form

[Sujanadh]

i don't think we need to validate it again in generate-project-date since we append and validate using validate-form endpoint then return valid appended form which is used while generating project data. For default form we don't need to do anything , instead we can change xls form to allow to create new entity with new geopoint.

[spwoodcock]

Thanks for the tests!

The issue about existing is from the old XLSForm that has calculated fields based on this. As you say, using a custom form without this logic works fine.

Agree we could validate once only!
The reason I left it in is because the API is independent from the frontend.

While it's quite likely our frontend is the only user of the API, calling validate-form prior to generating project files, we can't guarantee it. So validating again is just an addition check, without much performance penalty - what do you think?

[spwoodcock]

i don't think we need to validate it again in generate-project-date since we append and validate using validate-form endpoint then return valid appended form which is used while generating project data. For default form we don't need to do anything , instead we can change xls form to allow to create new entity with new geopoint.

Regarding the default forms, I already updated the forms to remove the verification fields (so they are injected).

I will update them again now to remove the FMTM mandatory fields entirely, and have them injected the same way as a custom form.

This should improve maintainability, as we don't need to update the entire XLSForm library when we change something in the mandatory fields 😄

Related issue I just made: hotosm/osm-fieldwork#300

[spwoodcock]

HTTPStatus.UNPROCESSABLE_ENTITY: XLSForm is invalid: There are two sections with the name survey_questions. yes while appending there is a problem since we need to get common fields and then append likewise

I used default form here.

This is a valid error though! We are having issues with the dataframe merging logic

[Sujanadh]

Thanks for the tests!

The issue about existing is from the old XLSForm that has calculated fields based on this. As you say, using a custom form without this logic works fine.

Agree we could validate once only! The reason I left it in is because the API is independent from the frontend.

While it's quite likely our frontend is the only user of the API, calling validate-form prior to generating project files, we can't guarantee it. So validating again is just an addition check, without much performance penalty - what do you think?

Ah got it, you are right that makes sense.

[spwoodcock]

Made some updates!

Hopefully fixed project creation and XLSForm updating 😄

[spwoodcock]

I made some updates to osm-fieldwork for this, so the main things that need to be done:

• Do some tests all the way through to the ODK Collect usage, to check the logic all works as intended.
• Deploy this to staging for Monday.

I am merging this now so we can more easily test - we can make another PR if needed.