Shared devSat survey

[atopal]

In our last call, we discussed that it would be helpful to see the questions that will go into the Google DevSat survey. I'm sharing here what is currently planned to be asked in that survey. The actual survey has a lot of actual logic to filter based on answers, that I won't be able to replicate here, but I'm hoping the questions are illustrative of the survey itself. I have skipped the screening and demographic questions, but can also add those, if there is interest in that.

Questionnaire

Q1. How would you rate your overall satisfaction with the Web, as a platform and set of tools, to enable you to build what you need or want? Select the response that best describes how you feel.

5 Very satisfied
4 Somewhat satisfied
3 Neither satisfied nor dissatisfied
2 Somewhat dissatisfied
1 Very dissatisfied

Q2. Please tell us why you are with the Web, as a platform and set of tools, to enable you to build what you need or want. Enter your response in the box below. Be as specific as possible.

Q3. What are the biggest challenges you face while developing your web site or app? Please select up to three.

1 Achieving my desired UI
2 Creating the experiences I want because some capability is not there
3 Achieving the performance I want
21 Keeping up with changes to best practices for how to build for the web
22 Keeping up with new features in web standards (JS, HTML, CSS)
23 Keeping up with changes to existing APIs in web standards (JS, HTML, CSS)
8 Automation of frequently performed small tasks
10 Prioritization of web efforts within my organization
11 Developer tools
14 Supporting old browsers
15 Making a design/experience work the same across browsers
20 Testing end-to-end user flows (e.g., add item to shopping cart, sign up as new user, etc.) on multiple browsers
24 Conducting other tests like performance, unit, or component tests
17 Understanding and implementing security measures
18 Keeping up with a large number of new and existing tools or frameworks
19 Understanding how to use existing tools/frameworks
12 Other challenges; enter here: [OTHER: M]
25 Deprecations and removals of language features or Web APIs
13 Exclusive: None of the above

Ask those who find developer tools a challenge when developing for the Web
Q4. What aspects of developer tools make it challenging when developing your web site or app?
Select all that apply.

1 There is no developer tool for the problem I’m trying to solve
2 Correct developer tools exists, but they are missing key features or they don’t give me sufficient information
3 Developer tools are hard to use
4 Developer tools are slow, unreliable or have bugs
5 Developer tools are missing documentation
6 Other challenges; enter here: [OTHER: M]

Ask if making a design/experience work the same across browsers is a challenge
Q5. Why is making a design or experience work the same across browsers a challenge for you?
Select all that apply.

1 Missing features in some browsers
2 Bugs in features in some browsers
3 I don't know what will work across browsers
4 Other reasons; enter here: [OTHER: M]

Ask if has challenge with keeping up with changes to best practices for how to build for the web, with new features in web standards and/or with changes to existing APIs in web standards
Q6. Why is keeping up with changes to the web platform/web standards a challenge for you?

1 I don't know where to go to discover these changes
2 I'm aware of the changes to be made but don't know how to do it or don’t have the tools and guides to do it
4 I'm aware of the changes and know how to do it but don't have the resources (team or bandwidth) to do it
5 I'm aware of the changes, know how to do it and have resources, but they don't align with my current priorities
6 Other reasons; enter here: [OTHER: M]

Ask if testing end-to-end user flows or conducting other tests is a challenge
Q7. What tests do you create and perform on a regular basis? Select all that apply.

1 Manual tests
2 Automated end-to-end tests
3 Performance tests
4 Cross-browser tests
5 Unit tests
6 Component tests
7 Screenshot tests
8 Other tests; enter here: [OTHER: M]

Ask if testing end-to-end user flows is a challenge
Q8. What are the challenges you face when testing end-to-end user flows? Select all that apply.

1 Running tests on different device sizes
13 Running tests in different browsers
2 Generating/Mocking random test data
3 Keeping test scripts up to date with the code changes
4 Flakiness of test results/Unreliable test results
14 Unreliable selectors/IDs for controls
5 No clear guidance on how to create/maintain end-to-end tests
6 No resources (team or bandwidth) to automate it
7 Lack of support from the organization to automate end-to-end tests
8 Setting up my machine to run end-to-end tests
9 Setting up the CI infrastructure to run end-to-end tests
10 Tests are not on the critical path to ship new features
15 End-to-end testing generally feels too time-consuming
11 Other challenge; enter here: [OTHER: M]
12 Exclusive: None of the above

Ask if testing end-to-end user flows automatically is a challenge
Q9. In what languages do you write your automated end-to-end user flows tests? Select all that apply.

1 Java
2 JavaScript
3 C#
4 Python
5 Other language; enter here: [OTHER: S]

Ask if testing across browsers is a challenge
Q10. How do you perform cross-browser tests?

1 Manually only
2 Automation only
3 Both manually and automated

Ask if R tests cross-browsers manually only
Q11. Why are you not automating cross-browser tests? Select all that apply.

1 Manual tests are sufficient
2 Not necessary because majority of my users are using only one browser
3 Only perform cross-browser testing when bugs are reported
4 No time or expertise to automate cross-browser tests
5 Unclear what framework/tooling to use to create automated cross-browser tests
6 Lack of support from the organization to automate cross-browser tests
7 Other reason; enter here: [OTHER: M]

Ask if R automates cross-browser tests
Q12. What are the challenges you face when automating cross-browser tests? Select all that apply.

1 No clear guidance on how to create/maintain cross-browser tests
2 Setting up my machine to run cross-browser tests
3 Setting up the CI infrastructure to run cross-browser tests
4 Getting tests to run on multiple browsers
5 The tests take too much time to run
6 Finding the root cause after a test has failed
7 Keeping test infrastructure up to date (test framework version, browser versions)
8 Flakiness of test results/Unreliable test results.
9 Time investment to automate cross-browser tests is too high
10 Tests are not on the critical path to ship new features
11 Other challenge; enter here: [OTHER: M]
12 Exclusive: None of the above

Ask if testing across browsers is a challenge
Q13. Which types of features prevent you from making a design or experience work the same across browsers?
Select all that apply.

1 CSS/layout
2 Scrolling
3 Loading
4 Forms
5 Javascript
8 Available APIs/capabilities on different browsers
6 Other features; enter here: [OTHER: M]
7 Exclusive: None of the above

Q14. What browsers do you support? For the purposes of this research, support means ‘build for and continue to support access on this browser’. Select all that apply.

1 Chrome
2 Firefox
3 Safari
5 Edge
7 Samsung Internet
8 Opera
9 UC Browser
10 QQ Browser
11 Other browser; list here: [OTHER: M]
12 Exclusive: None of the above

Capabilities Section

Q15. How much do you agree or disagree with the following statement?
All of the functionality I need to build my web app is available on the Web platform.
Select the response that best describes how you feel.

5 Strongly agree
4 Somewhat agree
3 Neither agree nor disagree
2 Somewhat disagree
1 Strongly disagree

Ask if R doesn’t agree that the Web platform has all the functionality needed to build their web app
Q16. What user journeys within your web app have been hard or impossible to create due to missing Web platform capabilities?
Enter your response in the box below. Be as specific as possible

Ad hoc section

Q17. Among these browsers, which ones would need to support a web feature before you can use it?
Select all that apply.

1 Chrome
2 Firefox
3 Safari
4 Chrome for Android
5 Edge
6 Safari on iOS
7 Internet Explorer
8 Samsung Internet
9 Opera
10 UC Browser
11 QQ Browser
12 Other browser; list here: [OTHER: M]
13 Exclusive: None of the above

Q18. For , how far back does it need to support a web feature before you can use it?

1 Current version is enough
2 Needs to be supported for at least one year
3 Needs to be supported for at least two years
4 Needs to be supported for more than two years
DK Don’t know

[captainbrosset]

For context, this is also related to #15 which is where we started discussing about tracking, over time, the top pain points that web devs experience.

At first sight, it seems to me like devSat does this very nicely, and I would hate to have to replicate this. If Google already runs this on a regular basis, and is willing to consider sharing the results (also to the demographic questions), I don't think there's a need for the WebDX group to try to reinvent something.

Some random comments about the questions:

8 Automation of frequently performed small tasks

I'm not sure how this would be used. What is the intent of this question?

14 Supporting old browsers

Just a thought: now that IE11 is behind us, it might be nice asking a follow-up question for people who select this, asking about which older browser they find difficult to support.

12 Other challenges; enter here: [OTHER: M]

This is great to see. If new patterns start emerging over time within this field, then that could influence the list of questions.

[tidoust]

Q2. Please tell us why you are with the Web, as a platform and set of tools, to enable you to build what you need or want.

I note I find it hard to grasp what "why you are with the Web" means. Is that "why you use the Web", "why you picked up the Web", something else?

[tomayac]

"Why do you choose the web platform?"

[atopal]

@captainbrosset

8 Automation of frequently performed small tasks
I'm not sure how this would be used. What is the intent of this question?

This is not so much a web platform pain point, but more of a dev experience issues with workflows. Do you mean it's unclear how we'd address issues?

14 Supporting old browsers
Just a thought: now that IE11 is behind us, it might be nice asking a follow-up question for people who select this, asking about which older browser they find difficult to support.

Great point!

@tidoust

Q2. Please tell us why you are with the Web, as a platform and set of tools, to enable you to build what you need or want.
I note I find it hard to grasp what "why you are with the Web" means. Is that "why you use the Web", "why you picked up the Web", something else?

Yes, as @tomayac said, it's "Why do you choose the web platform?" as a dev, but I agree, it's not straight forward, I'll bring that up as feedback.

[atopal]

The screening questions have a lot of logic, but I'll follow up with them shortly

[atopal]

Screening Questions

In what country do you currently work?
Select one.

1 Canada
2 United States
3 Brazil
4 Germany
5 France
6 India
7 Japan
8 United Kingdom
9 Spain
10 Mexico
11 Russia
12 China
14 Indonesia
13 Somewhere else

Are you currently…?
Select one. If more than one answer applies, select the one closest to the top of the list.

1 Working as a full-time employee
2 Working as a part-time employee
3 Working as a contractor/on a temporary assignment
4 Working at a company or business I own
5 Working as a freelancer
6 Working as a consultant
7 Retired
8 A student
9 None of the above
REF Prefer not to answer

Show if (JOB_TYPE=2): Does your company…?
Show if (JOB_TYPE=1): Do you…?

1 Develop products for end consumers
2 Develop products for other companies
3 Develop products for both end consumers and other companies

Show if (JOB_TYPE=2): Does your company provide development services, including web development, to other organizations?
Show if (JOB_TYPE=1): Do you provide development services, including web development, to other organizations?

1 Yes
2 No, we only develop in-house to meet our own needs

Show if (JOB_TYPE=2): Does the company/business you work at have a web experience?
Show if (JOB_TYPE=1): As a <S3_SHOW> do you typically work with or consult on web experiences?

1 Yes
2 No

Which–if any–of these describe your experience with software development?
Select all that apply.

1 I write code and develop software on a regular basis
2 I manage or oversee software developers
3 I make decisions related to software development
4 I am a student
5 I am a teacher
6 Exclusive: None of the above

What is your level of experience in your area of work related to web development?

1 Expert: I can provide guidance, troubleshoot, and answer questions in my area of work
2 Advanced: I can perform the actions associated without assistance and am recognized within my immediate team as “a person to ask” when difficult questions arise
3 Intermediate: My focus is on applying and enhancing knowledge or skill. I require minimal guidance to perform successfully
4 Novice: My focus is on honing skills through on-the-job experience. I can understand and discuss terminology, principles, and issues related to my responsibilities
5 Fundamental Awareness: My focus is on learning and I have common knowledge or understanding of basic techniques or concepts
6 Not Applicable: I have not personally worked on development for any applications or solutions in my role

Is your main area of focus in your development…?

Web UI, CSS (Front of the front-end development)
Web app/page functionality (back of the front-end development)
Server side functionality (Back end development)
Both front-end and back-end development (Full-stack development)

Over the past 12 months, which of these platforms have you developed for?
Select all that apply.

Randomize

1 iOS (native apps/games)
2 Android (native apps/games)
5 macOS (native apps/games)
6 Social Platforms (e.g., Facebook, WeChat, Whatsapp)
7 The Web
9 Windows (WPF, UWP, Windows Forms, Win32)
10 Internet of things
11 ChromeOS
8 Others; list here: [OTHER: S]
DK Don’t know

Is your general web development for…?

1 Desktop
2 Mobile
3 Both

[atopal]

The results of the questions have been posted here: https://github.com/web-platform-dx/developer-research/blob/main/vendor-research/Web%20Dev%20Sat%20H1%202023.pdf

Please use this issue for any further questions or comments regarding the survey questions, methodology, results, and possible changes for a future survey