Background
Understanding the impact, usability and pain points of Snyk's Git experience
In a nutshell
In this research, I set out to understand the impact, usability, pain points, and feedback Snyk's customers have on the current Git experience.
What prompted the research
Snyk was getting feedback and critique regarding its Git experience. There were features being requested by the users and critique on the current ones, and we wanted to learn more in order to prioritize and create the upcoming roadmap.
Approach
- In depth interviews regarding the customers' current work processes.
- Exploration of specific Snyk features and how they're being used.
- A study of previous user requests regarding this topic, specifically blockers for working with Snyk.
Constraints
Only one month was allocated for all stages of the research: gaining in depth knowledge of the topic, planning, scheduling, executing calls, summarizing and presenting.
Noting my assumptions to focus the research
- Snyk's Git experience is missing features which some of its customers define as key to their work processes.
- It is worth investing in these features in the near future in order to create growth in the company.
- The Git experience affects the relationship between the AppSec team and the developers. Enough that they'd opt out if it wasn't up to par.
- The customer size does not affect the Git experience.
Goals
Identify blockers
Identify the key features which could be blockers for potential customers.
Missing features
Understand which missing key features would make the most impact.
Learn how customers work
Learn more about Snyk's customers' ways of working around Git and security.
Method
I used these steps to run a thorough research
01
Learn
Review existing knowledge & learn more about the subject
02
Assume
Come up with assumptions to validate / invalidate
03
Ask
Phrase the questions to validate / invalidate the assumptions
04
Interview
Conduct interviews to gather data
05
Review
Summarize data from interviews and existing knowledge
06
Conclude
Form conclusions and share forward
Forming questions using The Mom Test method
The Mom Test method teaches how to ask questions that encourage honest, unbiased answers by focusing on real experiences instead of opinions. I applied it in this project to help phrase our questions, following these key guidelines:
Don'ts
- Talk about your idea
- Ask about their opinions
- Avoid generic claims (I usually, I always, I never)
- Ask yes/no questions
- Lead, pitch
Do's
- Talk about their life
- Gather facts
- Ask about specifics in the past / current behavior
- Keep questions open ended
- Listen
Examples of questions
Can you tell us about the last time you imported code into Snyk?
Can you tell us about the last time you had a PR check fail due to security? What did you do?
Looking at this auto-PR (created by Snyk), can you read through it and explain what you see?
We were able to dig further by asking follow-up questions to get to the bottom of the pain points.
Participants
- Snyk developers who use Snyk regularly (dogfooding)
- Existing customers (AppSec Engineers)
- Developers who are not Snyk users (to understand whether they have security processes in place and how they work)
Research findings
Interaction models emerge to tell the detailed story
Throughout the many calls conducted during this research, a wide range of workflows emerged.
Below is an overview of the different approaches observed.
A branching in the process indicates different approaches we've heard from various customers.
Vulnerability discovery and fix, by notification
Periodical security meetings
Using a new Open Source library
As described by developers who are not using Snyk
Research findings
What customers told us, and what to do about it
The conclusions presented here are a sample of the breadth of conclusions the research generated.
Not all are shared due to data sensitivity.
It needs to be a great developer tool first, we'll work around the problems in the security team. We prefer having everyone do the right thing.
01
AppSec Engineer & Developer relationship
AppSec teams put developer experience first, even ahead of their own security and reporting needs.
- Customers may stop using the product if developers are frustrated or if the relationship with them is hurt.
- All AppSecs we spoke with emphasized how important their relationship with developers is.
- Some customers even said that the relationship with the developers is more important than having a robust security solution.
- One customer consciously made a decision to have less reporting visibility (for the AppSec team) in order to use Snyk because it's better for developers than other tools.
- Although some tasks are better handled in the UI, ideally developers shouldn't need to leave their SCM frequently (or at all).
02
Scaling
Snyk could do a better job catering to big companies.
- Customers are experiencing import time out issues and hitting GitHub rate limits.
- Errors in resolving the PR status check (and then failing the PR) for large repositories.
03
Understanding communication errors
Customers can't diagnose failures on their own, so they depend on support to find out what went wrong.
- Snyk's import logs aren't detailed and customers have to reach out to the TSMs to understand why PRs failed.
- Snyk's customers need a centralized way for viewing PR check errors to help unblock developers.
- There isn't a history of failed PR checks, which makes it hard to debug and differentiate Snyk errors from actual vulnerabilities.
- When customers contact support about issues, it's very hard to tell if Snyk is set up correctly. For example, it's hard to tell whether PRs are active for a repo through the Settings. It won't appear on the Settings page for the project but only in the Settings page for the integration.
04
Greater customization and visibility
Customers want finer control, so developers are only blocked when it's necessary.
More granular Policies and Status Check Rules would mean developers are only blocked when necessary. Examples from the research:
- Setting up status check rules with more granular parameters
- Disabling certain types of vulnerabilities, like License vulnerabilities
- Creating Jira Issues for multiple vulnerabilities, selected by the user
What next?
Recommendations for further exploration
This project surfaced many valuable insights, and I recommended further exploration in order to:
- Speak with more developers - most of the conversations have been with AppSecs and PMs.
- Allow more time to base the research on feedback from Snyk's own customers, rather than developers who aren't current users.
- Conduct the research closer to when a feature is planned for release to ensure the results are as accurate and relevant as possible.
- Investigate competitor offerings further and analyze churned customer cases.
- Dive deeper into the human interactions around PRs and better understand the related workflows.