Exploratory research - patterns emerge
We don't want to wait for developers to have to ask us to look at these issues.
I asked these key questions
- How many developers are there in your organization? How many AppSec engineers?
- Can you tell us about the last time an issue was remediated?
- What are your general guidelines around remediation/ignoring?
- Who has permissions to ignore issues?
- Tell us about the last time an issue was ignored
- Who ignored it?
- Why was it ignored? (it can't be fixed, it doesn't actually present risk, etc.)
- How long was it ignored for?
- How was it ignored? (manually or through a policy)
Through analyzing the feedback, customer types and pain points emerged:
Ignoring freely
Developers have permissions to ignore. AppSec team monitors ignored issues after the fact.
Pain points
- Risky issues may be ignored, leaving the product vulnerable.
- AppSec governance and control is harder / not possible.
Goal
Maximizing security by easily monitoring which issues are ignored, so that vulnerable projects aren't being introduced into the product.
Only AppSec team ignores
Developers don't have permissions to ignore. Only the AppSec team ignores issues.
Pain points
- Ignore requests come in through different, unaudited mediums.
- Developers are blocked while waiting for issues to be ignored.
Goal
Reduce developers' time to merge and AppSec team overload by centralizing the requests and needed context, reducing response time.
In-house approval workflow
Developers can request to ignore issues from the AppSec team through a custom-built solution.
Pain points
- These solutions require ongoing maintenance and training.
- Decentralized audit trail - ignore rationale stored externally to Snyk.
Goal
Provide centralized auditing available both internally and for external audits, as well as reduce maintenance cost and effort.
A deep rooted issue crystallized through the main flow
When analyzing the user flows, it was apparent that ignore requests can come through many different surfaces (Snyk UI, CLI, IDE, PR Checks). The consistency across these surfaces, as well as across different development branches, was partial at best. It was clear that consistency needs to be achieved before implementing the approval process, because:
- If the same issue appears in different places, many different requests would be created for that same issue
- Approving one request would still leave the rest of the similar issues open and would not be reflected across the different surfaces
This resulted in a technical project - "Consistent Ignores" - which was developed as the base for implementing the approval workflow.
Tested initial design to validate the concept
Having mapped the flow and created an AI prototype, I tested the process and initial screen designs with users.
I collected feedback on these topics:
- Is the process clear? Does it make sense?
- Are we providing enough information to approve/reject the request?
- Based on what information would the reviewers triage the requests?
- How and when would the users like to be notified of events in the process?
- How would the different parties communicate about the request? For example - if the reviewer needs more information to approve.
Ignore request list
Ignore request list - viewing request
Synthesized the research findings into digestible information
- The overall process was on the right track.
- This flow would allow AppSec teams to streamline issue ignoring and reduce developer downtime.
- Audit history would be available (as opposed to when using workarounds like Slack)
- Notifications would help speed up development and improve efficiency.
Discovered these key points for improvement
1. In-product conversations instead of a "Details" field
Instead of a Details field which can be edited by anyone, customers needed a way for all parties to leave distinct, timestamped comments throughout the request process, highlighting the importance of an auditable, uneditable history.
2. More details needed to respond to a request
The AppSec team would need as much information as they could get in order to effectively investigate and respond to the request. For example, the repository, branch, line number, details on the vulnerability, and more.
3. Assigning reviewers to a request
This provides visibility into who is handling the request, preventing duplicate effort and allowing the requester to be informed that their request is in progress.
Incorporated feedback into the design
The users provided valuable feedback regarding features they were missing or that they wouldn't be able to use. I used their comments to create the next version.
Ignore request list
Ignore request list - viewing request
Key changes
1. In-product conversations
- The reviewer and the requester would be able to add comments to the process at every stage
- These comments will be saved in the Comments tab and in the audit history
- The original requester comments would be locked to editing
- Notifications will be sent out when new requests are created, as well as when new comments are added to the request
2. More details added to the request details panel
While not all the requested information was technically available, much more could be included than was previously shown.
3. Ability to assign a reviewer to the request
Assigning a reviewer to the request to provide more transparency in the AppSec team as well as externally with the developers.
The new design was much better, and more feedback was received
I conducted a second round of research on the updated designs to test the changes.
Additional points for future consideration were brought up:
1. Showing even more information about the issue
For example, the relevant code snippet - this information is crucial to understanding the context of the request.
Under the hood (for the curious reader)
Showing the code snippet was not possible for most requests generated in the IDE and the CLI, because:
- Snyk doesn't save any of its customers' code.
- Where Snyk does show code snippets in the Snyk app, it is retrieved from the integrated SCMs.
- Most requests from the IDE and the CLI are created in the context of local code, which does not have SCM branch context.
Currently we are working on adding the code snippet where we have branch context.
In the future we will explore ways to connect the local code to the remote repository.
2. Different options for being notified about requests
Both in frequency and method of communication.
3. Saving request history for a longer period of time
Would be helpful for audit purposes.
4. Editing an approved request
Some customers mentioned wanting the requests to persist past rejection/approval, allowing the audit history to show how many times the vulnerability was ignored/requested to ignore, and when.
For now, this request would be answered by providing audit history for the issues, rather than the requests.
Under the hood (for the curious reader)
In one of the design iterations not presented here, the option to send approved requests back to Pending was presented as an option to edit/reject.
Users weren't sure what would happen to the ignored issue when using this option - would it be "unignored" (as it is called in Snyk)? Would it stay ignored until further changes to the request? Would changes kick off a new review process?
Following the feedback, I tried adding an "unignore" action to the approved request. This was just as confusing to users, who didn't understand what it would mean in this context - they didn't realize that the action was referring to the issue, and not the request.
At this point it was decided to separate the request actions from the issue actions, and to go back to this request when we have more feedback about the process as a whole.
Went back to user patterns to review the value
Ignoring freely
Value
Reduced risk due to greater control on which issues are ignored and for how long.
Measuring success
- Reduced number of ignored issues due to closer monitoring.
- Increased number of issue fixes, as fewer issues are ignored when merging.
Only AppSec team ignores
Value
- Minimizing the time developers are blocked waiting for issues to be ignored.
- Reduced AppSec team overload, centralizing request information.
Measuring success
Reduced time to merge, as the process of ignoring is more streamlined and effective.
In-house approval workflow
Value
- Reduced maintenance of custom, in-house solutions.
- Centralized audit on all ignore information, for internal audits as well as external ones.
Measuring success
Adoption of the feature as customers are able to stop using their own custom solutions.
Policies and context would reduce noise and workload
As the topic of Ignores continues to evolve, there are interesting opportunities in AI which can deliver value both to customers and to Snyk:
1. Suggesting policies to ignore more efficiently
As Snyk learns about the patterns of ignoring issues for different customers, it can make data-driven suggestions about ignore policies which cover specific use cases and are based on context.
For example: noticing that a repository only deals with backend services and suggesting to ignore all frontend related issues.
2. Better context for issue severity
In some cases it may be possible to go one step further and instead of ignoring more widely, Snyk could do a better job of flagging irrelevant issues in the first place.
As per the previous example: instead of ignoring all frontend issues in backend repositories, the issues wouldn't appear in the scans in the first place.
Testimonial
What the PM said
I've had the pleasure of working with Mor on a couple of projects during my time at Snyk, and it's always been an exceptional experience. She is so dedicated to being embedded with the team so there's no misalignment, and her output is both high quality and rapidly delivered. I think the best thing about working with Mor is that she is user hungry, and will be on every call possible to ensure the product and engineering team has absolute confidence in the experience she is delivering, as it's constantly validated.
Waleed Arshad
Staff Product Manager, Snyk