We implemented a new modal within the software that informs the user to sign up for a beta test. If they choose to sign up later, they can access signups through a new badge icon. Additionally, NPS-style feedback was implemented within the app being beta tested in order to quickly gather feedback from all testers.
The first time you log into the software, you will be greeted with a beta test sign-up modal. You can read the FAQs or sign up, but you can also choose to sign-up later.
The visual designer and I collaborated to create a new icon to indicate that a beta test is available. It can be clicked to access the sign-up modal. As we do more beta tests, we hope the icon becomes associated with beta testing.
Since it will be impossible to provide a tutorial for all parts of the new software, we will have to use Pendo guides to navigate the user through the experience.
The quickest way to get metrics on our new product is through an NPS-style modal. Our beta testers agreed to provide feedback when signing up for the test, so this will allow them to easily send us feedback. It can be used as many times as they need.
Initially, we used email to recruit beta testers and we only recruited 6 users. Recruitment for any UX research had always been done through e-mail, so beta testing was expected to follow the same pattern. This dismal number was massively disappointing, so I proposed a new way to recruit: in-app sign-ups. It was predicted that I might be able to get 6 more users with that method, but I was allowed to test my theory that we would be flooded with beta testers.
The KPI I was supposed to hit was 10 retail users. Instead, we had 140 retail beta testers from Friday evening through Monday afternoon, with more still signing up daily. Our conversion rate is around 22% after two business days.
With this multitude of new testers, we have to rework how we get feedback and provide help, since personal discussions with all testers is going to be difficult. I designed an in-app walkthrough and feedback module to organize the process.
Working with a core team of managers assigned to beta testing our new Retirement Planner tool, I planned and scheduled key milestones in order for us to hit a 9/20/19 deadline of releasing the product to users. With their approval and constant feedback, I worked on recruitment methods and research methodologies.
I took inspiration from open beta testing for video games, which often embeds beta test signups within their website. This means that recruits tend to be heavily invested in the game since they visit the website frequently. The same theory could be applied to a beta test in our product; the power users who log into our product daily would be the first to see the beta test signups and have the greatest desire to see what we're doing next.
We learned via heat-mapping, users greatly preferred to use a top navigation bar opposed to Next and Back arrows. By tracking clicks, we were able to see that adjusting retirement needs was uncommon, indicating our formula was syncing with the intuition of financial advisors. Around 7/10 plans created were promoted to Level 2, indicating that advisors need more detailed planning. Our next stage of developing additional modules would place more prominence on the top bar and potentially a hub-and-spoke experience. We left the ability to edit retirement needs for the advisors who indicated they like to put in their own numbers, but we would place less prominence on this feature in future modules. We were satisfied with advisors promoting to Level 2 plans, since they work with clients who may have extremely complicated circumstances.
