Paul Di Pietro

A screenshot of the revamped mobile app selection dialog.

Summary

Role: Product Manager

Time Period: Fall 2019 - Summer 2020

Objective: Begin a larger scale revamp of the onboarding experience for new Xamarin developers using Xamarin.Forms to develop cross-platform mobile apps using C# and .NET. As a side effect, increase the overall success of the onboarding funnel in terms of users successfully deploying a “Hello, World!” app to one’s Android device.

Technologies Involved: Figma, Visual Studio 2019, Xamarin.Forms, C#, .NET, Fluent Design System

Outcomes: Having been neglected for what felt like an eternity, we were able to build out an initial onboarding experience that yielded positive emotional responses from participant users as well as the broader, active Xamarin developer community. In addition, there was a possible small increase in users attempting to deploy to devices rather than virtual emulators; this would have an overall increase on successful deployments as the former generally had fewer issues.

Background

Xamarin was a company acquired by Microsoft in 2016 that specialized in providing developer tools for writing applications using Microsoft’s .NET framework, most notably mobile applications for Android and iOS. The .NET framework also empowers developers to write a variety of applications for Windows, the web, and more, using a variety of its SDKs (software development kits). Developers can choose from a number of supported programming languages. There are many options, but C# is one of the most popular modern options for doing so, and is the language used for developing apps with Xamarin.

How Xamarin worked, in short, is that it provided what we’d call “bindings” to each the native APIs (application programming interfaces), the set of controls and functions that enable a developer to build applications. Xamarin provided two toolkits with these bindings for Android and iOS, so aptly called Xamarin.Android and Xamarin.iOS, respectively, and a third toolkit called Xamarin.Forms for cross-platform app development, for which I was a part of the team.

Developers could write apps using either Visual Studio on Windows, or Visual Studio for Mac on a Mac, but the vast majority of users were developing on the former (and thus is the focus of this exercise). Using Xamarin.Forms, a developer could utilize a single button in their Xamarin application, for example, and the toolkit would be able to output the native control for each platform.

<Button Text="Click Me!"
              VerticalOptions="Center"
              HorizontalOptions="Center"
              Clicked="OnButtonClicked" />

Using the above example, instead of having separately write apps for each platform, a Xamarin.Forms app could define a control such as this, and the tool could translate a generic Xamarin.Forms button control into the native ones for Android and iOS. This is a simplification, but generally how the platform worked. Write once, run on multiple platforms, and maintain a singular code base in terms of application logic.

A visual representation of how the Xamarin.Forms APIs worked.

Using Xamarin.Forms, software developers who wrote applications for other platforms using C# could take their existing foundation of skills and easily learn how to write new applications for mobile devices. Due to the business benefits, the Xamarin.Forms toolkit was far and beyond the most popular option of the three Xamarin provided.

User Trends & Research

In my time with Xamarin and Microsoft as a program manager, I noted a very strong trend that I was almost sure to be true, but wanted to ultimately confirm: most of our users were older, more experienced developers who had been writing code for years.

Suffice to say that there’s nothing inherently wrong with an older user population, but when looking at the market trends and competitors such as React Native and the then up-and-coming Flutter developed by Google, languages such as JavaScript had a hefty market share and have been growing since (note: Flutter apps run on Google’s Dart language, but the toolkit was gaining popularity).

Using my own previous experiences trying to get back into programming, I felt concern about Xamarin and .NET’s ability to gain mindshare with younger developers trying to learn to code. While this could be the subject another piece entirely, C# and .NET could hold a certain stigma, and Xamarin specifically was not necessarily as new user friendly as it could have been.

First, I wanted to confirm this for myself, and so we ran one of our regular surveys, I included specific questions not pertaining directly to age but pointed enough to gain insights on our developers. That survey received over 700 responses across both targeted responses (via notifications inside the Visual Studio IDE) as well as over a call for feedback over Twitter.

Some takeaways from my original writeup:

  • Based on survey responses, overall CSAT is at 77.34%. In addition, we have found that Xamarin CSAT tends to correlate not only to total experience with Xamarin, but total years of experience as a developer.
  • While Xamarin CSAT is at 83.65% amongst developers with over ten years of coding experience, for those with less than ten years of experience, CSAT falls to 70.11%. When isolated to those with less than five years of coding experience, CSAT falls to under 65% (64.38%).
  • When focused on total Xamarin experience, the highest CSAT occurs amongst developers with 6-12 months of experience (83.6%), with 1-3 years of experience close behind (81.76%).
  • When asked how many years of coding experience they had, less than 2% of survey respondents had under a year, and less than 6% indicated 1-2 years. The rest of the answers followed a much more even curve of distribution.
  • While roughly a quarter of respondents indicated that they code for over ten hours a week in their spare time, almost a third spend less than two hours a week, with 10% of respondents not writing code at all.
  • Of all respondents, 40% have developed 1-2 apps in their spare time, but almost a quarter haven’t done so at all.

Barely any developers responding to the survey were new. The top response was from people with up to 20 years of experience!

In short, the vast majority of respondents were long-term developers who’d been writing code for years, and developers who had more experience were more likely to put up with a less-than-stellar user experience in terms of CSAT (customer satisfaction). This satisfaction dropped significantly in relation to how much newer an individual was to writing code.

Users with less than a few years of coding experience had a very low customer satisfaction score.

This is of course a survey which could self-select, but my own experiences and the sheer size of total responses led me to believe that my general hypothesis around the user base was not too far off. Regardless of user breakdown, our CSAT for newer Xamarin users definitely skewed much lower than we’d ever like to see, and thus I wanted to spend more time ensuring that the new user experience was much more straightforward. We knew we could start with our first time onboarding, specifically.

Creating a New App

Using the context of creating a cross-platform mobile app for Android and iOS, as this was the most common goal using Xamarin.Forms, new users learning to use the platform would traditionally go through the following workflow:

Step 1: Install Visual Studio and selecting to include the mobile workload (i.e. the package of utilities that provide the ability to write apps using Xamarin’s tools).

Step 2: Create a new app using a project type (e.g. ‘Cross Platform App’) and template.

Step 3: Create a new emulator or ensure their existing device is enabled for debugging (testing a running app).

Step 4: Run the basic template to verify that everything works correctly.

Step 5: Start following tutorials, documentation, etc.

First and foremost, through some sessions with individuals where we observed them going through the onboarding process for the first time, we knew there were issues with our template selection, which looked like this:

What creating a new Xamarin application inside Visual Studio historically looked like.

Two major UX issues were:

  • There was no differentiation between the templates. A descriptor to the right helps a little, but only conveys so much information when the layout isn’t visualized.
  • Why is the Windows platform unchecked by default? Xamarin.Forms is also meant to be cross-platform, so why are these listed here at all? Technically, the answer is that each platform being selectively enabled provides underlying mechanisms for running the app on each targeted platform. If a user will never build for a given platform, it reduces the overall size of the project. However, this can still cause confusion.

Assuming the user got through this and created a new project: they’d land inside the Visual Studio UI:

The overwhelming VS interface. What does one do next?

A lack of guidance didn’t help the user here, and they’d have to figure out how to get an emulator (a virtual representation of an Android device) up and running. After figuring out how to get in there, that UI was exceptionally overwhelming:

The long-standing emulator configuration, with what feels like an endless amount of options.

Most of these options were often never changed based on our data, but a new user might not know what they need. Most individuals using Visual Studio and Windows were targeting Android devices to start with, and this was notably easier than trying to build for iOS, which still required a Mac, and will be left out of discussion for this exercise. That said, we wanted to do better than provide an overwhelming set of options for a new user while not removing the ability for advanced users to configure these emulators should they wish.

If they did have an Android device, it would be relatively easy to set up for debugging, provided Developer mode was enabled on the device in question. However, this was a fairly hidden process and required navigating to a specific menu in the OS’s system settings and tapping a part of the screen a certain number of times.

We also had a strong preference for using devices when possible for another reason: in our user research sessions, the emotional response of a user was visibly stronger when they saw an app running on their own device, even if it was a simple “Hello, World!” without modification. If we could elicit such a strong “Wow, that’s cool!” response, we reasonably assumed they would be more likely to keep learning.

If we could quickly get them through this initial process, however, the subsequent experience should be as simple as clicking “Run” to start their app, all in less than a few minutes after installing Visual Studio.

With this, we had three primary goals:

  1. Increase the success of our new user funnel from VS installation through a successful running of an application

  2. Better visualize the application being created, as well as improve terminology and expectations when creating an app for the first time.

  3. Ensure the user could easily create a new default emulator or connect their own device.

An early set of lo-fi mockups envisioning what I had in mind for the experience. (Right click → Open Image in New Tab for a larger version if desired.)

(Right-click -> Open Image in New Tab for large)

After putting together some concepts, I was able to present them to various stakeholders, including an internal review board for implementing features and/or experiments in Visual Studio. The proposed work was approved, and enabled us to start development of the feature under an experimental flag. I then collaborated with a UI designer to refine my original workflow and mockups, which were then built in coordination with an engineer on my team.

During the process, we were able to source participants to run through the onboarding with us and provide additional feedback for refining the experience. For example, explanation text could be made clearer with regards to instructions in certain places.

Resulting Improvements

After our work was completed, when creating a new app, users would be prompted with the following selection view:

The new visualization following Microsoft's Fluent Design guidelines, with better illustration and terminology.

(Note: the above is an actual screenshot reflecting the actual UI; the image at the top of the page is a higher resolution SVG rendering.)

After creating their new project and landing inside the Visual Studio UI, we could detect if they had recently installed it and the mobile workload, and present them with a first-time walkthrough of setting up their Android device or emulator. If they opted to use a device, they received guidance and a visual indicator of when the device was recognized by Visual Studio:

Clear steps are provided for setting up a device for USB debugging.

If opting for an emulator, the experience was even simpler:

Our new default emulator creation.

Resulting Improvements

After running our experiment, we noted the following results:

  • We had an extremely positive public response to the changes that we made once finally live.
  • Net usage of devices went up 25-30% in the treatment cohort.
  • Emulator usage dropped.
  • First deployment success rates (i.e. once a user got to the point of clicking “Run”) stayed about the same. This last part was not surprising as we knew that once a device was connected, it was very likely to work. However, due to more individuals using devices, the net success rates would be higher by 3-5% against weekly new users in the hundreds.

My time at Microsoft ended before I was able to delve deeper into long term funnel impacts.

Concluding Thoughts

I was extremely proud of what my small team working on this project accomplished at the time. Longer term I would have loved to build out more integrated walkthroughs as I strongly believe new users could have benefitted from understanding how to navigate the project and build their apps.

I cannot speak for the current marketshare of the toolkit (which is now known as .NET MAUI after a rebranding), but I strongly believe that younger, burgeoning developers love engaging, visual ways of learning, and anything that can be done to make those things easier will succeed in the long term.