Customer survey for fun and profit

This is what my friend Oliver Grahl did with his product PDF Annotator.
This is the story he told me last month at FemtoConf about how he handled a survey to his customers, what he learned about it, and how he tapped into the results.

PDF Annotator has been around for ˜10 years now. So its user base is quite large. Nevertheless, you don’t need to wait this much time to survey your users.
Having a lot of users allowed him to only survey the most active ones, which may have been about 20% of them at that time.
The survey took form as an email sent with open-ended questions.
The response rate was about 1%.

This is Oliver talking here:

Since PDF Annotator is more a tool than a solution to a single problem, we wanted to learn about use cases: what people actually do with our software. We also invited them to attach examples (i.e. documents they had worked on).

The majority of responses was very detailed, up to multiple pages long.

Some responses also included questions, or we had additional comments or questions, so we got into hundreds of conversations in parallel, which could not be managed anymore at some point.

Remember, only 1% response rate. 10 response are manageable, meanwhile, when numbers add up you get a lot of information but the problem lies more in having the bandwidth to handle it by hand.
At one point Oliver saw that he wouldn’t be able to personally respond to everyone and made sure he sent a generic « thank you / sorry if you did not get a personal reply » to all of them, just in case.

Oliver again:

Later on, we set down and tried to summarize everything we learned from these responses, and I hired a guy who worked on this for one or two weeks. Just to read everything took him days, then his task was to summarize the stuff.

So we got summaries of use cases, suggestions, and major pain points to address.

First benefit: better understanding of how people use the product, along with a prioritized improvements list.

Mining customer verbatim

Oliver:

From the use cases, I compiled a list of ~100 key phrases and keywords, as well as a weighted list of industries where our software is in use.

From those lists, we created articles (like how-to articles, step-by-step instructions) to post on our website, email newsletter and Facebook page.

We also created ~20 long tail landing pages. We created those manually, including copy, illustrations/screenshots etc.. I doubt this could have been automated in a reasonable way.

This is where this get interesting.
Let me summarize what Oliver extracted from what his customer are saying

• uses cases
• key phrase and keywords
• which led to articles, landing page, newsletters, and social media post

Harness results for traffic acquisition

Oliver then worked with a PPC expert to set up campaigns with a provided keywords list so he could start to buy traffic for them, pointing to the landing pages. At least for keywords that he was not yet bidding on.

How does a free-form survey fares in term of effectiveness

Oliver talking:

All in all, it was an effective way for us to get insights on what people were using our product for, so we could apply that knowledge to our product planning. At the same time, we were able to produce content and build new keywords for our PPC campaigns.

The downside was that it bound a lot of resources for quite some time, and some customers who engaged significantly by providing very detailed feedback may have been frustrated with our late and probably quite short responses.

Free form survey against fill-a-form survey

This was quite an experience, going through all these answers. But…

Oliver:

So, I would not do such a « free-form survey » (just reply to this email!) anymore, and in later surveys, we switched to a more traditional form based survey style.
But the open « Anything else you want to tell us » question is still part of those surveys, and still very valuable.
Just, people don’t expect immediate replies to what they put into such a survey form compared to an email reply.

Thanks a million Oliver for having taken the time to explain in detail the ins and outs of customers survey.