102. Testing Startup Ideas and Assumptions ⚖️ John Li of PickFu Interview

Video version of the interview

In this episode I talked with John Li, co-founder of PickFu. I wanted to interview him because of his focus on testing ideas and assumptions in startups. This is an issue I constantly hammer on in my advising sessions.

John Li is the co-founder of PickFu, a consumer research platform for getting instant feedback from target audiences. Thousands of global brands rely on PickFu to help them make better data-driven decisions about their products and marketing. John is passionate about building systems that help others succeed. He has degrees in computer science from UC Berkeley and the University of Washington. John and his family live in the San Francisco area, where he shuttles his kids around in his dream car (a minivan).

You can follow him at https://www.linkedin.com/in/johnli/

Introduction:

In the fast-paced and competitive world of business, gathering insights and feedback is crucial for making informed decisions. Traditionally, companies have relied on focus groups to test product packaging, marketing strategies, and more. However, these methods often came with high costs and long turnaround times John shares how his company PickFu is changing the game by offering a self-service platform for digital focus groups.

We also explore his experience with growing a 100% remote company.

Introducing PickFu

PickFu provides a user-friendly and self-service platform for creating digital focus groups. John, on of the co-founders of PickFu, recognized the need for unbiased feedback on design decisions and developed a tool to solve their own challenges. What started as a side project quickly gained traction and led to the birth of PickFu.

The Power of Digital Focus Groups

PickFu enables businesses to tap into a global consumer panel of over 15 million individuals. The platform allows companies to ask open-ended questions, gather votes, and receive written feedback from participants. From testing product ideas and pricing strategies to evaluating marketing messages and visual assets, PickFu covers a wide range of testing possibilities. The written responses provide valuable insights into not only which option participants prefer but also why, allowing businesses to understand the thought process behind consumer preferences.

Testing Beyond the Surface

While PickFu excels at testing specific attributes like packaging, price, or visual assets, the platform also encourages businesses to dig deeper. By asking open-ended questions, companies can gather feedback on broader challenges or pain points within a particular industry. This approach allows entrepreneurs to uncover new insights and shape their solutions based on customer needs, rather than simply reacting to their own assumptions.

Remote Work Culture and Scaling

As a fully remote company, PickFu has embraced the advantages of remote work long before it became a necessity during the COVID-19 pandemic. With a team spread across different continents and time zones, PickFu relies heavily on written communication and asynchronous collaboration. They have developed a comprehensive handbook and culture documents to facilitate onboarding and ensure alignment across the team. Regular team-building events and social gatherings over virtual channels further enhance the sense of connection and camaraderie within the organization.

Transcript

Lance

John, welcome to Feel the Boot.

John Li 

Thanks so much for having me.

Lance

So we've talked a little bit before this recording and we exchanged some emails. Why don't you tell us a little bit just for context of what your startup does.

John Li 

Yeah, so my startup is named is called PickFu. You can think of PickFu as a digital focus group. So the same way that large brands and CPGs like Procter & Gamble use large focus groups to figure out their product packaging, testing, marketing, all of that stuff, PickFu lets you do that in a self-service way. And we tap into the same global consumer panels that are used by the large CPGs.

So, PickFu is self-service. You can come on the site, choose your target audience, ask a question, post up to eight different options to either choose from or get feedback from, and you can get responses and feedback from your target audience in typically 30 minutes or less.

Lance

Very cool, but that's not actually where you started, right?

John Li 

No, it's not.

Lance

So why don't you tell us that story? Because I think pivot stories are always interesting because so many companies end up doing things that are radically different than where they started.

John Li 

Yeah, so it's kind of a pivot story. It's kind of not really a pivot story. So my co-founder and I, we have been entrepreneurs for over a dozen years working together. Our backgrounds were in large software development. I was at Microsoft and he was at Hewlett Packard. And we left our jobs to start building websites actually. So our initial...

company together was building a site that aggregated restaurant menus and put those online, learned a lot about sort of just building sites, monetizing sites, SEO, or getting traffic and so on. Along the way, we built a handful of other side projects. But the main thing where PickFu came from is that we, being a team of two, we kept having disagreements about design decisions. Like both of us are software engineers. And so, you know, without

a ton of design experience. And so his, you know, his guess was as good as mine. And my guess was as good as his and every time we would butt heads, we'd ask our friends and family for feedback. And, you know, that they would be, we knew that they would be biased. And at some point they kind of got tired of giving feedback, you know? Um, and so when faced with the problem of, of trying to get unbiased feedback, um, we, we did what software engineers do, which is we built the tool. So we built a tool.

to get us unbiased feedback on our design decisions, logos, you know, logos, page layouts, marketing copy collateral, all of that stuff. And we just kind of put it on the side. I think we put like a payback button on it, posted it to a handful of communities, but continued working on our main project. And that went on for a couple of years. And every couple of months or so we would check in and we saw that PickFu was growing, you know, hey.

More and more people found it. We see people seem to like it. And then we'd go back to our main, then we'd go back to our main project. A couple more months past, we check again, same thing. Oh, now it's showing up and being used in other verticals as well. That's fascinating. And then we'd, we'd focus on the main project again. And so eventually after a couple iterations of that, we realized that, you know, despite our best wishes, the traction on our main project was not going where we wanted it to go and the traction on PickFu was growing.

John Li 

And so we decided to make a pivot there in terms of focusing our efforts, our time and energy on PickFu and growing it. And that's what we've been doing ever since.

Lance

Yeah, it's funny how building a product to solve a problem that you can't find a good solution to tends to be a good way of identifying problems that need solving.

John Li 

Yes. Yeah. When, um, yeah, when you're, I guess the easiest way to build something that people might want is to start looking at your own problems first, solving those. And maybe if that's a big enough problem that you're solving, then that solution can prove useful for other people too.

Lance

So obviously you take testing seriously. How do you feel that testing fits into the process of building a startup, building a company of any sort?

John Li 

I think that testing assumptions and there's a phrase we use like test before you invest, like testing your assumptions, testing your hypotheses, not only potentially lead to better outcomes, but they reduce a lot of downside risk. So I see it. It's kind of like the two aren't exactly the same, right?

John Li 

skyrocket, like it's going to be a rocket ship. But you can very quickly, and if you can test, you can very quickly disprove a hypothesis about, oh, well, you know, I think that this kind of marketing, this kind of marketing message towards this audience is going to be the thing that resonates or this kind of product feature or this kind of product benefit towards this potential target customer set is going to resonate or not. If you can test that out before you actually commit it to whether it's code or print or publication or whatnot, then

you improve your chances of having a better outcome for that project.

Lance

Right, yeah, the more ways that you can fail, you eliminate before you spend money, the better off you are. Do you have any examples of cases where your testing and experimentation led to important effort savings or failed directions or made a critical impact on your?

John Li 

Exactly.

John Li 

Yeah, on our path or on our customers? Sure. I mean, we've definitely helped a lot of customers and some of the biggest names in e-commerce as well. There was one company where we work with a lot of people who sell on Amazon. And there was a company named Thrasio where they had a business, which was to buy up individual Amazon shops and then optimize.

Lance

on yours or your customers.

John Li 

their, uh, optimize their offerings, like their listings and then their products and so on. And so there is this one product called angry orange. It was a pet deodorizing product. It worked really well. You know, it worked really well, but it, the design on the bottle and everything else was kind of subpar. Um, and so the, the theory, the thesis behind, um, that the, that the thrashy team had, was that they could purchase this, uh, listing, rebrand the product, rebrand the bottle and

and do everything, don't change the composition of the product at all, and that it could do better. So, the decision that they faced was that rebranding, they actually went through about 300 different designs for the rebranding process. And the one that they wanted involved a $50,000 production design change because they wanted an orange bottle because the product was named Angry Orange. They wanted it to pop in the listings and on the store shelves and everything. And so to justify that decision.

uh, that go no-go decision. They went to PickFu and ran about six polls cost about $500 in total pulled about 200 to 300 people. Um, so they got those results in about three days and those results, uh, validated that yes, not just did, not only did the design that they wanted to go with beat out the original design, it also beat out all the other variations that they were considering. So that gave those tests, gave them the confidence to pull the trigger.

on that rebranding decision. And then they pulled the trigger on it. And the day after they relaunch the product on Amazon with the new packaging and everything else, their sales 10x'd overnight. So it went from a $2 million a year brand to a $19.5 million a year listing, purely on the basis of the rebranding, like, you know, full marketing rebrand and the packaging and everything else.

Lance

that just to show how much of the decision making that we do, we like to pretend that we're rational about these things, but the packaging, the look, the feel, the name, all those things matter so much. So what sorts of things does PickFu lend itself to testing? Where are the boundaries around?

John Li 

Yeah.

John Li 

Sure. I mean, so on PickFu, you can ask open-ended feedback. So PickFu basically lets you get votes and written feedback from a large set of panelists. We have about 15 million panelists globally. You can test between one to eight different options. So you can get open-ended feedback. Like if you had an idea for a startup, let's say around like nursing homes or car washers or so on, you could just put that...

idea concept, including pricing and everything out there and get feedback on that idea. If you needed to test eight different images for a Facebook ad or something, you could put that there too. You can test for images, audio, text, video. And the nice thing is that because all the panelists provide written responses, you not only understand which option they prefer, you also understand why.

Lance

And what's the ability to target a particular type of user or demographic or background?

John Li 

Absolutely, we are really proud of the depth of targeting that we offer. So we cover all the standard demographics of age, gender, income, and so on. And then we also have really deep targeting, like if you're an Amazon Prime member, if you're a pet owner, if you're specifically a dog or a cat or a horse or a fish owner, if you exercise frequently, whether you take nutritional supplements, the types of video games that you like to play. So almost anything that, if you're selling online, any, almost any

any product category, a lot of digital entertainment categories as well. And we're constantly actually building new targeting capabilities based on the requests from our users.

Lance

It sounds very B2C oriented. Do you have options for more of a B2B kind of targeting?

John Li 

We have some options. Our main focus is because our panels draw from general consumer panels. Most of our targeting is B2C targeting. We do offer some limited B2B targeting, like small business owners and so on, but we're looking to expand those options too.

Lance

So what's your thinking about how to test kind of larger questions about the model? So the examples you're giving tend to be sort of here's the name, here's the price, here's the image, things like that, but trying to validate whether or not kind of the concept is moving in the right direction. Do you have thoughts about the best way to do that?

John Li 

Thank you.

John Li 

Yeah, and so it can be challenging to ask. If you're thinking about getting feedback from the perspective of asking strangers at a coffee shop, like it's, I think the approach towards gathering feedback and data can inform that a little bit. Like I think if you're going in with a very specific value proposition and solution, you can get feedback on that.

But that might not be the most valuable feedback that you could get if you're still in the ideation phase, right, because you've already honed in on a solution and you're asking people to react to that one thing. Whereas I found and sort of from what I've seen, asking...

Asking questions to draw out issues around the problem and the area oftentimes provides more insight into the shape of the solution that you could provide. Let's say you're starting like an Airbnb type company or something. Rather than trying to go out and provide your exact pitch and so on, I think...

it'd be more interesting if you're in the ideation phase to ask open-ended questions about, you know, what are the biggest challenges or downsides that you've faced with short-term rentals, like in your experience, right? Or like, what are the factors that motivate you from choosing one short-term rental versus another? So asking sort of those broader questions can help drive a lot of insight and also uncover new things and sort of angles that might inform you to make a better, like come up with a better solution.

Lance

I like that. It actually reminds me of something that I saw someone post just recently talking about Henry Ford and that thing. You know, you couldn't ask people what they wanted in a automobile because they had no idea of what that was or what to ask for, you know. But you could ask them a lot of questions about their frustrations with horses and how they need to move and how they travel and, you know, where the friction points are. And you go back and think about how that applies to cars.

John Li 

Exactly. Yeah.

John Li 

Yes.

John Li 

And so actually it's funny, it's funny Lance, because we actually have a couple MBA professors who actually have their classes use PickFu for exactly that purpose. And so as part of their, as part of a capstone project, when they're supposed to come up with a, with a business idea and then a business plan and so on. Part of that, part of that process is to, for those students is to go on PickFu and literally ask these questions around, you know, define the problem space.

like the area that you're trying to tackle, and then go on, ask open-ended questions about frustrations, challenges, everything that people have who are operating within that problem space, and then use that to inform and justify what they're proposing as part of their business.

Lance

I really wish I could convince more of the founders that I work with to do that first. So often people will come to me for advising and they've already built the whole solution. They've thought things through their way down the path. And of course, many engineers are resistant to that kind of social interaction, which is I think one of the reasons I'm intrigued by this, your solution is it gets around that. It allows you to do that kind of research in an organized way.

John Li 

That's it.

John Li 

Yeah.

John Li 

Of course.

Lance

without having to pick up the phone and talk to human beings, which people like me have challenges with.

John Li 

It kind of makes sense that two introverted engineers decided to build the solution to get unbiased feedback instead of actually going to the nearby coffee shop and talking to strangers.

Lance

That's right, yeah, nothing worse than talking to strangers. So when we were sort of preparing for this interview, one of the things that you brought up was that when you started to scale your business, you made the decision to go fully remote. And I wanted to sort of understand what your thinking is about that, because that certainly is a trend that I'm seeing in most startups now.

John Li 

Hehehe

John Li 

Yeah, so actually my co-founder Justin and I, we've always been remote. So we met in college and then we went to our jobs our separate ways, but for over the, over the dozen plus years that we've been working together, it's always been remote. So it's kind of been natural for us to just figure out how to work together in a remote environment. So when we made that decision to start scaling the team, it naturally just made sense for us to say, well, we're not.

we only work from our home offices and co-working spaces and so on, we're not going to have a central office. So if we've made that decision that we're never gonna be in the same place, we might as well lean hard in the other direction and embrace the fully remote working, like the working style.

Lance

How did you find that technology's evolution over that dozen years that you were working together changed your ability to do that effectively?

John Li 

Uh, it's been amazing. Um, I think, um, early, early on there wasn't, there wasn't easy screen sharing. There weren't easy shared tools or anything. A lot of our conversations, you know, we, we did have to fly back and forth a lot and we didn't do some sprints in person. Um, and so just over time having things like digital white, you know, shared digital whiteboards and zoom and all like the improvement of video conferencing instead of just, um, what was it that?

Slack, uh, no, not Slack.

Lance

Oh, that Skype. Yeah.

John Li 

Skype, yes, exactly. And so just sort of moving through the evolution of remote communication had just made it easier and easier to be able to see who you're talking to, collaborate with them, work in shared spaces and share notes and documents and so on. It's been fantastic.

Lance

It's been interesting to see how COVID really took a lot of people who were more older generation had been sort of uncomfortable with virtual communications and wanted to really focus on companies that they could drive to talk to the person and finally pushing them over the edge to, okay, we now know how to operate virtually and that is almost as good.

John Li 

Yeah.

John Li 

Yeah, exactly. And I think it wasn't, you know, being remote during COVID wasn't a huge issue for myself and Justin, where we had been working together that way. I think as we started scaling the team during that time, there were definitely issues for some of the, some of the other team members who joined, who were used to working in a office first environment. But I think we've gotten, at this point now, we've gotten past all of that. And I think I'm really happy with where the team is.

Lance

So how did you overcome some of those challenges of building a fully remote team and onboarding culture, things like that?

John Li 

Yeah, we have a really big written culture in terms of our handbook. We had a great people's operations manager who helped us put together sort of like team handbooks and documents and sort of culture documents that were integral to every new team members onboarding. We have very regular team sort of socials, gatherings, team building events in a virtual environment.

to be able to give opportunity for team members to get to know each other in a non, I mean, in a less work-like environment, right? And so we, I think we lean pretty hard into those things to make sure that people are able to build connection beyond just the work items that they are working on.

Lance

And so I'm interested that you've got a large written culture. So what do you put in your written culture materials?

John Li 

Well, it's a lot of project management stuff. Like, I mean, I think, you know, in terms of the remote work and so on, like, we have a lot of, most of our work items are all written down and very async. So we try to have as few meetings as possible. So our team is, I think, pretty much every continent now and I think nine, nine or over like a dozen countries. And so...

Lance

Hmm.

John Li 

We're spread across all these different time zones. We have a narrow band of time where most of the meetings, most of our team meetings occur. Otherwise it's very much a async, you know, async work culture where people are allowed to work their own hours at their, you know, their own pace, their own hours. We have people who on the team who are digital nomads who travel around to different time zones. But basically as long as we're able to coordinate during those...

those key overlapping hours and everyone's able to do their work, then everything is great.

Lance

And how big is the team now?

John Li 

We're at about 17 right now.

Lance

Okay. That's an interesting size. I've always found that there's a real, there's a big phase transition when you go from just the founders and co-founders to starting to hire people. And then around about you hit 20 people, suddenly a perfectly flat organization kind of stops working. And are you starting to experience that? And how are you thinking?

John Li 

Mm-hmm.

Yeah, we've definitely experienced that. And I think we went through an up and down sort of a growth phase and we pulled back a little bit in terms of team size over the past couple of years. So we've gotten as big as low 20s and then pulled back a little bit with the economy and everything else. But what you mentioned is absolutely right, Lance, that I've definitely felt the phase shift between

small team, everyone in a room, everyone can work together on the same problems, um, to sort of a much, not a much larger team, we're still quite a small team, but it just, work doesn't operate the same way, right? Where it's like a perfectly flat organization that you have different teams and different, uh, different parts of an organization and, and layers of management and so on.

Lance

Yeah, you need to have some more organization around what gets done, who does what, who's responsible, as opposed to just leaning over or going into a Slack channel and hey.

John Li 

Yes.

John Li 

It's definitely been a phase transition for us as founders to go from working on everything and understanding every, you know, when it's just the founding team to from understanding everything that goes on in the business and the product to taking a step up and working with managers and work and not being involved in sort of the nitty gritty work as well. So that's been an interesting transition.

Lance

Right, yeah, because you're moving into the levels of abstraction in the business and, you know, yeah, giving away responsibilities as

John Li 

Yes, exactly.

Giving away the Legos is a key thing.

Lance

Right, it is utterly important and almost I think everyone finds it challenging to, especially when you've been doing it, you know exactly how this works and you give it to someone else and they do it differently and maybe to begin with not as well and you know, but you still need to let them do it or they'll never be able to take over from

John Li 

Yeah, exactly, exactly. And we keep, we tell ourselves that all the time, right? Different is not worse. Different could be better, you know, like you, we have to operate with a principle of trust and a principle of, you know, of assuming best intentions in terms of both work and conversation and so on. And so I think we're making progress on that. So excited for what's ahead.

Lance

Excellent. So are there any other sort of closing thoughts you'd like to share with the Feel the Boot audience?

John Li 

Um, I think that sure, um, test often and have patience. I think that, uh, I think that success, um, I think that success comes through repetition, uh, repetition of effort. And oftentimes I see a lot of other entrepreneurs, um, give up on their efforts before, before they get to that point where the, where their efforts take fruit and pay off.

Lance

I like that. Yeah. Because testing isn't a one-time thing. It's not like you could say, I have done testing. It's a perpetual thing. Excellent. Well, I think that's a great spot to end. Thanks so much. It's been a real pleasure.

John Li 

Thank you, Lance. Great chatting with you.