[Metric Change] increased click on a search result

10/20(수) Lauren과의 인터뷰연습

Q1. You notice that the number of users that clicked on a search result about a FB event increased 10% week-over-week. How would you investigate? How do you decide if this is a good thing or a bad thing?

Q2. You notice that the percentage of users that clicked on a search result about a Facebook Event increased 10% week-over-week. How would you investigate?

<피드백> delve into with this 2 information -> with these 2 pieces of information event holders (홀더 발음할때 l의 경우 혀를 내밀며) that will affect the search / that affects the search

related material: https://towardsdatascience.com/mastering-facebooks-metric-change-data-science-interview-question-part-2-2-8bf61df63828

Mastering Facebooks Metric Change Data Science Interview Question Part 2/2

1. Clarify: assume nothing, if something is not absolutely obvious, then ASK.

Q. What is the search result: A search result refers to when a user searches for something in the search bar on Facebook? And these search results could be different categories, like a Facebook Event, Page, or Group?

Q. When we look at the percentage of users that clicked on a search result about a Facebook Event, do we mean: of the users that made a search and were exposed to a list of search results, the fraction of those that clicked on a result around a Facebook Event (instead of a group, page, etc)? So, if 100 users searched for a term and 50 of them clicked on a result around Facebook Events the percentage of users would be 50%?

Q. And a 10% week-over-week increase in the percentage of users means that if last week 100 users searched for a term and 50 of them clicked on a result around Facebook Events, this week 200 users could have searched for a term but 110 clicked on a result around Facebook events? So the percent of users is now 55? (i.e., a 10% increase from last week)?

Q. Finally given this, our goal is to try to figure out what might have caused this increase?

[TIP] It's important to not only clarify the question but also reiterate the goal.

2. Hypothesize

Before I dive into investigating the source of the spike, I want to outline a few high-level hypotheses. Broadly speaking, the cause could be internal or external. 

Potential internal causes might include a recent update that we made to our ranking algorithm to show Events at the top of our search result list. Or, there may even be a bug in our logging, meaning we are not correctly capturing our click count and so there may not be an actual increase in event-related clicks.

Potential external causes might include major events like festival season causing users to show a greater interest in event-related search results. 

To test these high-level hypotheses, and uncover other potential causes, I'd like to walk through the data that I would explore. Does this sound reasonable? 

[TIP] Use the word hypotheses. Use it like it's your job. Because if you nail this interview, it really will be.

Continuously ask for feedback. This is a conversation, not a monologue. If you don't stop periodically and engage the interviewer, you won't know if you're heading in the right direction or if there is something you may have missed and you risk losing the attention and interest of the interviewer.

3. Diagnose

The first data point that I want to consider is the role of time. I would check to see if

1) This 10% week over week increase really means that there is a growing interest in events. What has been the week-over-week change over the last few weeks (i.e. how have we been trending)? If the week before we saw a 20% increase, this weeks number may actually indicate a decline in interest.

2) I would also want to check if the increase is evenly distributed throughout the week, or if it is the consequence of lots of clicks on just a single day. If it's the latter, this might indicate a bug in our logging systems or a big event that happened on the day in question.

3) Does seasonality play a role? What happened this time last year? For example, if school is out and students are going into summer vacation mode, then we may actually see this rising interest in events from students that are looking for things to do. In which case, we would most likely have seen a similar spike this time last year.

 

Well, the next thing I would like to see if how this 10% increase breaks down:

1) Is this change concentrated in a specific region or is it evenly distributed globally? For example, we are slowly coming out of the pandemic and some cities have started to reopen. In which case, the rising interest in events may only be concentrated in those cities.

2) Is there a similar spike in any of our other products or offerings? If an interest in events is going up, do we see a similar jump in Instagram or Facebook stories because users attending this event will have more content to post about?

3) Is this increase evenly distributed across mobile versus desktop? This would be particularly useful to know if we believe there is a bug in our logging or if we made a change to our ranking system. So, if we see that the change is disproportionately skewed towards one of macOS, Windows, iOS or Android, then it may give credence to our initial hypothesis that the change is driven by internal causes.

4) How are other search result categories like Groups and Pages affected? If the percentage of users clicking on events is going up, then we are clearly cannibalizing engagement from these other categories. Is there a specific category that we're cannibalizing from or is it evenly distributed? For instance, is it only users that previously clicked on Groups (not Pages) that are clicking on Event snow? This may indicate that we made a change to the ranking of Groups in our search results. Did we downrank it? Or accidentally remove it completely?

 

A few other segments I would like to look at are how this metric varies by:

event type: is the increased interest focused on a certain type of event? like music? In which case it might offer evidence for the earlier hypothesis around festival season.

user demographics: is the increased interest coming primarily from younger users in school (offering evidence for the earlier hypothesis around summer break for students)?

 

* Lets say we made an update to our ranking algorithm as you suggested earlier and are now down-ranking groups in favor of events.

[TIP]

• Notice how I not only say what data I want to look at but why I think it will be useful. Bridging this gap is key. Imagine if you just regurgitate the framework without thinking about how it uniquely applies to the question being asked. Then, the second the interviewer asks you ‘hmm, that’s interesting, why would you look at the break out between mobile and desktop…’, and you can’t answer him, you may as well consider the interview over (just kidding, kind of).
• I didn’t bring up the effects of Industry & Competitors even though its in the framework because I didn’t think it would apply in this scenario. The framework is a guide, not a prescription. It’s up to you to determine what’s relevant. How do you do that? Practice (I know).

4. Solve

Interesting. It seems like we made an intentional decision to add greater visibility to events and the 10% increase is a consequence of that. Why did we make this change?

[TIP]

• If the interviewer doesn’t end the interview even if you think you’ve ‘answered’ the question, then dig deeper. Fair warning: they will most likely turn your questions back to you.
• The five whys is a really good reference for what digging deeper looks like.

My hypothesis is that we changed our ranking algorithm because we wanted to encourage people to go to events. So I think the next two things we want to consider are:

1. Were we successful? Just because people are clicking on events in the search results, does that mean they are attending them?
2. If we were successful, given that we are cannibalizing from other products like Groups and Pages, is this change good or not.

[TIP]

In this framework, I walked through possible next steps to take once you’ve diagnosed the reason for a metric may have changed. And though I write ‘change’, most of them pertain to a metric drop. But if the metric actually increases, then what do you do? 

The best way to do this is to tie these goals to the original mission statement and ask ‘are the goals in line with the company’s original mission’?

 

=====================================================================

We hypothesized that Facebook intentionally changed the ranking algorithm to encourage more users to attend events. This left us with two questions that we will address today:

1. Just because the number of people that clicked on events increased, does that mean the number of people that actually attended these events also increased?
2. If more people are attending events, is that a good thing? Especially if it’s at the expense of their interaction with other Facebook products like engaging in Groups? Because time is finite, if people are doing more of one thing, they're doing less of another.

1. How can we tell if more people are actually going to events?

Consider the user journey above as well as your own experience with Facebook events. Given the two, we can make the assumption that the more engaged a user is with an event, (i.e. the further down the ‘Yes’ rabbit hole they went in the user journey above) the more likely they would be to attend that event.

 

Before the Event

1. Did the user click Interested, Going, Share, or Add to Calendar on the event page?
2. Did the user post on the event page?
3. Did the user interact with notifications related to the event?
4. Did the user buy tickets to the event?

During the Event

1. At the time of the event, do we see the user in the same location as where the event is supposed to take place (assuming we have access to their location data)?
2. At the time of the event, is the user going to the event page often? For instance, to check the event address and other details.
3. Is the user posting a lot of stories during the time of the event?

After the Event

1. Did the user post photos and tag the location the event was hosted in?
2. Did the user post on the event page? For instance, to thank the event hosts.
3. Did the event page post photos and tag the user?

A complicated (and more precise approach) would be to use a weighted combination of the answers to some of these questions. But for the sake of simplicity, let’s assume that a user clicking Interested on an event page is the closest proxy for whether they attended that event. We can compare the number of users that clicked Interested on an event before and after the algorithm change. If the latter is greater than the former, it may indicate that the number of users attending an event increased as a result of the algorithm change.

[TIP] Keep it simple. Interviewers want to that know you can dive deep if you had to. And that is the purpose of the follow up questions that they will inevitably ask you. But until they prod you, keep it simple and succinct.

 

2. If more people are going to events, is that a good thing?

 

Facebook’s mission is to give people the power to build community and bring the world closer together. But this is pretty vague. If more people are attending events, does that mean that more of the world is coming closer together? Maybe. Are events more effective in bringing people closer than other features like Groups? Again, maybe.

 

Therefore, to determine if more users going to events is a good thing we need (1) a mission and (2) a metric.

The Mission. Giving people the power to build communities and bring the world closer together.

The Metric. Let’s take user retention, the number of users returning to Facebook.

The Upshot. If users attending events are coming back to Facebook more often than users not attending events, then the change is good. If retention drops, i.e. users attending events are not coming back to Facebook, then the change is bad.

💡 TIP: You can choose any metric. For this example, I chose user retention. But as long as you can justify the decision behind your choice, there is no ‘right’ or ‘wrong’ answer.

If the change is good, this indicates that user retention is correlated with event attendance. In this case, we may want to consider other ways to influence users to attend events.

If the change is bad, this indicates that event attendance appears to disincentivize users from returning to Facebook. In this case, we may want to re-evaluate the decision to change the search result ranking algorithm.

[TIP] In practice, it’s more complicated than the three line answers above. But in the context of the interview, interviewers usually just want to know that you’re able to show initiative and call out next steps. Remember, the best way to end an interview answer is to (1) summarize whats been said and (2) suggest next steps.