Technology

Govind Sharma

January 10, 2022

We all have heard the saying: â€śChange is the only constant,â€ť which translates to the fact that everything changes, every time. While that is too philosophical, and since marketers â€“ more so data-driven B2B marketers â€“ are not really fans of philosophy, there is a need to â€śmeasureâ€ť change. And not just that, a business needs to â€śattributeâ€ť, â€śrankâ€ť, â€śpredictâ€ť, â€śexplainâ€ť, and hopefully even â€śbring about" change using data analytics. Since statistics is the mother of all real-world measurements, and probability is the grandmother thereof, the science of measuring change too lies somewhere close by.

â€Ť

Since surface (i.e., not deeply thought-out) definitions are easy to make but difficult to follow with real-world data, the real challenges are solved by taking support from the well-researched areas of data science and probability. One extreme is to discuss the topic of change with its extreme roots, but that could easily fill-up a book (which is under way), but we stick to measuring change w.r.t. marketing analytics.

â€Ť

â€Ť

The main questions from a marketing perspective are: â€ś**What main factors changed from last week to this?**â€ť, and â€ś**By how much did these main factors change?**â€ť. But before that, let us understand something even more basic.

â€Ť

â€Ť

Marketers take various targeting decisions on a daily basis â€“ which audience to target, how frequently, and with what kind of marketing campaigns. They also have to keep track of various prospective customers, their journey, their overall statistics, and most importantly, reasons why a particular technique worked (or didnâ€™t work) â€“ be it a campaign or a strategy. And there are various goals when it comes to marketing. While some techniques could be motivated towards increasing reach (where itâ€™s important to maximize the number of eyeballs a webpage could get), others increase conversions (at various points in a funnel â€“ form-fills, email cold/warm calls, prospects, pre-sales, sales, etc.).

â€Ť

The foregoing marketing goals are achieved by measuring both static marketing performances (i.e., what happened today / this week), and also more dynamic, time-aware ones. While analyzing performance over time is a comprehensive view of marketing goalsâ€™ achievements / shortcomings and hence makes it cumbersome to digest multiple metrics of interest in a single frame, **summaries** of the same are preferred. And one of the summaries of a dynamic measurement of marketing performance comes in the form of **change** â€“ that is, what changed, by how much, and why. In this article, we would focus on measuring **overall** changes, and would dig deeper into measuring the causes for the same in a follow-up blog.

â€Ť

â€Ť

Based on the business requirements, marketers focus on tracking and eventually improving relevant KPIs (key performance indicators). **For the scope of this article**, we would explain change analysis taking the example of one important metric marketers are concerned about â€“ the *number of** **leads** they generate every week*. They achieve this by driving relevant visitors to their website every week. Any change in the number of leads that they get as compared to what they expected calls for an investigation on the reasons for the difference between **expected** and **actual** metrics.

â€Ť

Since the first step into any such investigation is to measure and compare (with last week) some global performance indicators measuring the reasons for change, the same is the focus of the current article. Hence, keeping a webpage in mind, we take the example of measuring the number of visitors (those who reached the website), the number of leads (visitors who reached target), and the conversion rate therefrom (leads per visitor).

â€Ť

â€Ť

â€Ť

When we perform these measurements for two given periods of time (say consecutive weeks), we could compare them. Let V1, L1, and C1 represent the number of visitors, number of leads, and conversion rate of week 1 respectively, and let V2, L2, and C2 represent the same measurements from week 2.Â

â€Ť

â€Ť

The four simple change measurements one could perform are the following:

**Total change**: This is the most basic measurement of interest that preserves both the â€śunitâ€ť and the â€śsignâ€ť of change. For example, if the number of website visitors who filled out a form changed from 50 last week to 75 this week, we get an absolute change value as**+25 form-fillers**. In short, it answers the question: â€śHow much more/less?â€ť.

More formally though, one could measure absolute change in visitors (đťš«V = V2-V1), leads (đťš«L = L2-L1), and conversion rates (đťš«C = C2-C1).**Relative change**: While absolute change remembers the unit of the entity one is measuring, it is â€“ more often than not â€“ more convenient to adopt a normalized change score. Taking the same example as before, change in the number of form fillers from 50 to 75 means a change value of +25, but the same is true if form-fillers had increased from 150 to 175 (viz., +25). What separates the two cases is relative change (i.e., how much did the unit change per unit original), which is**+0.5**(=25/50) for the former (50â†’75), and**+0.17**(=25/150) for the latter (150â†’175). The specialty of this score is that it remains a fraction between -1 and +1, and helps in comparing two â€śchangesâ€ť. In a day-to-day language, a â€śpercentageâ€ť variant of this metric is used by marketers.

Again, a formal representation of relative change in visitors (đťš«rV = đťš«V / V1), leads (đťš«rL = đťš«L / L1), and conversion rates (đťš«rC = đťš«C / C1) would also help engrave the idea.**Percentage change**: As described earlier, itâ€™s easier to understand when one says â€śthe number of visitors saw a 50% increaseâ€ť (as opposed to saying the relative change of visitors was +0.5). Therefore, as a human-friendly change metric, the percentage variant is more popular than its math-friendly counterpart (relative change).

A notable caveat is as follows. â€śFrom X to zeroâ€ť would mean a â€ś100% decreaseâ€ť (and vice-versa), but â€śfrom zero to Xâ€ť turns out to be an â€śinfinite% increaseâ€ť, which is absurd. One workaround to rectify this is to call â€śfrom zero to Xâ€ť as â€śfrom min to Xâ€ť, where min could be set based on the metric of interest (e.g., min could be just 1). Another workaround is to call â€śfrom zero to Xâ€ť as a â€ś100% increaseâ€ť. Another interesting point is that â€śno changeâ€ť, itâ€™s called a â€ś0% changeâ€ť â€“ even if it is â€śfrom zero to zeroâ€ť.

From the perspective of our current example, percentage change in visitors (đťš«pV = 100 x đťš«rV), leads (đťš«pL = 100 x đťš«rL), and conversion rates (đťš«pC = 100 x đťš«rC) could be computed by simply multiplying the relative change by 100.**Factor change**: Some marketers (and sometimes marketers) like to express change in percentage differences, and sometimes in factor increments/decrements (and some do both), and this is purely a personal/company-wide choice. Picking the same example from above, whether it is convenient to say â€śthe visitors increased by 50%â€ť or â€śthis weekâ€™s visitors are 1.5x last weekâ€™sâ€ť differs from use-case to use-case, but is only a different way of expressing relative change.

Although one has to be cautious, however.Â

- â€ś1xâ€ť means no change (a â€ś0% changeâ€ť). For example 100â†’100, 1â†’1, 0â†’0, etc.
- â€ś0xâ€ť means a â€ś100% decreaseâ€ť, from, say 100 to 0, 5 to 0, but not from 0 to 0 (since we choose to call it a â€ś1xâ€ť change.
- â€ś2xâ€ť means a â€ś100% increaseâ€ť.
- â€ś1.5xâ€ť means a â€ś50% increaseâ€ť.
- In general, â€śkxâ€ť means a â€ś100*(k-1)% increase/decreaseâ€ť, where itâ€™s an increase when k > 1 and a decrease when k < 1.
- When saying â€śkxâ€ť, k never goes negative.

â€Ť

Going by the earlier example of website visitors, we could be interested in the factor change in number of visitors, (đťš«fV = V2 / V1), leads (đťš«fL = L2 / L1), and conversion rates (đťš«fC = C2 / C1).

â€Ť

â€Ť

Depending upon the business and the audience, there are multiple combinations of possibilities. We only cover some of them, summarizing overall (global) change. And as mentioned earlier, our next article on this topic would dive deeper into digging up the â€śreasons that drive this overall change.â€ť

â€Ť

Let us start with a simple world, and slowly drift towards complex scenarios. Suppose our website had 1,000 unique visitors last week, and 1,000 new unique visitors this week (i.e., there was no change in the number of visitors). Of the 1,000 users last week, 20 signed-up for our newsletter, and the same trend continued this week as well. In other words, both last weekâ€™s and this weekâ€™s conversion rate was 2% (20*100/1,000). What does this tell us about this weekâ€™s performance over last? That it remained the same! In other words, there was a 0% increase/decrease in both reach and lead conversion rate.

â€Ť

â€Ť

Now, if we had more visitors (say 1,500) this week as compared to the last, with a proportionate increase in leads so as to maintain the 2% conversion rate, it would amount to a 50% increase in visitors and leads, but a 0% change in conversion rate.

â€Ť

â€Ť

On the other hand, if the total number of visitors had remained the same, and leads would have increased by 50%, this would increase the conversion rate by 50%.

â€Ť

â€Ť

If, however, the number of leads would have remained the same despite an increase in visitors (by, say, 50%), we would see a 33% fall in conversion rate.

â€Ť

â€Ť

It is also possible that the number of leads increased by a disproportionate amount, which led to an increase in conversion rate.

â€Ť

â€Ť

â€Ť

We just discussed how measuring **overall** change is straightforward: simply report the signed/relative/percentage difference or factor change in visits, leads, and conversion ratio between the two weeks. But this is only half battle won. What is ideal is to measure the â€ścausesâ€ť for the change. For example, we know that a 2x increase in visitors (V2 = 2V1) and 1.5x increase in leads (L2 = 1.5L1) â€“ and hence a 25% drop in conversion rate (C2 = L2/V2 = 0.75L1/V1 = 0.75C1) â€“ happened. But **why it happened** is one of the most important questions change science has to answer.

â€Ť

â€Ť

A seasoned marketer can quickly understand the main factors that led to a drop or a rise in scale (#visitors) or conversion (leads/visitor). But where data analysis comes in is in short-listing such factors from the rest, and hence help the marketer with her weekly (or periodic) decisions. In this article we only give an intuitive idea of what causes change (and how we measure it). In **part 2** of this series on Change Science, we discuss the exact procedures and methods to measure factors that cause change periodically.

â€Ť

The more we know about our customers, the more our analysis benefits. While measuring change, as it was mentioned above, we usually track the number of visitors (V), the number of leads (L), and the respective conversion ratio (C). Along with mere counts, one ought to measure the **profiles** of such visitors and leads â€“ in both weeks. For example, with a 2x and 1.5x raises in visitors and leads, to know what caused it, one has to track how the properties of visitors and leads have changed. The properties we are referring to are usual user/event based properties â€“ from simple demographic ones such as Location (country, city, etc.), User Agent (browser, OS, etc.), etc. to marketing-oriented ones such as Referrer (domain, campaign, etc.) â€“ along with their values (a â€śproperty=valueâ€ť combination looks like â€ścountry=Indiaâ€ť, for example). In summary, in addition to tracking overall statistics (visitor and lead counts), we also track counts â€śgrouped-byâ€ť their properties.

â€Ť

Needless to mention, such properties are too many to count, and property=value combinations are even more. This is where Factors.AI comes in. We track-down each factor, measure the change attributed to the factor, and surface top insights. And we repeat the process for any event pair combination (in say, a funnel), for every property, and for every consecutive period of interest. We urge you to follow-up with us in the next article that describes how exactly we surface the main change factors via our very own Weekly Insights feature.

â€Ť

Stay tuned.

â€Ť

Get the latest best practices in Marketing Analytics

delivered to your inbox. You don't want to miss this!!