DemandTec: How can you make complex rule permutations understandable?
DemandTec was a big data / data science pioneer in using algorithms to set retail and promotional prices in stores. They modeled the price demand curves present in sales data across a grocery or drugstore chain in order to set optimal pricing for a goal of share or profit. Models have a habit of doing quirky things, necessitating a rule system to control errant behaviors. As with most roles, I entered DemandTec as sole designer/researcher, practicing what would be called Lean UX.
This is an older project, but it illustrates my concerns and a lean approach very well. Sometimes good usability solutions are timeless.
I developed an understanding and empathy for users at DemandTec by close observation and open-ended interviews of the in-house users, in conjunction with contextual inquiry and open-ended interviews with end-customer users, typically price managers at grocery chains. DemandTec’s internal services organization was the primary user of rules.
DemandTec ran on a blended customer/services/data scientist model. I sat near professional services and spoke, listened, observed and tried ideas with them constantly and informally.
Most services people came from a general business background, but not a grocery specific background; all exhibited fluency in excel. They didn’t necessarily understand the math or science behind DemandTec’s price optimizer, but they could evaluate and discuss results with clients. They tended to work longish hours and did travel reasonably frequently. They all worked off of laptops with what today we might consider small screens. To a person, they existed in excel, save for setting up optimizations, and importing data.
Professional services used rules, actually constraints, as a way to limit the model’s price optimization to keep certain conventions and relationships viewed as essential: Don’t raise prices too fast, make sure larger sizes offer a better deal, respect the psychology of $.99 or $.88.
Rule systems are confusing
In the old desktop app, and the first betas of the web app, rules looked a little like this:
The inflexibility of a form input makes rules confusing. I observed many consultants confused when the setting a decrease limit in the fixed structure of the rule:
The [average price] must be [<=] Lower limit: [ ] and Upper Limit [ -1][dollars\/]
which really means: “Make sure the price drops by at least $1”
And I remember getting asked what rules meant. We did try some reorganization of the forms (mockups since lost) but the problem still remained, specifically with price decreases. Experienced users kept making the same mistakes, a big sign of a usability issue.
One other problem cropped up, a very common one when you use a set of inputs with rules that may have odd restrictions: rule traps. With the price rules, if a user chose [volume weighted average price] instead of [price] as a metric, and then set limits in $, they got an error, actually a couple of errors that required slowly backing out of a set of changes in order to make the desired switch. It really penalized people for experimenting, and that’s not the way to encourage people to learn or adopt anything.
Services might often ask for tactical fixes to the forms structure,which wouldn’t really address the problem: Sometimes, only an outside observer sees a different way.
Initial prototype: making rules readable.
I observed this firsthand, but it was a late night call that spurred me to build an engineering prototype of a natural language rules engine. A confused consultant about to run a 3 day optimization called me to ask if the rules they set would do what they wanted.
"For 'Packaged Cereal' in 'San Francsico Bay Area,' the Price must drop by 2%" (0,-2%)
they might click on ‘must drop’ and select from a pick-list ‘cannot drop’ to create a lower limit rule:
"For 'Packaged Cereal' in 'San Francsico Bay Area,' the Price cannot drop by more than 2%" (-2%,0)
should they then click on the “2%” in ‘more than 2%,’ a box appeared showing the lower limit(-2), and units (relative %, relative$, absolute$). Changing the limit to (+2 relative %) rewrites the rule:
"For 'Packaged Cereal' in 'San Francsico Bay Area,' the Price must rise by at least 2%" (2%, 0)
I went from desk to desk with the first version to our services people and let them try it out. Luckily at the time the response was overwhelmingly positive: There was no PM support for any feature that might push back the timeline.
Services people provided a lot of feedback, in two veins: A big one was tuning the language and control. The second came in the form of how to avoid those rule traps. Most feedback was informal.
The second prototype: Allowing forgiveness and experimentation.
The second major evolution of the mockup took an additional couple weeks. I wrote a system to look ahead thru permutations whenever a user made a choice that boxed them in. Instead of limiting their action this look-ahead provided two choices. Users could make the change they wanted and the rule engine will help make it work, or they could revert to the last good state. This allowed services people to fluidly go thru permutations and try things.
Later Refinement: Getting the language right
After the second mockup, engineering formally adopted and professionalized the code set. We set about working with services to test and develop the right language over the next couple of months, and did a localization proof of concept into french.
I never received a late night call again asking what a rule would do. The company closed a large contract, partly on a demo for the rule system, which provided a compelling and clear answer as to how we control the black box.
DemandTec applied for a patent on the underlying engine I built.