How To Ask Vendors About Automated Valuation Models
Selecting an AVM is an important decision. How can
you be sure you’re picking the product that’s right for you?
Here’s 10 questions to consider.
By George Livermore
It has been more than a decade since automated valuation models (AVM)
were introduced into the primary mortgage market, and only nominally longer
since they were developed as portfolio valuation tools in the post-origination
segment.
With growing data availability and greater distribution capability of
the Internet, new vendors now enter the AVM marketplace with their own
variations on established modeling themes. Some offer innovations; others
simply parrot what already exists.
Companies that have adopted AVM have discovered that the technology significantly
changes their approach to the mortgage business. It opens the door for
process change in a number of different parts of the organization, from
qualifying new lending opportunities to servicing and retaining customers.
Selecting an AVM is an important decision. How do you know what product
is right for you? Here are ten questions to ask AVM vendors as you evaluate
their technology.
What are your data sources?
AVM performance is data dependent. Regardless of the model’s method
for estimating value, it will perform better if the data is complete,
current and accurate.
The better-performing models use several data types:
• tax assessment data to identify the property’s location, assessed value and basic characteristics;
• publicly recorded sales data to provide price trend and comparable sales information; and
• data obtained through visual inspections of property, such as multiple listing service and appraisal records, to provide detailed property features and amenities.
The more robust the data, the better the models perform. While AVM can
be fueled by property data collected for legal research, direct marketing
or other purposes, they work best using information collected specifically
for real estate applications.
Real estate information companies pay particular attention to transactions
and data elements critical to AVM accuracy, such as the physical location
of the property and whether a home sales transaction was a full-market
value transfer.
AVM vendors should be willing to disclose the type of data used in their
model, the purpose for which the data was collected and, ideally, the
name of the company providing the data. This will help you evaluate
the efficacy of the model and its suitability for a particular purpose.
How does your model work?
If the answer to this question is vague, you should be suspicious. AVM methodology is well documented, with mainstream models using one of four methodologies:
Are there other methods besides the ones listed above? Yes, but they fall outside of the limits of universally acknowledged approaches. The point of this classification is not to describe in fine, technical detail how models work, but to provide a context for understanding how models approach the task of valuing a property.• price indexing methods, which arrive at a sales price on the basis of regional sales price trends;
• hedonic models, which adjust calculated values on the basis of physical property characteristics, such as square footage and room counts;
• adjusted tax assessed value models, which statistically adjust a property’s tax assessed value to derive a current market value; and
• hybrid models, which combine some or all of the above methodologies.
At a minimum, you should understand which method a model employs. Ideally, the AVM vendor can describe, using language familiar to you, exactly how the model selects and analyzes data to arrive at a value conclusion. Is the model property-specific or does it generalize about properties in a specific geography?
If the model employs multiple methods, what are the criteria for determining which methodology is eventually applied? The more you understand a model’s approach to valuation, the more enlightened you’ll be when you decide what to use and how to use it.
How long has your model been commercially available?
While you should not summarily discount emerging technology, models that have been in the market for at least three years have demonstrated applicability over the course of at least one real estate cycle. Products with a commercial track record have also likely been extensively tested for performance. A commercial track record implies they have demonstrated competency for some collateral valuation applications.
New, untested models may indeed offer breakthrough performance or functionality, but these models should be rigorously tested to ensure they satisfy your requirements and the representations of the AVM vendor.
What organizations have accepted the model?
Endorsements are few and far between in the AVM world, but users will occasionally allow themselves to be used as references in the market. You can usually pick up candid comments about AVM strengths and weaknesses by speaking with current users.
Referral organizations include end users, resellers of AVM products (who do some of the most extensive due diligence on products, since their brand is associated with the product) and independent organizations, such as ratings agencies.
If an AVM vendor offers you names of their users, contact as many as you can, and get their assessment of the good and the bad about the model and the vendor.
If a vendor cannot provide referrals, you may be one of the product’s pioneering customers. (This has its own set of rewards and punishment that merit a separate column.)
What quality-control methods do you use to monitor your model’s performance?
AVM performance can be inconsistent. Changes in data quality and availability, buyers’ home purchase criteria and short-term market conditions can alter a model’s usability. A vendor should be able to identify and explain changes in AVM performance before users notice it. This requires ongoing and systematic testing by the vendor to identify changes in hit rates and accuracy.
Systematic testing also assumes the vendor has the means to correct for a performance downturn by modifying the modeling methodology, correcting for data inaccuracies or choosing alternative data sources. Ask vendors how they make such corrections.
Can I test the model?
These five words can cause an AVM vendor’s heart to pause because it may mean the start of a protracted sales cycle with an element of competitive risk. But experienced AVM vendors welcome testing because it helps them understand how their models rate competitively.
Prospective buyers benefit because they have the opportunity to test drive the models, sometimes under the same conditions in which they will be used.
AVM can be subjected to various kinds of tests, two of which are common. The first is batch testing, where you provide the AVM vendor a file containing properties for which you have independently established the market value. The test involves the vendor estimating the value for as many properties as possible. You can evaluate the model’s performance on the basis of accuracy and hit rate.
The second is on-site testing, where the model is used in your business environment for a limited period of time. This gives you the opportunity to put the model through its paces in whatever simulated or real-world tests you designate. At a minimum, an AVM vendor should be able to support both kinds of tests.
If an AVM vendor declines to participate in a legitimate, well-defined test, it is likely due to a lack of resources to support a test or a lack of confidence in the model’s performance, or both. In this case, you may have saved yourself time evaluating a product not well suited for your needs.
How well does your model’s geographic coverage and performance match my business needs?
Five years ago, geographic coverage was a major inhibitor to AVM adoption. Users traded between model accuracy and geographic coverage because of the limitations in real estate data. Today, that gap has narrowed as real estate information providers have improved the coverage, quality and completeness of data used in AVMs. But the gap hasn’t completely closed.
AVMs can deliver 100% coverage for applications such as portfolio valuation and mortgage prescreening. Depending on the model and your policy governing its use, effective coverage can be lower for loan underwriting and appraisal process improvement.
Geographic coverage claims provide a starting point for understanding how much of your business can involve AVMs. Representations by the vendor of their model’s suitability for certain uses, supported by your independent testing, are important to helping you understand its effective coverage of your business.
Can you help me integrate the models into my business applications?
AVMs provide time and cost savings in collateral valuation by streamlining data research and analysis and providing fast and consistent answers. More benefit is created if the AVM is closely integrated into business processes, with the results of a valuation session streamed directly into an application or process that incorporates the result.
To support a program like this, an AVM vendor must have the technical resources to provide the programming interface and adapt the technology to the customer’s platform. This includes the ability to provide detailed technical specifications and instructions for your technical team to install and configure the AVM correctly and quickly.
To support AVM integration, the vendor must be willing to begin to open the model to customer scrutiny. Understandably, some companies are loath to cooperate if they think the methodology and services are easily replicable or if they harbor high anxiety over their intellectual property rights. The latter can be addressed with an appropriate licensing agreement.
How does your firm support the model?
At a minimum, your AVM vendor should be able to offer you product training and technical support to ensure the product performs to your specifications.
While in the past five years AVM products have become more “plug and play” and their interfaces more intuitive, the vendor should take steps to ensure the product operates in your environment and you are trained in how to get the most out of the application.
Experienced AVM vendors can offer additional training in best uses and practices, and can advise you of the importance of complying with state and federal regulations governing appraisal and AVM use.
If your vendor offers training, we encourage you to participate. This will improve your knowledge of the product’s capabilities and help you discover new product uses in your organization. If training is not available, you should speak with current users referred by the vendor to understand the steepness of the learning curve and any surprises they encountered.
What is your firm’s long-term viability?
Decisions about which AVM product to use should include information about the vendor’s prospects for survival. AVM products can influence – or be influenced by – risk management policy, and your final selection should be supported by a thorough analysis.
Unless there is a compelling performance benefit or economic advantage to be gained, you do not want to switch vendors or products often. Having such a decision forced on you by a vendor unable to continue to support the product is disruptive and embarrassing.
With that said, I do not want to discourage new market entrants, nor do I want to discourage you from evaluating new providers. I merely recommend you perform due diligence on their business prospects.
You should inquire about the vendor’s
Finally, I have seen AVM products evaluated using a variety of criteria, from accuracy to hit rate to cost, and even on the basis of how easy it is to interpret the results.• data relationships (have they secured long-term access?),
• business financing (do they have the capital to invest in ongoing R&D?), and
• operating performance (do they have ongoing revenues, or are they operating on investment capital?).
In my experience, the best purchase decisions, defined by customer satisfaction, result when buyers decide in advance the criteria most important to them, then select the product that best meets that criteria. It is easy to become sidetracked by the promise of new technology, but in the end, it is usually reliability and predictable performance that satisfies the AVM user.
George Livermore is president of First American Real Estate Solutions, a real estate information and technology business within The First American Corp., Anaheim Hills, Calif. He has been involved in all aspects of valuation modeling since it was first introduced, from supporting developers with data for AVM design to the purchase and creation of valuation models for commercial distribution.
This article was previously published in the May 2001 Issue of Secondary Marketing Executive
