Death by a thousand paper cuts February 25, 2007
Every once in a while a realtor or broker from out of state will ask me to develop an IDX web site for them. Unfortunately, supporting a new MLS is very similar to supporting a foreign language. It is a large software engineering task that takes a lot of time, and since I don’t already have the code written and don’t already have access to their MLS’s feed, I inform them that time is money and the conversation usually ends there. Someday, that may not be the case, but I’d rather be small & profitable than large & broke.
The problem is made worse by the fact that many Realtors don’t know what format or protocol their MLS uses for data downloads or even who to contact in their MLS to get a feed for an IDX vendor. If you ever want to change IDX vendors, hire a software engineer or are crazy enough to do it yourself, you should know this. Knowing how your MLS distributes your listing data is like knowing how to change the oil in your car or how to defragment your hard drive. You don’t have to know, but it’s good to know. It may seem like I’m ranting about some MLS techie mumbo jumbo thing again, but it is preventing the industry from taking advantage of the low cost IT innovations that could be. I don’t think folks fully appreciate the challenges that an IDX vendor faces and how those challenges are retarding the industry’s growth and health.
For example, the NWMLS (Northwest Multiple Listing Service - serves mainly Seattle, WA and western Washington) uses software from Rapattoni. It provides listing data via a proprietary SOAP interface and all the photos are accessible via an FTP server. Listing data is updated constantly (a new listing usually appears in our feeds about 15-20 minutes after it’s been entered into NWMLS by a member as I understand it).
By contrast, EBRD (East Bay Regional Data - serves mainly Oakland, CA and the east bay area) uses Paragon by Fidelity MLS Systems provides it’s listing data via nightly updated CSV text files, down-loadable by FTP. The new and updated listings images are accessible via ZIPed files via FTP. The photos for active listings which haven’t been recently added or changed are not available (unless you bug the IT dept).
The only way they could make their systems more different is if the EBRD encoded their listings in EBCDIC! In order to support both, I need to develop 2 very different programs for downloading the listing data onto my server, importing the listing data in my database, dealing with differences in the listing schema (for example, the EBRD doesn’t contain a “Number of Photos” field or a “Community Name” field), dealing with differences in the photo location downloading (the NWMLS stores all photos in an uncompressed format in one of a thousand sub directories while the EBRD just stores the fresh photos in one big zip file). So I can spend my limited time improving my software for new markets (that have no customers) or improving my software for my home market (which has paying customers). Unfortunately, given the current market realities I can only afford to support my home market at this time since MLS IDX programs can be very different and there is no place like home (so far as I know anyway).
I keep waiting for RETS to save me from this madness, but until it happens in Seattle or the East Bay, I’m not holding my breath. After all, if two of the larger MLSes in the country in the two most tech savy areas of the nation don’t support it yet, I interpret it to be a vote of no confidence. I suppose, RETS could be going great guns in the rest of the country, but if it was, I’d expect the NWMLS & EBRD to be all over it, like the establishment on Redfin.
The Center for REALTOR® Technology Web Log, paints a rosy a picture regarding RETS deployment in the industry. Unfortunately, according to Clareity Consulting, an IT consulting firm that serves MLSes and other parts of the real estate eco-system, RETS is the NAR’s unfunded mandate. Although, everybody wants the benefits of RETS, nobody is willing to pay for it. Furthermore, it appears back in days before I got sucked into real estate technology, there was an effort to promote the DxM standard and that went nowhere (which is a bad omen). What’s worse is that they keep moving the goal posts. We don’t even have widespead RETS 1.0 support, and they’ve already depreciated that standard going full bore on RETS Lite and RETS 2.0. It seems the biggest problem is one of vision and scope. They keeping adding more features to cover more scenarios, when we don’t even have wide deployment of the existing standard (assuming that we had standards to begin with at all). It reminds of the recent software industry debacle that is known as “Longhorn reset“. The problem is that RETS is just too complicated, in an environment with too many legacy systems in place, too few resources to support it, and excessive aspirations. The idea of RETS is great, it’s the implementation and deployment that’s disappointing and at least Microsoft pulled Vista out if it’s death spiral…
The sad thing is that computer industry already has great tools for moving data around over the Internet in efficient and well supported (if sometimes proprietary ways). They allow you to query, slice, and dice your data in a near infinite number of ways. They’re called database servers. They are made by multiple software vendors and there are even some excellent open source ones out there. They let you set permissions on what accounts can see what tables or views (gee, sounds like something an MLS would want). The better ones, even have this level of security to the field level. Even better, most of these so called database servers have the ability of exporting data into spreadsheets, reporting tools, and even GIS systems. All of them provide a well defined and often times well implemented API that software developers can use and exploit to implement what hasn’t been invented yet!
Why doesn’t the NAR & the MLSes save us all the trouble, standardize on a few good database platforms (I’m a fan of MS SQL Server and MySQL, but I’d settle for anything that has ODBC, .net & Java support at this point), and provide everybody RDBMS accounts? It’d lower the cost for us IDX vendors (less code to write, since everything is just SQL), it’d lower the costs for MLS vendors (since data access, security, programmability, and scalability is now the RDBMS vendor’s problem), provide more choices for agents and brokers (since getting Excel talking to MS SQL Server is a cakewalk compared to RETS) and it will lower IT costs for the MLS (because the MLS vendors don’t need to invent an industry specific solution to a problem that’s been largely solved already and I’m betting that the MLS vendors already use somebody else’s RDBMS to implement their solutions anyway). Granted, a SQL Server won’t enable all the scenarios that RETS wants to enable (if RETS was ever well implemented and widely deployed enough for that happen). However, I’m of the belief that it’s not going to happen until after Trulia or Google Base becomes the de facto nationwide MLS by providing a single schema with a simple REST like web services interface.
So, what does your MLS do to support IDX vendors? Do they provide all the data all the time, or just daily updates? Have they deployed RETS yet? Are they going to? Who is their MLS software vendor or do they have a home gown solution? What do you want to do, that you can’t do today because the data is in a format that you can’t use easily? Would you be willing to pay more in membership dues for better software or better service from your MLS? Are we at the dawning of the RETS revolution, or is it too little, too late?
PS - Anybody, know anybody from an MLS / IDX dept or MLS vendor that blogs? I’d love to know what things are really like on their side of the listing data fence.
Check out these related posts:
- Where should the MLS end and the IDX begin?
- NWMLS to take 3 more baby steps into the digital age
- There you go again - the MLS doesn’t scale
Article Tags>> Clareity | data | deployment | download | FTP | google-base | IDX | Microsoft | MLS | MySQL | NAR | Paragon | Rapattoni | REST | RETS | SOAP | SQL-Server | standard | Technology | Trulia
- Posted in : General Real Estate
- Author : Robbie
Comments»
RETS has come a long way from the days when we (Clareity) wrote that white paper, though as your own frustration shows, there’s still room for increased adoption and standards improvement. That’s really all the white paper was about - urging people to put resources into RETS so that it would be successful. 2006 saw some wonderful new RETS servers released by MLS vendors - servers that provide a lot of easy to use control over data access authorization and control over which fields and listing statuses get exposed for specific RETS users. Almost all of the major vendors have made RETS servers available to their customers at this point, so I’m hopeful that your situation will improve.
Hey Robbie,
I have to agree with Matt the RETS systems have come along way. The large adoption of RETS has made many things a lot easier for us. You just happen to be in 2 systems that have chosen not to distribute using RETS. Both Rapattoni and Paragon have RETS available but its the MLS that has chosen not to take advantage of it. The NWMLS uses its own proprietary app not the one provided by Rapattoni.
Yeah, since the NWMLS & EBRD have decided not to deploy RETS (and they are the only 2 largish MLSes I’ve been exposed too), I really have no idea how widely deployed RETS is, how stable the standard is, or really how difficult it is to support. Unfortunately, since my local MLS hasn’t deployed it, I don’t really have a server to develop or test an implementation against. I wonder what’s blocking their respective deployments?
I wish there was a scorecard, an MLS conversion guide, better RETS / SDK documentation, perhaps a Clareity blog or something that let me what the state of the industry is w/r/t to standard support. I want to embrace RETS, but due to the luck of the draw, I can’t. A prosepctive IDX customer often has no idea on what the answers to my questions are and as a small IDX vendor, the only real sources of information I feel I have is Google, hanging out in the Real Estate Webmasters forum, or reading white papers from Clareity. Maybe I should have lunch with the engineers at Redfin or RealTech, since they are dealing with the same growing pains that I am. (Calling Mr. Goyer & Mr. Davis)
I know one thing for certain, the next MLS I port my code over, will be RETS compliant, and it will be the last MLS standard I’m going to deal with. Until then, I’m just going to make my code base faster, more feature rich, and more search engine friendly since that work will benefit all my future potential customers (regardless of what MLSes I end up supporting). Hopefully that stance will twist a few more arms in the right direction.
First, a shameless self promotion. Want excel to talk to RETS? Check out the ezRETS ODBC driver. Its also cross-platform with Linux and Windows support with OS X support not far way. (I just need to improve the installers a bit.) ezRETS is built on libRETS, an cross-platform C++ API that aims to let you ignore most of the transport issues and just concentrate on your app. You still need to know about RETS, but we feel it greatly simplifies things. We also currently support bindings for libRETS to the .NET languages, Ruby, and Python, and unlike ez, libRETS fully supports all three platforms I mentioned.
Also, a previous poster already pointed out the fact that Rapp has RETS, your MLS just hasn’t deployed it. Sounds like MLS policy needs to be updated.
As for “major MLS” support of RETS, both MRIS (#1 in size in the ocuntry) and MLSNI (#2) support rets. I forgot the exact figure, and its true, only 35% of the MLS in the country support RETS, but 80% of the listings in the country are available via RETS.
As for RETSLite, that was a discussion I started within the RETS community to explore a REST-like interface for RETS. For various reasons, that discussion has been put on hold. The first being that Jeff Brush was going to introduce a change proposal for RETS2 that would open it up to a REST-like interface as well as the current SOAP interface. Without discussions like these, the standard gets stagnant and doesn’t address needs like the REST interface you were calling for above.
As for RETS moving too fast… Its been my experience that when the RETS commitee slows down people predict that RETS is dying. (Reminds me of an old netcraft joke…) When RETS moves at a good speed, people complain that they are moving way too fast and killing the past.
While I agree that some of the rhetoric around RETS has been “RETS2 is here, only chumps aren’t using RETS1″ the pragmatic truth is that it took 8 years or so for widespread deployment of RETS1. I would think that RETS2 would take at least that long, with overlapping of RETS2 and RETS1 for some time. RETS1 isn’t going away any time soon.
In any case, it sounds like you have more of a MLS policy issue than a technology issue.
I forgot to mention that along with a new rets.org that is launching today, a wiki is being added as well. You can find the new rets wiki at wiki.rets.org.
The goal is for people both as part of the RETS committee and outside it to provide information such as you cited in the first paragraph of comment 3. Eventually, some of that knowledge will be polished and “promoted” to the stable, uneditable, rets.org.
Regarding having a resource for you to find out the extent of standard support, you can read up on what MLS products are certified RETS compliant at http://www.rets.org/server
I agree with Keith that your issue is specific to the policies of the MLSs you are dealing with and suggest that you may want to encourage the MLSs you deal with to deploy RETS through all of the MLS subscribers that you have influence with.
Hi Robbie,
What’s an independent Seattle brokerage to do for a decent interative map based search ? We use currently two systems on two different sites. An interactive map based search from Wolfnet and a Web 1.0 map static map search through Logicaldog.
1. Can’t use both on one website and combine the back office functions.
2. Wolfnet is not really “pretty” and cannot be separated into 2 sites unless you buy two solutions
3. Real Tech are exclusive to John L Scott (which is the best interface out there at the moment ) and Coldwell Banker in WA
3. Graphical Data are exclusive to Windermere in WA
4. DsSearch Agent do not cover the NWMLS
5. The interfaces of the rest are not really good.
6. Your Zsearch is functional but the demo on Ardells site also looks cosmetically “functional”
Surely there is a market developing a cosmetically appealing and functional Microsoft Virtual Earth system for independents that also has a static map based search for the technically challenged. Maybe I am just frustrated.
Matt / Keith,
I’ll have to play with the .net wrapped libRETS one of these days. Thanks for all the updated information. One thing that would be helpful is a list of who doesn’t support RETS or who is in the process of supporting it. Couple of questions…
How is RETS doing in Canada? Judging by the XML returned, the standard supports no-US locales (as it should)…
How are photos, and other attachments handled (say an MP3 file or a MOV file)? The test web client on your site, only displays a single photo per listing, and I’m assuming that’s a limitation of the sample data or the client, and not the standard?
Where RETS hasn’t been deployed yet, what are the leading reasons given my MLSes?
Any advice on effective NWMLS arm twisting tactics?
I think it’s initially hilarious that the MLS’s do not use database’s of which you’re describing like it’s new technology and hasn’t been available for the last 30-40 years. Then ultimately downright depressing that none of them have done any innovation since their inception.
Not only that, but our clients (the agents) seem to want to blame the tech companies for this when in reality it’s the MLS’s and how protected they are. RETs would be a dream come true, but an honest to god XML/RSS feed would be even better. Or make a new standard called MLSRSS. I’ve been toying around with writing a microformat for listings as well. That would provide even more functionality for when Real Estate moves into the 20th century for web based systems.
I work as a 3rd party provider after an IDX to serve websites to agents, and the number of middle men to provide data to the agents who need it is sickening. The cost could be insanely cheaper if we could just get around the MLS’s.
Ian Bell,
We should get together someday. Even if you don’t do business with me, I’d still like to talk with you and find out what features I am missing. I can use your insight when I get around to developing the “next generation” of Zearch.
Have you seen my work on rpaseattle.com? I’d like to think it’s more cosmetically appealing than my work on RCG and it has a few more “back office” kinds of features. I’ll have to respond to all your other points in a separate blog posting…
Robbie,
No problem doing the update.
As for Canada, I know they’ve been investigating RETS, but as my membership is US based, I haven’t personally paid much attention. Paul Stusiak, who’s one of the technical co-chairs of RETS would be a good person to ask on that. He lives in Canada and has a good handle on that market. You’re right, RETS1 didn’t have much in the way of localization. RETS2 had that in mind as it was being developed.
As for binary data, RETS1 or RETS2 can both handle any time pretty easily. RETS1 defined 7 types of objects, but you could pass any mime-type within those 7. And those were “well-known names.” You could add more outside those 7 if needed. RETS1 can pretty much handle any arbitrary binary for any arbitrary number of them. Its all in how the server is setup or whatever. RETS2 is even more flexible in this regard.
As for that client you mention, CRT won’t claim it as “theirs” as its not, but it is on the RETS committee resource box. That client was written by others, but that only 1 image thing is either a limitation of the client or of the demo server its pulling from. Its certainly not indicative of any of limitation of RETS.
Where it hasn’t been deployed why hasn’t it? There’s a number of factors: ignorance of the standards, old wives tales of RETS being insecure, and the biggest one, as you’ve seen, a control issue. Many MLS’s do not trust their own members with the data.
As for arm twisting, a bottom up approach is also useful. Also pointing key board or AE type people at resources like NAR’s CTO, or people on the RETS committee who can explain that RETS isn’t a scary thing, that its good for the industry, and things like that. Protecting and locking up the data and making it more difficult to get access to is a losing battle in the long run, in my opinion.
Especially when its so easy for agents to just post that data on Google, Yahoo, Zillow, and others. The worst thing MLS’s can do to combat those new threats is to tighten up even more. But now I’m editorializing and speaking for myself rather than my employeer.
I’ve been using RETS in San Diego since 2002. Biggest problem I’ve run into is the MLS and the double standards when it comes to who gets access.
As I see it, the problem with RETS is that client developers are under represented. The face-to-face meetings, where things are voted on and decided, are overwelmingly MLSs and MLS vendors. No one speaks up representing the client-side interest.
It is like the MLSs think it isn’t in the MLS’s interest to make data access easy. (Can the DOJ say ‘monopoly’?) It reminds me of corporate DBAs not wanting people to use ‘their’ databases.
BTW - a MLSRSS sounds like a great idea. Of course the data would need to come out of an MLS. ;-(
Robbie, to answer your Canada question, RETS is very much in use in Canada. While not every single Canadian MLS is a client, all of my Canadian MLS clients provide access via RETS, which to Keith’s point about ‘old wives tales’, can be made far more secure than http://FTP.
Regarding the question of where the listing content should go, the individual listings belong to the brokers, and they are welcome to post them to whatever advertising media they believe is appropriate. The listings are provided to the MLS by brokers and their agents for specific purposes - facilitating cooperation and compensation between real estate professionals. Sometimes the MLS goes beyond immediate mission that in order to meet brokerage and real estate community needs - but it is not for the MLS to send listings willy-nilly all over the Internet.
There’s a balance to be made between making “data access easy” and appropriate protections on data access, as needed to protect the consumer. If consumers had their information put into the MLS and immediately, that every evening, they were deluged with phone calls, emails, and postal mail with offers from moving companies, inspectors, mortgage companies, home decorating and landscaping companies “Improve your curb appeal!!!”, etc. - the days of the MLS as useful tool for cooperation and compensation among real estate professionals would be over. Consumers would request their information *not* be put into the MLS. Perhaps the MLS would even have liability for the problem in some way. That’s when the government would get involved. At any rate, if putting my property listing in the MLS meant there was uncontrolled access to the listing information and I was subsequently harassed by marketeers, I certainly wouldn’t put my information there. Don’t get me wrong - I’m not against syndication of real estate listings for advertisement purposes - it just needs to be done, by those with rights to do it, in a controlled way that balances different legitimate interests.
Scooter, calling the MLSs a “monopoly” and invoking the DOJ is just flame-bait. What you call a “monopoly”, others would call an “efficient marketplace”. The brokers and agents in areas with multiple MLSs - what you might consider healthy competition - are not particularly happy about it. Having multiple MLS systems in a geographic area has proven to be highly inefficient and real estate professionals spend more time and money to accomplish the same thing. There certainly hasn’t been any demonstrated benefit to the consumer of creating the inefficient marketplace.
Comments and opinions are mine alone and may not represent my employer’s position.
By ‘easy data access’ I meant ‘easy to code to’ with common protocols and fields. If there were an easy, uniform way to access multiple MLSs in a region it would be possible for a single agent/broker application to query, or even update!, all the MLSs at once. While this does not directly address membership costs, it might give some leverage during negotiation. A single benevolent MLS is still the most desirable.
The ‘monopoly’ comment was a poor attempt at a joke about the recent DOJ/MLS lawsuits with a very bad delivery. Reading it now, I think I’ll put myself in timeout.
Scooter, I agree with the ‘easy to code’ to part, for sure - its a good goal and one that many of the regionalization efforts are aiming for - when they go beyond ‘data sharing’ and spend the time to establish common business rules and work to move their data definitions and uses as close together as possible - though I fear necessary localizations will always make the technical solution a challenge. Of course, there’s historical data/reporting from funky formats to contend with when trying to conform- I still love that, in some markets, 4.555 bathrooms = four full, 3 half baths. My brain is starting to hurt… and I don’t know what they would do if they ever added 3/4 baths in that market, but that could get really ugly!
As long as you intended a ‘wink’ on the monopoly comment … not everyone would. It’s always hard to tell tone in a blog!
[...] Doc ‘Trulia’ Holliday is not dumb. A master gun fighter in his own right, nothing is preventing Trulia from embracing the Zillow feed standard as their V2 spec. If that happens, RETS may suffer the same fate as Lester Moore. Out here on the wild web of the west, there’s the quick and the dead. [...]
Phentermine and methamphetamine….
Phentermine and methamphetamine….