Twing: Searching the Deep Web
The term “Deep Web” is shorthand for “stuff Google doesn’t bother spidering,” thus keeping it essentially invisible to the majority of web users, who use Brother Google as their sole guide to the web. A new search service named Twing shines a light into one vast, overlooked and easily derided shaft of the Deep Web: discussion forums.
Yes, we’re talking about those retro-web gathering spots, many designed during the early years of the Clinton Administration–back when blogs, social bookmarking and YouFaceMyTubeSpaceBook rumpus rooms were barely gleams in a delusional VC’s eye. Forums have the visual elegance of parking tickets. You often feel like you need a red-tipped cane to navigate them. Some even have little spinning logos.
And so I was startled to discover how many of these forums are going strong.
On Twing’s list of active forums I came across the following, all active as of this week and (per Twing’s calculations) growing:
- 3.2 million posts and 37,000 members in the Long Hair Care forum
- 2.5 million posts and 23,000 users in the Arofanatics forum (aquarium fish)
- 2 million posts and over 30,000 users in the OffshoreOnly forum (power boating)
Of course there are huge numbers of forums devoted to cars, guns, pets and games. And celebrities.
But let me remind you how ugly, dysfunctional and user-indifferent this forum software is. Here’s what the Long Hair Care forum looks like:
A good question is: Under what conditions might you want to use Twing as a search engine?
The answer, per Scott Germaise, Twing’s Director of Product Management [I’m paraphrasing from his e-mail response to my questions]: Whenever you’re looking not for information about the thing, but for conversations about the thing. I did a bunch of searches and can report that for many topics [one I tried: Bolshevism], Google returns basic high-value content pages. Twing delivers odd little snatches of conversation–people are still discussing Bolshevism!–that occur in active forums.
Google often surfaced active forums among search results. Twing delivered those only, and more of them.
Messing around with Twing reminded me of the early days of Yahoo, when I could spend hours muttering to myself, “who are these freaking people?” as the world’s remarkable obscurities slid across the screen. It also illustrated how. . .burnished and predictable and gamed the web has become. A lot of the loopiest, most strangely human content has sunk from view.
I have no idea whether there is a business here, or whether I will use Twing for any real-life searches that GooHoo can’t handle.
But Twing has shown me this: That whole creating communities-of-shared-interests thing? That whole power-of-collective wisdom thing? That whole long tail thing? You know, that whole web 2.0 thing?
It’s been under our noses all along. We just forgot where to look
Explore posts in the same categories: forums, search, social networks, Uncategorized, Web 2.0Tags: forums, vertical search, Web 2.0
You can comment below, or link to this permanent URL from your own site.
Web 2.Oh. . .really? 
18, April, 2008 at 4:07 pm
[…] 18 04 2008 Craig Stoltz’ post today is a tad long, but I really encourage folks to read it. https://2ohreally.wordpress.com/2008/04/17/twing-searching-the-deep-web/ […]
18, April, 2008 at 5:09 pm
Love the writing style. Fun metaphors.
The thing is with forums though, perhaps with anything, it’s all about what you search for. Try searching for “chron’s disease” or browse the directory down in the Health & Medicine area. Or try “archery” or “learn to fly” That last one can also get you fly fishing results, so in the results, use refinements and choose Select Categories to scope the results to Recreation and Hobbies. In all of these places, tons of value. Real people. Real conversations. Not someone just talking at someone. But people really engaged with their communities and their passions; be they professional or recreational or some other need.
I’m don’t know what all your search keywords were. But I’d suggest you look again. The forum space – like any other online communication space – has it’s stunningly bizarre little corners. But there’s as much or more of these oddities going on in the vapid exchanges on a great many social networks. And quite often, a lot more real value to be had from forums.
Forums may still be a bit retro when it comes to look and feel. But that’s both fixable and arguable not all that important anyway. And I think you’ll start to see more contemporary skins applied to the forum UIs in the near future. But the main point you made that I agree with completely is with regards to “that whole long tail thing.” “It’s been under our noses all along. We just forgot where to look”
Stay tuned Craig. I think you may find as with many things; all that is old is new again. And forums could be one of those overnight successes 10 years in the making. Sexy like the hot new stuff? Maybe not so much. Real undiscovered value? Yeah.
Scott
Director, Product Mangement
Twing.com
18, April, 2008 at 5:11 pm
(ahhh… apologies for the typos/grammar. I’ve got to learn to not hit submit so quickly!)
20, April, 2008 at 8:06 pm
Thanks for the comment, Scott.
Let me add an observation I didn’t include in my (already overlong, as “Flip the Media” mentions above) blog item.
One of my favorite features of Twing is the ability to discover simply that these forums exist. Search results not only unearth buried comments on very specific issues, but they inevitably point to forums where related topics are discussed.
6, May, 2013 at 10:53 am
[…] neat little search engine searches forums. I came across it on Web2.ohreally, where Craig Stoltz discusses what Twing essentially does – it searches the “Deep […]