FavRatio blog

catching up

Wed, 09 Feb 2011 18:51:07 -0600

The crawler’s been trying to make the best of having only 150 API hits per hour for a couple weeks now. I haven’t had much time to work on the site lately, but once I dug in tonight, I discovered that for some reason the OAuth token and secret for @FavRatio had changed. That’s all it was. I reconfigured it to use the new token and secret, and now it’s back to 350 API hits per hour. The crawler is well on its way to catching back up and staying totally up to date.

I feel kind of stupid realizing that it would have been such an easy fix this whole time.

It's nice to have such a clean look on the site. But, one of the things I like to do is check out people who I see frequently have starred tweets that I've also starred (a way to find new people to follow). Is there a reason beyond creating white space that you've made the icons/avis under the tweets so very, very tiny?

Thu, 03 Feb 2011 23:57:49 -0600

I hoped to accomplish a couple things by making the faver avatars small:

To maintain a somewhat clean look, as you mentioned. (thanks!)
To create more focus on the content and author of the tweet and less on the people who starred it.

That’s just a site default, though. Once I have profiles completely working (login with Twitter, at the top right), this will probably become a setting you can configure: whether to show avatars at 12, 24, or 48 px.

By the way, there’s a part of the user detail page (http://favratio.com/user/yourTwitterUsername) that shows people who seem to have had similar taste to you recently — they have the most favs in common with you. Does that help?

new stats

Thu, 03 Feb 2011 23:49:56 -0600

There’s now a whole page of system-wide stats at http://favratio.com/system-stats. For each interval you can see who starred the most, who earned the most stars, who earned the highest cumulative FavWeight score, etc.

There will be more statistics to come, but these are a few basic ones to get started.

Can you think of other system-wide statistics that would be interesting to see?

crawler changes

Wed, 02 Feb 2011 16:43:24 -0600

Over the past few days I’ve been doing a lot of behind-the-scenes work: rearranging code, refactoring it to be simpler and execute more quickly, etc. Not much has changed with the appearance of the site, but the crawler now operates a bit differently than before.

In addition to dynamically speeding up or backing off depending on how many tweets were found, the crawler now has the ability to “catch up” if it falls behind by requesting additional “pages” of tweets (groups of 20) until it’s sure it’s found everything it had missed by waiting too long to crawl. This is limited at the moment to 5 pages, or 100 tweets.

The statistics now show how far the crawler is behind, if at all, and how many crawls are queued, i.e., how many tweeters should have been crawled already according to their current crawl rate.

This is helping the crawler adjust to the recent development of being allowed to use only 150 API hits per hour, which is down from 350 just a week or so ago. This may have been caused by some bug in FavRatio that’s appeared now that I’ve been trying to get OAuth (sign in using Twitter) working.

You should soon be able to star tweets directly from FavRatio after signing in…that’s the first goal of the OAuth integration.

faster page loads

Mon, 24 Jan 2011 18:12:19 -0600

The system-wide stats available in the right-hand column used to be calculated every time a page was loaded. Now they’re cached for a few minutes, which should speed up the site a bit.

simplify the display

Thu, 20 Jan 2011 19:28:15 -0600

There is now a settings menu on the right hand side that lets you hide any of the additional information about tweets that you may not care about or want to see, including:

the avatar of the person who wrote the tweet
the time the tweet was posted
the links to Favstar and/or Twitter
the avatars of people who faved the tweet
the total number of stars
the total FavWeight

Make it as minimal or as complex as you like. It’s up to you.

For the time being this is session-based, meaning it will reset if you close your browser and come back later. Once OAuth is working, you’ll be able to save these preferences as part of your profile, and FavRatio will retain the look you want every time you visit the site.

links to Favstar

Mon, 17 Jan 2011 19:06:32 -0600

Each tweet now has a link to its page on Favstar. The Favstar links are there to provide information that only Favstar can, such as the total number of stars each tweet gathered across all of Twitter (not just the FavRatio members), the number of RTs, etc. It’s also meant to be a goodwill gesture; I think FavRatio and Favstar should complement, rather than compete with, each other.

I made some other changes to the layout today to try to reduce visual clutter.

new stat: similar tastes

Mon, 17 Jan 2011 13:03:13 -0600

The tweeter detail page (http://favratio.com/user/yoyoha, for example) now checks all of the stars that person has given over a certain time period, looks at who else starred those tweets, and shows a list of the top ten people that starred the same tweets — that is, the other tweeters whose “tastes” have been similar over that time period.

Right now it only pays attention to stars given over the past day, because this particular stat is “expensive” from a database and computation standpoint. If I can improve that, I’ll probably make it look at stars given over the past week in order to match the time period of the other stats.

What other stats would you like to see?

more crawler tweaking

Sun, 16 Jan 2011 20:07:59 -0600

Making a computer try to anticipate when you’ll have new stars to collect is tricky.

The latest approach works like this:

When FavRatio first starts checking your stars, it will check every 30 minutes.
At any time, if it finds 0 stars, it will reset to checking every 30 minutes.
At any time, if it finds 20 stars, it will crawl a lot sooner next time; the current “panic” rate is one crawl per 7 minutes.
If it finds 11 or less stars, but not 0, it will back off the crawl rate a little bit.
If it finds between 12 and 19 stars, that’s the “sweet spot,” and it’ll continue crawling at whatever rate it has settled on.

These are the design principles:

Finding 20 stars, the maximum that Twitter will tell you about, is bad; it’s more likely that some stars were missed than that, by sheer coincidence, you’ve given exactly 20 stars since the last crawl. Panic! Confusion! Loud noises! Actually, it’ll just crawl a lot sooner, in the hope that less than 20 stars will be collected next time.
Finding 0 stars is bad; maybe you’re sleeping or just away from Twitter…why waste an API hit if it’s not going to find anything?
Finding only a few stars, like 3 (up to 11, actually), means FavRatio can probably crawl a little slower and still not miss any stars, so it’ll do so in order to make the most efficient use of the API hits.
Finding 12 or more stars, but not 20, seems just about right.

faster FavRatio

Fri, 14 Jan 2011 18:44:02 -0600

All of the boards (recent, weighted, most stars, and underfollowed) now load and “page” much faster due to some database denormalization and query optimizations I made today. If you use the site regularly, I hope you’ll notice a difference.

If you don’t use the site regularly, why not?

counting every star

Thu, 13 Jan 2011 23:14:15 -0600

In the past if you starred a tweet written by someone who wasn’t on FavRatio, it wouldn’t be counted. This was because during a crawl, every author of a starred tweet would be added to the DB and automatically included in the crawler…but not all tweet authors found during crawls do any starring at all; API calls to crawl them are wasted, so the tweet authors were simply never added to the DB at all…which meant their tweets didn’t show up if they weren’t members.

Now FavRatio differentiates between tweeters that will be crawled and tweeters that have just written tweets that FavRatio members have starred. Of the thousands of tweeters now in the DB, only a subset of them (members) are crawled. Many, many more tweets are now showing up because of this.

The other nice side effect is that all of your stars are counted, regardless of who received your stars.

"underfollowed" leaderboards

Thu, 13 Jan 2011 15:19:36 -0600

There’s a new maxfollowers GET parameter that will limit any leaderboard or “recent” page to display only tweets from accounts with no more than the specified amount of followers.

So, for example, if you want to see today’s highest-FavWeighted tweets from people with no more than 500 followers, try: http://favratio.com/top/day/weighted?maxfollowers=500.

To see all of the tweets from people with 1000 followers or less during the past hour in chronological order (newest first), try: http://favratio.com/recent/hour?maxfollowers=1000.

Of course, you can combine any of the GET parameters in any way you’d like. This URL shows the week’s top FavWeighted tweets with at least 3 stars from people with 300 or less followers: http://favratio.com/top/week/weighted?minfavs=3&maxfollowers=300

One of these days I’ll build something into the interface so that you can use something like number input form fields or slider controls instead of directly changing the URL like this. In the meantime, try experimenting with different minfavs, minweight, and maxfollowers numbers!

What other little “tweaks” like this would you like to see on FavRatio?

fans and crushes

Wed, 12 Jan 2011 16:58:13 -0600

The “user detail” page on FavRatio now shows the top 10 people over the last week who:

received faves from the person (crushes)
gave faves to the person (fans)

For example: http://favratio.com/user/3hoss

better crawling

Tue, 11 Jan 2011 20:51:00 -0600

FavRatio now automatically adjusts how often it crawls (checks your stars) based on how many new stars it saw the last time it crawled you. For the moment, the logic — which I’m still tweaking — works like this:

If it found 0 stars, it’ll wait an hour before crawling you again.
If it found only a handful, it’ll crawl you 20 minutes later than it did the last time.
If it found more than a handful, but less than 10, it’ll crawl you 10 minutes later than last time.
If it found 10 or more, it’ll crawl you 10 minutes later than last time.
If it found 20, it’ll crawl you 20 minutes earlier than last time.

The idea is to use each crawl to gather as many new stars as possible — that is, close to 20 — without missing any stars. The intent is that it will back off the crawl rate when not much starring is occurring (while you’re sleeping, for example) but crawl faster if it notices that you’re starring tweets more often.

It doesn’t make much sense to use an API hit every hour for 6 hours while you sleep when it’s not going to find any stars anyway. This algorithm, or something like it, will be a way to make sure those precious few 350 API hits per hour can be used to efficiently crawl faves for over 400 tweeters…and counting.

new tweeter detail pages

Tue, 11 Jan 2011 17:06:00 -0600

I was traveling during the past week and didn’t have much time to work on the site, but I wanted to mention a couple new features:

recent tweets: all of the tweets someone has posted in the last week
given favs: all of the tweets someone starred in the last week, hour, day, month, etc.

Both of these pages are now linked from the tweeter detail page (http://favratio.com/user/badbanana, for example)

Neither of these are new to anyone who’s used star sites before; they just weren’t available on FavRatio until now.

[EDIT] I forgot to mention a new feature that’s actually new! The earned:given ratio now displays on the tweeter detail page. It shows how many tweets someone starred for every star they earned — or vice versa — over a given period…whichever is larger.

customized fav and weight thresholds

Tue, 04 Jan 2011 16:20:34 -0600

Other fav sites’ various leaderboards have used the hard-coded minimum thresholds 3, 10, 30, 50, 100…sometimes the exact number isn’t published. FavRatio now lets you choose whatever arbitrary threshold of favs you’d like, and you can do it with FavWeights, too.

By changing the URL, you can dynamically filter the leaderboards (both traditional and weighted) to show only tweets with a minimum number of favs, a minimum FavWeight, or both. Use the minfavs and/or minweight GET parameters in the URL.

For example:

Compare http://favratio.com/top/day/weighted?minfavs=22 to the regular weighted leaderboard (http://favratio.com/top/day/weighted) Even if a fav is heavily weighted, it won’t show up at the former URL unless it wasn’t faved by at least 22 people. Thus, there are many fewer pages to scroll through at the first URL.

http://favratio.com/top/day?minweight=0.5 will show less tweets than the standard leaderboard, since a tweet has to be heavily weighted in addition to earning a lot of individual favs to appear at this URL.

http://favratio.com/top/day/weighted?minfavs=13&minweight=0.6 will only show tweets that earned at least 13 favs and also have a FavWeight of at least 0.600, ranked by FavWeight.

If you want to see a standard leaderboard of just those tweets that earned over 10 favs, like some other fav sites, use http://favratio.com/top/day?minfavs=10.

This allows you to create leaderboards that are as inclusive or restrictive as you prefer. The combinations are endless. Please feel free to experiment!

one crawl per hour

Mon, 03 Jan 2011 23:00:10 -0600

I’ve come to discover that the Twitter API rate limit isn’t based on a rolling hour as I previously thought; it begins counting down an hour when you make a request, and it will stop you if you reach the limit before that hour has passed.

This is totally different from the way I mistakenly thought it worked, which was thus: as long as you haven’t made over 350 requests over the last 60 minutes at any given moment, you have API hits available. Not true. This is why you’re supposed to read the manual before you plug something in…

Now the crawler code is much simpler because it only needs to check if there are enough hits remaining to do a crawl, and if not, wait until the “reset time” the Twitter API kindly provides.

This is bad because ever since the number of tweeters passed 175, FavRatio can no longer crawl multiple times per hour. Once per hour for everyone: that’s it.

This is good because the code is simpler, and it makes no difference to the crawler if we add another 150 tweeters or so. Come on over, and tell your friends!

However, the crawler is still inefficient. It checks the favs of every tweeter during every crawl. Some tweeters fav frequently and some don’t, but this doesn’t account for that. This wastes API hits checking tweeters that the system should be able to predict will probably not have enough favs to make them worth checking yet.

As alluded to previously, the crawler operation will soon be changed from a single all-or-nothing crawl to dynamic crawls based on the likelihood that close to 20 favs will be collected for each user in the crawl, to make far more efficient use of the limited requests per hour. In fact, a crawl could probably run every minute, only checking favs for tweeters that, based on their recent starring behavior, have probably accumulated close to (but not over!) 20 stars by then.

The need for this smarter crawler is more pressing, because until this happens, some favs will be lost if you give more than 20 stars per hour.

the temperamental crawler

Mon, 03 Jan 2011 21:46:03 -0600

The crawler still requires a fair amount of babysitting and tweaking. I’ve been steadily adding new tweeters, and the crawl interval has been automatically inching upward in response.

This is good because it’s designed to do that, and as a result it’s being well behaved in the Twitter API’s eyes. This is also bad because it now has to wait over a half hour between crawls.

Pretty soon one of 3 things needs to happen:

Twitter whitelists FavRatio
I stop adding new tweeters and/or remove tweeters that don’t give many/any stars
We accept that some stars will be missed as the crawls happen less and less often

I don’t consider either of the last two to be preferable, but there’s only so much you can do with 350 API requests per hour. I’ll probably also start working on a priority system so that not every tweeter gets crawled on the same schedule; those who only tend to give no more than 5 stars per hour, for example, will only be crawled once per 4 hours, since that would collect all 20 of their most recent stars. That should make better use of those precious, paltry 350 API requests.

new stats

Mon, 03 Jan 2011 11:55:15 -0600

New tweeters have been joining (hi!) and FavRatio still isn’t whitelisted, so it’s still restricted to 350 API calls per hour by Twitter. With 145 tweeters (at the moment), FavRatio can’t check everyone’s favorites every 15 minutes anymore. That would require 580 API calls per hour.

I’ve made a few changes to the crawler recently that allow it to dynamically adjust the frequency of crawls according to the number of API requests available per hour, factoring in the number of tweeters whose favs will be crawled. There’s a new stat, minimum crawl interval, that shows how often crawls can occur.

The time of the last crawl as well as the time at which the next crawl can occur are shown as well.

Other stats, provided directly by the Twitter API, include API hits used, API hits remaining, and the API hourly limit. I hope the latter increases from 350 to 20,000 soon.

The crawler “backs off” if, for whatever reason, it has been crawling too frequently and the current crawl would exceed the rate limit before completing. This should keep the crawler “well behaved” in the eyes of the Twitter API, never exceeding its rate limit. And it won’t perform a crawl unless it knows it’ll be able to collect favs for every tweeter in the system.

Crawler efficiency refers to how many of the available API calls over the last hour were used for fav crawling.

FavWeight minimums

Sun, 02 Jan 2011 18:47:29 -0600

I noticed that as FavRatio began looking at new people, the first few favs of theirs would get recorded with a pretty high FavWeight: 0.5 if it had seen only 2 favs over the past “week” (really the same as only having seen 2 favs over the past hour or just during the last crawler run). Same for 3 favs seen: 0.333. It doesn’t sound like much, but these are orders of magnitude higher than FavWeights over the same time period for tweeters that have been getting crawled longer.

That inappropriately skews the leaderboards and unfairly awards heavier weights to favs from newly-added tweeters. And the FavWeight is supposed to reflect the relative value of the fav according to how many favs the faver gives, not the “newness” of the faver.

To fix this, I created some minimum thresholds. For example, it’s currently set so that when a new person is added to the crawler, their favs won’t count toward the weighted scores until the crawler has recorded at least 8 favs of theirs within the past week. That shouldn’t take long…less than a day, possibly less than an hour. After that, their favs will count toward the weighted leaderboards, in a more equitable manner compared to the tweeters that have been getting crawled longer.

These settings are configured in the INI file under the [favweight] section: MIN_FAVS_PER_MINUTE, MIN_FAVS_PER_HOUR, MIN_FAVS_PER_DAY, MIN_FAVS_PER_WEEK, and MIN_FAVS_PER_MONTH.

The crawler is now using these minimum thresholds, and past FavWeights that were recorded too high have been lowered to reflect the maximum FavWeight possible during each time period…so the weighted leaderboards are displaying what they would have been if the minimum thresholds been in place all along.