February 2011
4 posts
1 tag
catching up
The crawler’s been trying to make the best of having only 150 API hits per hour for a couple weeks now. I haven’t had much time to work on the site lately, but once I dug in tonight, I discovered that for some reason the OAuth token and secret for @FavRatio had changed. That’s all it was. I reconfigured it to use the new token and secret, and now it’s back to 350 API hits...
serenebabe-deactivated20110225 asked: It's nice to have such a clean look on the site. But, one of the things I like to do is check out people who I see frequently have starred tweets that I've also starred (a way to find new people to follow). Is there a reason beyond creating white space that you've made the icons/avis under the tweets so very, very tiny?
new stats
There’s now a whole page of system-wide stats at http://favratio.com/system-stats. For each interval you can see who starred the most, who earned the most stars, who earned the highest cumulative FavWeight score, etc.
There will be more statistics to come, but these are a few basic ones to get started.
Can you think of other system-wide statistics that would be interesting to see?
1 tag
crawler changes
Over the past few days I’ve been doing a lot of behind-the-scenes work: rearranging code, refactoring it to be simpler and execute more quickly, etc. Not much has changed with the appearance of the site, but the crawler now operates a bit differently than before.
In addition to dynamically speeding up or backing off depending on how many tweets were found, the crawler now has the ability to...
January 2011
18 posts
faster page loads
The system-wide stats available in the right-hand column used to be calculated every time a page was loaded. Now they’re cached for a few minutes, which should speed up the site a bit.
simplify the display
There is now a settings menu on the right hand side that lets you hide any of the additional information about tweets that you may not care about or want to see, including:
the avatar of the person who wrote the tweet
the time the tweet was posted
the links to Favstar and/or Twitter
the avatars of people who faved the tweet
the total number of stars
the total FavWeight
Make it as minimal or...
links to Favstar
Each tweet now has a link to its page on Favstar. The Favstar links are there to provide information that only Favstar can, such as the total number of stars each tweet gathered across all of Twitter (not just the FavRatio members), the number of RTs, etc. It’s also meant to be a goodwill gesture; I think FavRatio and Favstar should complement, rather than compete with, each other.
I made...
1 tag
new stat: similar tastes
The tweeter detail page (http://favratio.com/user/yoyoha, for example) now checks all of the stars that person has given over a certain time period, looks at who else starred those tweets, and shows a list of the top ten people that starred the same tweets — that is, the other tweeters whose “tastes” have been similar over that time period.
Right now it only pays attention to...
1 tag
more crawler tweaking
Making a computer try to anticipate when you’ll have new stars to collect is tricky.
The latest approach works like this:
When FavRatio first starts checking your stars, it will check every 30 minutes.
At any time, if it finds 0 stars, it will reset to checking every 30 minutes.
At any time, if it finds 20 stars, it will crawl a lot sooner next time; the current “panic” rate...
faster FavRatio
All of the boards (recent, weighted, most stars, and underfollowed) now load and “page” much faster due to some database denormalization and query optimizations I made today. If you use the site regularly, I hope you’ll notice a difference.
If you don’t use the site regularly, why not?
1 tag
counting every star
In the past if you starred a tweet written by someone who wasn’t on FavRatio, it wouldn’t be counted. This was because during a crawl, every author of a starred tweet would be added to the DB and automatically included in the crawler…but not all tweet authors found during crawls do any starring at all; API calls to crawl them are wasted, so the tweet authors were simply never...
"underfollowed" leaderboards
There’s a new maxfollowers GET parameter that will limit any leaderboard or “recent” page to display only tweets from accounts with no more than the specified amount of followers.
So, for example, if you want to see today’s highest-FavWeighted tweets from people with no more than 500 followers, try: http://favratio.com/top/day/weighted?maxfollowers=500.
To see all of the...
fans and crushes
The “user detail” page on FavRatio now shows the top 10 people over the last week who:
received faves from the person (crushes)
gave faves to the person (fans)
For example: http://favratio.com/user/3hoss
1 tag
better crawling
FavRatio now automatically adjusts how often it crawls (checks your stars) based on how many new stars it saw the last time it crawled you. For the moment, the logic — which I’m still tweaking — works like this:
If it found 0 stars, it’ll wait an hour before crawling you again.
If it found only a handful, it’ll crawl you 20 minutes later than it did the last time.
...
new tweeter detail pages
I was traveling during the past week and didn’t have much time to work on the site, but I wanted to mention a couple new features:
recent tweets: all of the tweets someone has posted in the last week
given favs: all of the tweets someone starred in the last week, hour, day, month, etc.
Both of these pages are now linked from the tweeter detail page (http://favratio.com/user/badbanana, for...
2 tags
customized fav and weight thresholds
Other fav sites’ various leaderboards have used the hard-coded minimum thresholds 3, 10, 30, 50, 100…sometimes the exact number isn’t published. FavRatio now lets you choose whatever arbitrary threshold of favs you’d like, and you can do it with FavWeights, too.
By changing the URL, you can dynamically filter the leaderboards (both traditional and weighted) to show only...
1 tag
one crawl per hour
I’ve come to discover that the Twitter API rate limit isn’t based on a rolling hour as I previously thought; it begins counting down an hour when you make a request, and it will stop you if you reach the limit before that hour has passed.
This is totally different from the way I mistakenly thought it worked, which was thus: as long as you haven’t made over 350 requests over the...
the temperamental crawler
The crawler still requires a fair amount of babysitting and tweaking. I’ve been steadily adding new tweeters, and the crawl interval has been automatically inching upward in response.
This is good because it’s designed to do that, and as a result it’s being well behaved in the Twitter API’s eyes. This is also bad because it now has to wait over a half hour between crawls.
...
new stats
New tweeters have been joining (hi!) and FavRatio still isn’t whitelisted, so it’s still restricted to 350 API calls per hour by Twitter. With 145 tweeters (at the moment), FavRatio can’t check everyone’s favorites every 15 minutes anymore. That would require 580 API calls per hour.
I’ve made a few changes to the crawler recently that allow it to dynamically adjust...
1 tag
FavWeight minimums
I noticed that as FavRatio began looking at new people, the first few favs of theirs would get recorded with a pretty high FavWeight: 0.5 if it had seen only 2 favs over the past “week” (really the same as only having seen 2 favs over the past hour or just during the last crawler run). Same for 3 favs seen: 0.333. It doesn’t sound like much, but these are orders of magnitude...
coming soon: most-wanted features
starring from FavRatio: This is going to require a bit of work, since I need to figure out how the hell OAuth works in order to let FavRatio star on your behalf. OAuth is that thing where it asks you “Do you give permission for FavRatio to connect to your Twitter account?” In addition, after you’ve signed in once this way, you’ll start getting your favs crawled, if...
New Years Baby!
It’s been a good day.
There was a brief outage caused by a stupid typo in the code and an even stupider decision to change production code without testing it first, but the site’s been running smoothly since then. Lesson: learned.
I added a UserVoice widget for feedback and some more stats, and made a change to the FavWeight algorithm that should keep newly-crawled tweeters from...
