Recommending Twitter Users to Follow Using Content and Collaborative Filtering Approaches

This paper was published in RecSys 2010 and is authored by researchers from University College Dublin, in Ireland. The problem studied in the paper is recommending Twitter followees (information sources). The authors propose several profiling strategies - based on tweets, followers, and followees - in order to represent users' interests, enabling accurate recommendations.

The authors of this paper were working on a real system, called Twittomender, for user recommendation based user profiles and query terms. Therefore, they give a lot of detail on the architecture of such system, which is not actually my focus as a reader. So, I will skip most of these details here.

The profiling strategies proposed are the following:

1) User's tweets;
2) Tweets of the user's followees;
3) Tweets of the user's followers;
4) User's followees (ids);
5) User's followers (ids).

Profiles were indexed using the Lucene platform, which is a traditional platform for indexing and querying text documents. Each user is mapped into a document and the features (words or ids) are represented as terms. Recommendations are made using Lucene querying system. A user or a set of keywords is given to Lucene as a query. Terms are weighted based on TF-IDF.

The profiling strategies are evaluated in two experiments, an offline and an online one. In the offline experiment, a set of 20,000 users are divided into a training (19,000 users) and a test (1,000 users) set. The profiling strategies are evaluated in 9 different forms:

S1: Users are represented by their tweets;
S2: Users are represented by the tweets of their followees;
S3: Users are represented by the tweets of their followers;
S4: Users are represented as a combination of their own tweets, the tweets of their followees and the tweets of their followers;
S5: Users are represented by the IDs of their followees;
S6: Users are represented by the IDs of their followers;
S7: Users are represented by a combination of the IDs of their followees and followers;
S8: Ensemble that combines strategy S1 and S6;
S9: Ensemble that combines strategies S1-S7. The position of a user in the resulting list is a function of its rank in the results generated by the set of strategies.

The following figure show the recommendation precision of each of these strategies:

The evaluation is based on the overlap between the recommended lists and the actual followee lists of users. The authors do not explicitly state that they do not use test data to train their methods but I assume it. The best strategies in terms of precision are S6, S7, and S9. One interesting result is that the profiling strategy based on the followee list does not achieve as good results as the strategy based on the followers list.

Another quality measure considered in the offline evaluation is the average relevant document position for different recommendation list sizes (see the upcoming figure).

This evaluation give opposite evidences of the quality of the strategies, compared to the previous one.  However, I would expect that a good method brings more relevant users to the top-20 ranking (evaluations are based on the top-20 users returned). As a consequence, good methods would have higher average relevant user position, except if only the first relevant user position is considered. However, it seems like the authors were expecting good strategies to have lower average values, which is counter-intuitive.

The online experiment was based on evaluations provided by 34 volunteer Twittomender users. Only the strategy S6 was applied. The quality metrics considered were the number of relevant recommendations along the rankings. Users were invited to test both the profile and the keyword-based interfaces of the system. In general, results show that the relevant users were positioned on the top positions of the ranking for the two interfaces. Moreover, the query interface achieved worse results compared to those achieved using the interface based on user-profiles.

I see this paper as an application paper, not a research paper. It is well written in general, but the profiling and recommendation strategies are very simplistic and the results are not very conclusive. Both the offline and the online experiment were based on a small amount of users. As an application paper, it should focus more on the architectural and performance aspects of Twittomender rather than in user profiling and recommendation accuracy.

Link: http://irserver.ucd.ie/dspace/bitstream/10197/2524/1/john-hannon-recsys-v4-April-25.pdf