Overview

A CS5224 cloud-computing project that computes follower and followee counts for every user in a large Twitter follower graph, then serves those statistics through a web app. The point is to wire distributed computation, object storage, an API, and a frontend into one reproducible cloud pipeline rather than to run a single script locally.

Problem & Constraints

Analyzing large-scale graph topology locally is constrained by memory limits and single-threaded execution. Processing 2.4M edges requires distributed graph computation, cloud object storage, and automated partition aggregation while serving results cleanly to a low-latency web interface.

My Contribution

• MapReduce Algorithm Design: Designed single-pass Python mapper and reducer scripts that simultaneously emit and sum both follower (in-degree) and followee (out-degree) counts for 81,306 users from 2.42M raw edge tuples.
• AWS Infrastructure Setup: Configured AWS EMR (release emr-6.15.0) multi-node clusters, automated Hadoop Streaming step execution, and organized S3 bucket storage for inputs, scripts, logs, and output partition files (part-0000x).
• Serving Layer & Web Dashboard: Developed a Flask REST service (with CORS) that reads S3 partitions directly into memory, and paired it with a paginated, searchable React 18 dashboard.

Method & System Architecture

The data flows through five stages:

Pipeline stages

Dataset -> S3 -> EMR (Hadoop Streaming MapReduce) -> S3 -> Flask API -> React frontend

The raw edge list is uploaded to S3, processed by a MapReduce job on an EMR cluster, written back to S3 as output part files, fetched and combined by a Flask service, and displayed in a React interface.

Data & Input Pipeline

The input is the Stanford SNAP twitter_combined.txt social-circles dataset: a directed follower graph of 2,420,766 edges (about 42 MB), where each line is a follower followee pair. The MapReduce job reduces this to per-user statistics for 81,306 unique users.

Experiments & MapReduce Implementation

The job computes each user’s in-degree (followers) and out-degree (followees). The mapper reads each edge once and emits two tallies, so a single pass over the edge list feeds both counts:

mapper.py

# for each "follower followee" edge
print(f"{followee_id}\tfollower\t1")  # followee gains a follower
print(f"{follower_id}\tfollowee\t1")  # follower gains a followee

The reducer groups by user id and sums the two tally types, emitting user_id, followers, followees per user. The design is intentionally simple and stateless so it parallelizes cleanly across the cluster.

Cloud Cluster Setup

The job runs on AWS EMR (release emr-6.15.0) with Hadoop, across a small multi-node cluster, submitted as a Hadoop Streaming step that ships the Python mapper and reducer to the nodes:

EMR Hadoop Streaming step

hadoop-streaming \
  -files s3://<bucket>/scripts/mapper.py,s3://<bucket>/scripts/reducer.py \
  -mapper mapper.py -reducer reducer.py \
  -input s3://<bucket>/input/twitter_combined.txt \
  -output s3://<bucket>/outputs

S3 holds the input dataset, the scripts, the logs, and the output, which lands as three part-0000x files.

Serving & API Layer

A Flask service (with Flask-CORS) reads the output part files directly from their public S3 URLs, combines them in memory, and exposes a small REST API: /api/health, /api/stats, and /api/stats/<user_id>. A React 18 frontend (loaded via CDN) calls these endpoints with the Fetch API and renders a searchable, paginated table of per-user follower and followee counts.

Results

The pipeline produces follower and followee counts for all 81,306 users. For example, two reference users resolve to:

The most-followed user in the dataset has around 8,660 followers. The end result is a working demonstration of distributed processing, cloud storage, API serving, and frontend visualization stitched into one workflow.

Limitations & Lessons

• In-Memory Partition Assembly: Reading part files directly into Flask memory is efficient for 81K users, but scaling to tens of millions of nodes requires loading output partitions into a database index like DynamoDB or Redis.
• Hadoop Streaming Overhead: Python mapper/reducer scripts communicate via stdout/stdin, which introduces standard stream serialization overhead compared to native Java MapReduce or PySpark tasks.

References

1. J. McAuley and J. Leskovec. “Learning to Discover Social Circles in Ego Networks.” Stanford SNAP ego-Twitter dataset. snap.stanford.edu/data/ego-Twitter.html.