Network Analysis on AWS EMR

Overview

A CS5224 cloud-computing project that computes follower and followee counts for every user in a large Twitter follower graph, then serves those statistics through a web app. The point is to wire distributed computation, object storage, an API, and a frontend into one reproducible cloud pipeline rather than to run a single script locally.

Architecture

The data flows through five stages:

1Dataset -> S3 -> EMR (Hadoop Streaming MapReduce) -> S3 -> Flask API -> React frontend

The raw edge list is uploaded to S3, processed by a MapReduce job on an EMR cluster, written back to S3 as output part files, fetched and combined by a Flask service, and displayed in a React interface.

Data

The input is the Stanford SNAP twitter_combined.txt social-circles dataset: a directed follower graph of 2,420,766 edges (about 42 MB), where each line is a follower followee pair. The MapReduce job reduces this to per-user statistics for 81,306 unique users.

MapReduce Job

The job computes each user’s in-degree (followers) and out-degree (followees). The mapper reads each edge once and emits two tallies, so a single pass over the edge list feeds both counts:

1# for each "follower followee" edge
2print(f"{followee_id}\tfollower\t1")  # followee gains a follower
3print(f"{follower_id}\tfollowee\t1")  # follower gains a followee

The reducer groups by user id and sums the two tally types, emitting user_id, followers, followees per user. The design is intentionally simple and stateless so it parallelizes cleanly across the cluster.

Cloud Setup

The job runs on AWS EMR (release emr-6.15.0) with Hadoop, across a small multi-node cluster, submitted as a Hadoop Streaming step that ships the Python mapper and reducer to the nodes:

1hadoop-streaming \
2  -files s3://<bucket>/scripts/mapper.py,s3://<bucket>/scripts/reducer.py \
3  -mapper mapper.py -reducer reducer.py \
4  -input s3://<bucket>/input/twitter_combined.txt \
5  -output s3://<bucket>/outputs

S3 holds the input dataset, the scripts, the logs, and the output, which lands as three part-0000x files.

Serving Layer

A Flask service (with Flask-CORS) reads the output part files directly from their public S3 URLs, combines them in memory, and exposes a small REST API: /api/health, /api/stats, and /api/stats/<user_id>. A React 18 frontend (loaded via CDN) calls these endpoints with the Fetch API and renders a searchable, paginated table of per-user follower and followee counts.

Results

The pipeline produces follower and followee counts for all 81,306 users. For example, two reference users resolve to:

The most-followed user in the dataset has around 8,660 followers. The end result is a working demonstration of distributed processing, cloud storage, API serving, and frontend visualization stitched into one workflow.

Key Technologies

• Hadoop Streaming MapReduce: single-pass follower/followee counting over the edge list
• AWS EMR: managed Hadoop cluster running the streaming job
• AWS S3: storage for the dataset, scripts, logs, and output part files
• Flask + Flask-CORS: REST API combining the S3 output in memory
• React 18: searchable, paginated frontend over the API

References

1. J. McAuley and J. Leskovec. “Learning to Discover Social Circles in Ego Networks.” Stanford SNAP ego-Twitter dataset. snap.stanford.edu/data/ego-Twitter.html.