This is a story of struggle and friendship. We set to download the entire Instagram and came pretty close to achieve our goal. Along the way, we developed a fast and reliable stateful queue system based on Postgres and today you can easily integrate task delegation in your software architecture. https://youtu.be/g8P_w5dyW3c

1. can you download
the whole Instagram?
A bold tale of struggle and friendship

2. Agenda
➔ start a demo
➔ understand the challenge
➔ integrate people
➔ HOW WE DID IT
➔ see demo results

3. CEO COO Prod. owner Developer CTO
BusinessAwareness
THE COMPANY GAP
Technical Awareness

4. making
instagram profiles
SEARCHABLE

8. THE NEW GUY

9. CEO COO Prod. owner Developer CTO
BusinessAwareness
THE COMPANY GAP
Technical Awareness

10. Getting into engineering
LevelofBlackMagic
Technical skills

11. Level zero in dev.
Study & Google
copy/paste
npm install
NEED TO REDUCE POINTS

12. Getting into engineering
Technical skills
New guy level
LevelofBlackMagic

13. Gained more knowledge
Introduced to “THE BACKEND”
First full-stack feature

15. A NEW HOPE
• Look at competitors data
• Find public API
(json bases)

16. What is a queue?

18. REQUIREMENTS ● PARALLEL JOBS
● FAULTS TOLERANCE
● UNIQUE DOCUMENTS
● RESCHEDULE DOCUMENTS
● AVOID RE-INSERT
● COST EFFICIENT
● JAVASCRIPT

21. CREATE TABLE tasks (
subject CHARACTER … PRIMARY KEY,
next_iteration TIMESTAMP,
attempts INTEGER,
payload JSONB
);

23. DISCOVERY
PROFILE
TRACKER
PROFILE
BUILDER

24. LEARN

25. Dataprocessing
Knowledge
300k
100M+

26. 1 WORKERS ~ 10 profiles / minute
CHALLENGE
2 WORKERS ~ 13 profiles / minute
3 WORKERS ~ 14 profiles / minute

27. SELECT subject FROM tasks
WHERE next_iteration <= NOW()
LIMIT 1;
UPDATE tasks
SET next_iteration = NOW() + interval ‘5m’
WHERE subject = ‘xxx’;

28. SELECT subject FROM tasks
WHERE next_iteration <= NOW()
LIMIT 1;
UPDATE tasks
SET next_iteration = NOW() + interval ‘5m’
WHERE subject = ‘xxx’;
WORKERS GAP

29. UPDATE tasks SET
next_iteration = NOW() + interval ‘5m’
WHERE subject in (
SELECT subject FROM tasks
WHERE next_iteration <= NOW()
LIMIT 1
FOR UPDATE SKIP LOCKED
)
RETURNING subject;

30. Dataprocessing
Knowledge
300k
1M

31. 100% CPU CREDITS LIMIT
CHALLENGE
100% DISK I/O LIMIT
~200 $/month “m4.large” machine

32. INSERT INTO tasks
(subject, next_iteration)
VALUES
(`xxx`, NOW() ),
(`yyy`, NOW() + INTERVAL ‘5m’ ),
(`zzz`, NOW() - INTERVAL ‘1y’ );

33. for (const task in stuff_to_insert) {
const exists = await query(`
SELECT subject FROM tasks
WHERE subject = '`${task}`' LIMIT 1
`);
if (!exists) {
await query("INSERT INTO ...")
}
}

34. INSERT INTO tasks
(subject, next_iteration)
VALUES
(`xxx`, NOW() ),
(`yyy`, NOW() + INTERVAL ‘5m’ ),
(`zzz`, NOW() - INTERVAL ‘1y’ )
ON CONFLICT
DO NOTHING;

36. Dataprocessing
Knowledge
300k
1M
5M

37. 40s+ to pick a document!
CHALLENGE

38. SELECT FROM tasks
WHERE next_iteration < NOW()
AND attempts < 5
ORDER BY
next_iteration ASC,
attempts ASC
LIMIT 1;

40. CREATE TABLE tasks (
subject CHARACTER … PRIMARY KEY,
next_iteration TIMESTAMP,
status INTEGER,
attempts INTEGER,
payload JSONB
);

41. SELECT FROM tasks
WHERE status = 1
AND attempts < 5
ORDER BY
next_iteration ASC,
attempts ASC
LIMIT 1;

42. CREATE INDEX pending ON tasks(
next_iteration ASC,
attempts ASC
)
WHERE (
status = 1
AND
attempts < 5
);

43. Need to update STATUS
CHALLENGE

45. UPDATE tasks SET status = 1
WHERE subject IN (
SELECT subject FROM tasks
WHERE status = 0
AND next_iteration < NOW()
FOR UPDATE SKIP LOCKED
);

46. Dataprocessing
Knowledge
300k
1M
20M
5M

47. SCALE

48. Scaling in AWS
➔ EC2 Clicking
➔ Auto Scaling Groups
➔ CloudFormation

49. Speed differences among queues
Errors (thousands of them!)
CHALLENGE

50. WITH BIG DATA
COMES BIG TROUBLES
true story

51. MEET GRAFANA

52. What to measure?

53. # How much cumulated work?
SELECT COUNT (*) FROM tasks
WHERE status = 1;
# How much planned word?
SELECT COUNT (*) FROM tasks
WHERE status = 0;
# Other metrics…

55. CREATE TABLE metrics (
metric CHARACTER … NOT NULL,
value INTEGER
);
CREATE TABLE metrics_logs (
ctime TIMESTAMP,
metric CHARACTER … NOT NULL,
increment INTEGER
);

56. INSERT INTO metrics_logs (
ctime, metric, increment
) VALUES (
NOW(), `pending`, 1
);

57. FUNCTIONS

58. FOR VAR_r IN
SELECT
DISTINCT ON (metric),
metric, increment
FROM metrics_logs
ORDER BY ctime ASC
LOOP
…
END LOOP;

59. SELECT SUM(increment) INTO VAR_sum
FROM metrics_logs
WHERE metric = VAR_r.metric;
UPDATE metrics AS t
SET value = t.value + VAR_sum
WHERE metric = VAR_r.metric;

60. Dataprocessing
Knowledge
300k
1M
20M
100M+
5M

61. TAKEAWAY
➔ With good communication comes
good results
➔ Keep it simple
➔ No limits for new knowledge
➔ Mindset, curiosity, improvements

62. CEO COO Prod. owner Developer CTO
BusinessAwareness
THE COMPANY GAP
Technical Awareness