bq_sushi#6

1. BigQuery
@tyakachika

2. • (Masachika Yamaguchi)
• twitter @tyakachika
•
•
• Innova Structure(2015/04 ~ )
• DATUM STUDIO(2016/12 ~ )
• Toreta(2017/07 ) <-

3. • Toreta
•
•

10. Toreta

12. MySQL
BigQuery
Access log
iOS
MySQL to
BigQuery

14. ER
ER

18. …
esa

20. •
•
•

22. • BigQuery console
• Tableau
• Python(Jupyter notebook)
• R(Rmarkdown)

23. Python(Jupyter Notebook)
R
Tableau

25. •
•
•
•

26. •
•
•
•

27. 2

29. Airflow
…

31. digdag
• embulk (mysql postgresql )
• python ruby
•

32. & task
digdag-Server
local

33. Schedule.dig
TOP Parent Child
import_to_bigquery/main.dig
bigquery_to_salesforce/main.dig
execute_query/main.dig
backup/main.dig
ga/main.dig
incremental/main.dig
sync_event/main.dig
call
……
※
call
digdag-Server
digdag

34. 🎊
digdag

36. • Change log
•

37. •
•
•
•
• etc…
event
(create/update)
Change log
&Reservation

38. Reservation create/update
Sidekiq job
kinesis
event (json) put
CloudWatch lambda
S3
Schedule
Records
schedule
Records
1 put
event
gcs

39. GCP
• GCS
• GCS BigQuery load

40. GCS
• S3
• AWS access_key secret_key
• GCS

41. • Partition table Partition
• Append Delete/
Update
GCS BigQuery load

42. Partition table

43. (table_name_YYYYMMDD)
(Partition table)
BigQuery

44. (table_name_YYYYMMDD)
•
• Alter Table
•
• Web Console
•

45. (Partition table)
•
• Web Console
•
• Alter Table …

46. Alter Table

47. • Web Console Add New Fields
• Patch table or update table
• …
•
Alter Table

48. 1. MySQL
2.BigQuery
3.
5. or SQL
4. tmp
6. SQL
7.
8.tmp
9.tmp
Python Operater

49. • BigQuery API Error
• ->digdag retry retry
• digdag retry
• ->yml

53. • Table Detail Description
• Overwrite Description
• API Description
• markdown esa

54. …
BI

55. redash …
…

57. •
•
•
•
redash

59. metabase

60. • SQL
metabase

61. • metrics
• segment
• description
metabase

62. • redash
•
•
• https://github.com/metabase/metabase/issues/6275
•

63. • BigQuery gcp
•
•
• BigQuery

64. • …
•