4-year chronicles of ALLSTOCKER (a trading platform for used construction equipment and machinery). We describe how the system has evolved incrementally using Pharo smalltalk.
2. What is ALLSTOCKER?
● Online platform for trading used construction equipment
and machinery
○ Marketplace
○ Real-time Bid Auction
● Over 4000 worldwide buyers
● Over 400 machines/month listed on the site
● Most systems are built with Pharo Smalltalk
3. Smalltalk as a Hub
● We take polyglot microservices approach
○ Programming languages
■ Smalltalk, JavaScript, Ruby, Lua, Groovy, Erlang, Python
○ Databases
■ PostgreSQL, Redis, Neo4j, Tarantool, MongoDB
○ External APIs
■ Elasticsearch, SendGrid, Mixpanel, Fluentd
● Smalltalk is a great hub for leveraging these elements
4. Metrics in Four Years
● 1090+ classes
● 95 packages
● 34 prerequisites
● 285 tables
● 19400+ commits
● 78 releases
● We've come a long way, but we are still evolving.
5. Metrics Visualized
● Moose overview pyramid
● Some values
are biased by
OR-Mapper and
Test classes
● Keeping good
quality
6. 2015/02- In the Beginning
● First prototype was made in two weeks
● Only 4 prerequisites
○ Seaside
○ Glorp
○ Nagare-logger
○ AWS SDK for Smalltalk (S3, DynamoDB)
● User/Machine registration, photo uploader, watchlist
7. Nagare
● Smalltalk client to fluentd log collector
○ Customizable log level (info, debug, warn, error, etc)
○ Structured (MessagePack) logging
● Output plugins for fluentd are used
○ S3
■ storing lot of log files in time order
○ MongoDB
■ querying logs for further analysis
8. 2015/05 - Confronting Responsive Web Design
● Bootstrap3 needs many nested div tags
● Seaside way of rendering html was a burden
○ Many nested #div: message-sends hide essential parts of the code
● Mustaside
○ Seaside meets Mustache template engine
○ Rendering code got really slimmed down !
9. Mustaside
● Before ● After
html div class:'row'; with:[
html div class:'col-xs-12'; with:[
html div class:'panel panel-default'; with:[
html div class:'panel-body'; with:[
html div class:'text-center'; with:[
html paragraph with:'Upload'.
html span class:'btn btn-default fileinput-button';
with:[self renderFileUploaderOn: html].
].
].
].
].
].
].
self mustacheFrom: self
at: self fileUploaderTemplateKey
values: {
'uploadSection'->[self renderFileUploaderOn: html].
}
}
10. 2015/07 - Validation and Search Improvements
● Mold
● Elasticsearch for Pharo Smalltalk
11. Mold
● Form validation made simple
○ Declarative
○ Ajax-based validation
○ Client-side validation (with jQuery-Validation-Engine)
12. Using Mold
buildMold
mold := Mold new.
self buildMoldUserEmailField.
self buildXXX...
^mold
buildMoldUserEmailField
self mold emailField
key: #userEmailField;
addMaxSizeValidator: 20;
customize:[ :textInput |
textInput value: (self isLoggedIn ifTrue: [self mail_address]).
];
on: #user_email_address of: self.
renderFormOn: html
self mold canvas: html
● Building
● Rendering
13. Elasticsearch-Smalltalk
prepareSearch
esSearch := ESSearch new index: self index.
esSearch minScore: self minScore.
^esSearch
search
es := self prepareSearch.
es query: self buildQuery.
^ es searchFrom: self offset size: self limit
buildNameMatchQuery: words fieldName: fieldName ngramBoost:
boostValue
| phraseQuery matchQuery prefixQuery |
phraseQuery := ESMatchQuery new
matchPhrase;
field: fieldName;
query: words;
yourself.
prefixQuery := ESPrefixQuery new
field: fieldName;
query: words;
yourself.
matchQuery := ESMatchQuery new
field: fieldName, '__ngram';
query: words;
boost: boostValue;
yourself.
^ ESBoolQuery new
should: {phraseQuery. prefixQuery. matchQuery};
minimum_should_match: 1;
yourself.
● Building
● Searching
14. 2015/08 - Distributed Images
● Separated Pharo images
○ Application Server
○ Notification (Main/SMS) Server
● Communication was done by JSON-RPC
○ LtJsonRpc
○ SendGrid-Smalltalk
15. 2015/09-11 - I18N
● We need to support many languages
○ ja, en, vi, th (at that time) -> ja, en, vi, zh-cn(now)
● Nagoya (internal I18N template manager)
○ Selects localized template (xxx_<lang>.html) according to user
language
○ Terms are translated by GetText (.po/.mo files)
○ Fallback to English if localized contents are not found
16. 2015/09-11 - Enable Easy Reviews on Github
● Cypress is good, but it is difficult review pull-requests on
github site
○ So many small files (file-per-method)
○ Changes are scattered (example)
● We adopt STree
○ Chunk format with alphabetical method orders (file-per-class)
● CI with Jenkins
17. 2015/11-12 - Toward Richer Client UI
● JS and CSS files are handled with standard tool-chain
○ Grunt (at that time), npmScripts(now)
● Customized ScriptGenerator, JSLibraryManager and
CssLibraryManager for managing those file paths in Seaside
● Vue.js + Ajax
● We adopt Seaside + Teapot for API-access from JS
18. 2016/2 - Advanced Search
● Support advanced search
○ Aggregated by model number and last-update-time
○ Filtered by spec
● Elasticsearch was not enough to do complex aggregations
● Joining tables with Glorp was hard to maintain
● We adopted Graph database - Neo4j
20. Graph Model (2)
● We can traverse
complex
relationships in
Cypher query
language
21. Neo4reSt
● Neo4j database client and Object wrappers
○ Introducing Neo4reSt
db := N4GraphDb new.
node1 := db createNode: {#name-> 'ume'}.
node2 := db createNode: {#name-> 'Smalltalk'}.
relation1 := node1 relateTo: node2 type: #uses properties: {#years->18}.
db initialNode relateTo: node1 type: #people
22. 2016/2- Debug Reporting
● PharoDebug.log was not enough
○ Easily overwritten and lost
○ We cannot drill-down stacks, variables
● DebugReport catches unhandled exceptions and write html
reports antomatically
○ Like Smalltalk debugger in a web browser
24. 2016/3- Reliable Session Management
● For performance, we need to run multiple images as app-
server
● Image-based session storage is not shareable between
processes
● Redis was adopted for the session storage
25. RediStick
● A redis client supporting auto-reconnect and auto-switching
to alternate servers
stick := RsRediStick targetUrl: 'sync://localhost'.
stick connect.
stick beSticky. "Auto reconnect when server is not accessible"
stick onError: [ :e | e pass ].
stick endpoint info.
stick endpoint get: 'a'.
stick endpoint set: 'a' value: 999
26. 2016/4- Adding Behavioral Analytics Tool
● Google Analytics
● Mixpanel was newly introduced for fine-grained BA
○ Mixpanel-smalltalk was developed
27. Mixpanel-Smalltalk (1)
● Sending Events
○ Custom events can be send from anywhere in the code
event := self createBasicEvent.
event at:'photo_size' put: 'medium'.
event at:'file_name' put: fileName.
self mixpanel: #DownloadedPhoto json: photoInfo
28. Mixpanel-Smalltalk (2)
● Analyzing Events
○ Tracking events are nicely analyzable later in Mixpanel dashboard
○ You can also perform specific queries via JQL
29. 2016/7-9- Realtime Auction System
● Highly reactive realtime bidding auction system
○ Vue.js (presentation)
○ Pharo (business logic)
○ Erlang (bidding core)
● Web API + Ajax + WebSocket
● DEMO!!
31. 2017/1 Notifications via Slack
● Staff and developers like to know system activities timely
○ Order status changed
○ Sale item registered
○ Bug happened
● Phaslack was introduced to post various notification
messages from Pharo
33. 2017/2 Revamping Advanced Search
● Need to generate complex queries dynamically according to
various search options
● Hard-coded cypher queries were unmaintainable.
● SCypher was developed
○ “Manipulating Neo4j from Pharo Smalltalk” (Sample code project)
34. SCypher
user := 'user' asCypherObject.
friend := 'friend' asCypherObject.
friends := 'friends' asCypherObject.
query := CyQuery statements: {
CyMatch of: (CyRelationship start: user end: friend type: 'FRIEND').
CyWhere of: (CyExpression eq: (user prop: 'name') with: 'name'
asCypherParameter).
CyWith of: (user, ((CyFuncInvocation count: friend) as: friends)).
CyWhere of: (CyExpression gt: friends with: 10).
(CyReturn of: user) limit: 10.
}.
query cypherString. 'MATCH (user)-[:FRIEND]-(friend)
WHERE (user.name = $name)
WITH user, count(friend) AS friends
WHERE (friends > 10)
RETURN user LIMIT 10 '
● Generated query can be executed by
N4RestClient>>queryByCypher: queryString params: dictionary
35. 2017/3 Market Data Visualization
● Graphically show market price changes of specific machines
from past-to-present order data
○ For sellers and buyers
● D3.js (presentation)
● Tarantool (structured cache with Lua engine)
○ Tarantalk was introduced
36. Tarantalk
"Create a space"
tarantalk := TrTarantalk connect: 'taran:talk@localhost:3301'.
space := tarantalk ensureSpaceNamed: 'bookmarks'.
"Create a primary TREE index (string typed)"
primaryIndex := space ensurePrimaryIndexSetting: [ :options | options tree; partsTypes: #(#string)].
"Insert tuples"
space insert: #('Tarantool' 'https://tarantool.org' 'Tarantool main site').
space insert: #('Pharo' 'http://pharo.org' 'Pharo main site').
"Select"
primaryIndex selectHaving: #('Pharo')
● Asynchronous tuple stores with persistency
● We store price data and computation results for quick access
37. 2017/4-6 Replaing Glorp
● Eventually, we would like to perform joins on tables
○ Kobati (thin ORM)
■ + PostgresV3 driver
○ Get the full power of SQL!
● Master data should reside in RDB (Amazon RDS)
○ For interoperability
○ For availability
38. Kobati (1)
● Lightweight ORM library inspired by MyBatis
● Mappings are defined in XML files
● For SQL professionals
○ SQLマッピングフレームワーク「Kobati」の実戦投入
39. Kobati (2)
● Define queries in
references.xml
<sql id="selectBankAccountSql">
SELECT
account.id AS bankaccount__id,
account.asbankbranch_id AS bankaccount__asbankbranch_id,
account.account_number AS bankaccount__account_number,
account.account_type AS bankaccount__account_type,
branch.id AS bankbranch__id,
branch.name AS bankbranch__name,
branch.kana AS bankbranch__kana,
branch.branch_code AS bankbranch__branch_code,
FROM
ASBankAccount AS account
JOIN ASBankBranch AS branch ON branch.id = account.asbankbranch_id
</sql>
<select id="selectAllBankAccounts" arguments="sortCondition" resultMap="ASBankAccountMap">
SELECT * FROM (
<include refid="selectBankAccountSql"/>
) t
ORDER BY
${sortCondition.asOrderByString}
</select>
41. 2017/4-6 Processing with queue
● Message queue was needed in order to handle e-mail sending
requests reliably
○ With priority, persistency, retries
● Tarantube was selected as a lightweight message queue
○ A simple wrapper of the Tarantool queue module
○ No new server was needed
43. 2017/7-8 New Login Mechanism/Back-office Support
● SNS (facebook, google) login with JWT
■ hello.js + Seaside
■ OpenResty for processing JWT
● Order-processing system (for back-office)
○ Alpaca Forms + Seaside for generating lots of master-
detail style forms
44. 2017/11 Personalization in My Page
● My Page
○ Portal for login-users
○ Display news on the site
○ Change menu according to user role
● VueJS 1.0 -> 2.0
○ with much effort!
45. 2017/12 New Deployment Systems
● Revised deployment process for faster, continuous delivery
○ Jenkins -> Travis (with smalltalkCI)
● Kubernetes
○ All servers (pharo, middleware, and tools) are dockerized
● Setting files
○ JSON -> TOML (For better commenting)
■ with TOML package
46. 2018/1-3 Auto-login / Featured Page
● Added auto-login checkbox for simplifying login process
● Added "Special Sales" page generator
○ For specific buyers
○ URL is auto-generated and cannot be guessed
47. 2018/4-5 New Auction UI
● Auction Bidding Dashboard
○ Watch and bid many auction items at the same time in one page
● SSE + HTTP/2
○ Zinc-Server-Sent-Events
○ RediStick pubsub mode
48. 2018/7-8 Scaling out Auction System
● Scaling out by multiplexing Pharo image
○ 3 Pharo images
■ auction-1, auction-2 (app server)
■ webhook (interact with Erlang bid server)
○ Web API and Redis pubsub for mutual cummunication
● OpenResty as a load-balancer
○ Cookie-based sticky session
49. 2018/10 Google Sheets API
● Google sheets are widely used in our organization
○ For reporting, entry form, etc
● Google API Generator was used to generate our customized
GoogleSheetsApi wrapper
50. Google Sheets API
client := ASGoogleSheetsApiSpreadsheetsValues new authenticate.
range := '{1}!!{2}:{3}' format: { sheetName. fromCell. toCell }.
updateRequestBody := {
'values' -> valueRows.
'majorDimension' -> 'ROWS'
} asDictionary.
client update: range urlEncoded
spreadsheetId: sheetId
body: updateRequestBody asJsonString
options: updateOptions.
● Update values of sheet with ranges specified
51. 2019/1-2 Mail Batch Processing
● Send customized marketing mails to various targets
○ Different locales, situations, types of customers
○ With attachments, links
● Need to handle complex mail templates
● SendGrid-Smalltalk was refurbished to support V3 API
○ Dynamic templates and personalizations
52. template := (SGMailTemplate name: 'SampleMail-1') ensureRegistered.
template ensureVersionNamed: 'v1' setting: [:v |
v subject: '{{event}} ({{date}})'.
v textContent: '
{{initialGreetings}}!
{{body}}
---
{{signature}}']
● Registering a template
SendGrid-Smalltalk (1)
53. mail := SGMail forDynamicTemplate.
mail fromName: 'Info'.
mail from: 'info@softumeya.com'.
mail addPersonalizationDo: [ :p |
p addToEntryDo: [ :ent | ent address: 'aaa@aaa.com'; name: 'AAA'].
p templateData: {
'event' -> 'Smalltalk-meetup'.
'date' -> '2019/03/10'.
'initialGreetings' -> ('Hello ', p to first name).
'body' -> 'aaaaa'.
'signature' -> '[:masashi | ^umezawa]'.
} asDictionary
].
mail addPersonalizationDo: [ :p | … ] .
mail sendByTemplateNamed: 'SampleMail-1'
● Sending with template and personalizations
SendGrid-Smalltalk (2)
54. 2019/5-6 Absentee Bid
● Supports absentee bidding of auction items
● Started from a simple implementation
○ Poll RDB records periodically and call a bidding API
according to the record status
55. Future Plan
● Refine Mixpanel tracking
○ Finer analysis of your behavior
○ Query dynamically from Pharo
● Utilize accumulated market data for recommendation
○ Data analysis and visualization
● Update web form validator
○ Integrate Mold with Vue.js validation library
56. To Sum Up
● Pharo Smalltalk plays a precious part of our system
○ In 4 years, we’ve faced various problems, but
dynamic aspect of Smalltalk enables us to progress
incrementally
● We will keep the challenging project, please stay
tuned for more updates!
