There are a lot of solutions for querying JSON data available, most of which are proprietary and require a steep learning curve. Couchbase's N1QL (Non-First Normal Form Query Language) is a very powerful query language built on top of the SQL we all know and love (well, mostly love). It's really amazing how easy N1QL is for current SQL users. In this session, we'll delve into the differences between SQL and N1QL, learning how it layers new features on top of ANSI SQL to support nested data and JSON types. We'll also go in depth into indexing JSON data using Couchbase, covering how to design and troubleshoot your indexes to drive spectacular performance at scale.

1. Querying Nested
JSON Data Using
N1QL and
Couchbase
TriNUG Data SIG
6/6/2018

2. Who is this guy?
• Brant Burnett - @btburnett3
• Systems Architect at CenterEdge Software
• .NET since 1.0, SQL Server since 7.0
• MCSD, MCDBA
• Experience from desktop apps to large
scale cloud services

3. NoSQL Credentials
• Couchbase user since 2012 (v1.8)
• Couchbase Community Expert
• Open source contributions:
• Couchbase .NET SDK
• Couchbase.Extensions for .NET Core
• Couchbase LINQ provider (Linq2Couchbase)
• CouchbaseFakeIt
• couchbase-index-manager

4. Content
Attributions
• Matthew Groves
Couchbase Developer Advocate
@mgroves
crosscuttingconcerns.com

5. What is Couchbase
• NoSQL document database
• Get and set documents by key
• Imagine a giant folder full of JSON files
• If you know the filename, you can get or
update the content
• Additional features:
• Query using N1QL (SQL-based)
• Map-Reduce Views
• Full Text Search
• Analytics (Preview in 5.5)
• Eventing (5.5)
• Couchbase is not CouchDB

6. Why Couchbase
• Scalability
• Availability
• Performance
• Agility

7. Agenda
Introduction to N1QL
Working with JSON Types
Joins In N1QL
Indexing in Couchbase
Query Optimization

8. Introduction to N1QL Pronounced “nickel”

9. What’s a Bucket?
• Large collection of JSON
documents
• Every document may have a
different schema
• Documents are accessed by a
string called the key

10. CustomerID Name DOB
CBL2015 Jane Smith 1990-01-30
Table: Customer
{
"Name": "Jane Smith",
"DOB": "1990-01-30",
"type": "customer"
}
Document Key: customer-CBL2015

11. So how do I query data from a bucket?
This Photo by Unknown Author is licensed under CC BY-SA

12. {
"Name": "John Smith",
"DOB": "1990-06-29",
"type": "customer"
}
Document Key: customer-CBL2016
SELECT Name, DOB FROM Bucket
WHERE type = 'customer' AND Name LIKE '%Smith'
ORDER BY Name
[{
"Name": "Jane Smith",
"DOB": "1990-01-30"
},
{
"Name": "John Smith",
"DOB": "1990-06-29"
}]
{
"Name": "Jane Smith",
"DOB": "1990-01-30",
"type": "customer"
}
Document Key: customer-CBL2015

13. What other SQL features are
supported?
• Aggregation (MIN, MAX, SUM, AVG, COUNT, etc)
• GROUP BY/HAVING
• OFFSET/LIMIT
• Subqueries
• UNION/INTERSECT/EXCEPT
• Joins (more details to come…)
• UPDATE/INSERT/DELETE/UPSERT

14. Accessing Nested Objects
Key: airport_3484
{
"airportname": "Los Angeles Intl",
"city": "Los Angeles",
"country": "United States",
"faa": "LAX",
"geo": {
"alt": 126,
"lat": 33.942536,
"lon": -118.408075
},
"icao": "KLAX",
"id": 3484,
"type": "airport",
"tz": "America/Los_Angeles"
}
SELECT *
FROM `travel-sample`
WHERE type = 'airport' AND geo.alt < 1000
SELECT *
FROM `travel-sample`
WHERE type = 'route'
AND schedules[0].day = 1

15. O backtick, backtick!
Wherefore art thou a
backtick?
• ANSI SQL delimits identifiers with double quotes
• SELECT * FROM "table-name"
• T-SQL also delimits identifiers with square
brackets
• SELECT * FROM [table-name]
• Both of these are used in JSON!
• {"array": ["string1", "string2"]}
• So, N1QL uses the backtick instead
• SELECT * FROM `bucket-name`
This Photo by Unknown Author is licensed under CC BY-NC-ND

16. Working with
JSON Types

17. Strings
Supported by JSON
Collation is always
case sensitive
1
Literals are delimited with
either double or single
quotes
x = 'my string here’
x = "my string here"
2
Various supporting
functions
•String concatenation (||)
•LENGTH
•LOWER
•CONTAINS
•TRIM, etc…
3

18. Numbers
Supported by JSON
1
Literals are included
without delimiters
x = 123456.05124
2
Various supporting
functions
• Arithmetic operators
• ABS
• CEIL
• SQRT
• TRUNC, etc...
3

19. Booleans
Supported by JSON
1
Literals are true and
false
x = true
2
Various supporting
operators
• NOT
• AND
• OR
3

20. Arrays
Supported by JSON
1
Literals are comma
delimited and surrounded
by square brackets
[1, 2, 3, "a"]
2
Various supporting
functions
• subqueries
• ARRAY_CONTAINS
• ARRAY_AVG
• ARRAY_INSERT
• ARRAY_LENGTH, etc…
3

21. Objects
Supported by JSON
1
Literals are comma
delimited key/value pairs
surrounded by curly
braces
{"key": "value"}
2
Various supporting
functions
• OBJECT _NAMES
• OBJECT_PAIRS
• OBJECT_VALUES, etc…
3

22. Nulls
Supported by JSON
1
Literal is the word null,
no delimiters
{"key": null}
2
Various supporting
operators and
functions
• IS NULL
• IS NOT NULL
• IFNULL, etc…
3

23. Missing attributes
Supported by JSON
Can’t be explicitly
declared
Similar to undefined in
Javascript
1
No literal, simply don’t
include an attribute in
an object
{}
2
Various supporting
operators and functions
• IS MISSING
• IS NOT MISSING
• IFMISSING
• IFMISSINGORNULL, etc…
3

24. Date/times
Not officially supported by
JSON
Can be stored using other data
types
Usually either ISO8601 string
or number of milliseconds
since the Unix epoch
1
Literal depends on the data
type
"2018-04-06T19:26:29.000Z"
1528140389000
2
Various supporting functions
• STR_TO_MILLIS
• CLOCK_MILLIS, ClOCK_STR
• DATE_PART_STR, DATE_PART_MILLIS
• DATE_DIFF_STR, DATE_DIFF_MILLIS
• etc…
3

25. Joins In N1QL

26. Key: route_10000
{
"airline": "AF",
"airlineid": "airline_137",
"destinationairport": "MRS",
"distance": 2881.617376098415,
"equipment": "320",
"id": 10000,
"schedule": [
{
"day": 0,
"flight": "AF198",
"utc": "10:13:00"
},
{
"day": 0,
"flight": "AF547",
"utc": "19:14:00"
}
],
"sourceairport": "TLV",
"stops": 0,
"type": "route"
}
Referenced 1:N Relationship
Key: airline_137
{
"callsign": "AIRFRANS",
"country": "France",
"iata": "AF",
"icao": "AFR",
"id": 137,
"name": "Air France",
"type": "airline"
}

27. Joining by Primary Key
SELECT route.sourceairport, route.destinationairport, airline.name
FROM `travel-sample` AS route
INNER JOIN `travel-sample` AS airline
ON route.airlineid = META(airline).id
WHERE route.type = 'route'
ORDER BY route.sourceairport, route.destinationairport, airline.name

28. Key: route_10000
{
"airline": "AF",
"airlineid": "airline_137",
"destinationairport": "MRS",
"distance": 2881.617376098415,
"equipment": "320",
"id": 10000,
"schedule": [
{
"day": 0,
"flight": "AF198",
"utc": "10:13:00"
},
{
"day": 0,
"flight": "AF547",
"utc": "19:14:00"
}
],
"sourceairport": "TLV",
"stops": 0,
"type": "route"
}
Referenced 1:N Relationship
Key: airline_137
{
"callsign": "AIRFRANS",
"country": "France",
"iata": "AF",
"icao": "AFR",
"id": 137,
"name": "Air France",
"type": "airline"
}

29. Joining by Attributes
SELECT route.sourceairport, route.destinationairport, airline.name
FROM `travel-sample` AS route
INNER JOIN `travel-sample` AS airline
ON route.airline = airline.iata AND airline.type = 'airline'
WHERE route.type = 'route'
ORDER BY route.sourceairport, route.destinationairport, airline.name

30. Key: route_10000
{
"airline": "AF",
"airlineid": "airline_137",
"destinationairport": "MRS",
"distance": 2881.617376098415,
"equipment": "320",
"id": 10000,
"schedule": [
{
"day": 0,
"flight": "AF198",
"utc": "10:13:00"
},
{
"day": 0,
"flight": "AF547",
"utc": "19:14:00"
}
],
"sourceairport": "TLV",
"stops": 0,
"type": "route"
}
Embedded 1:N Relationship

31. Flattening Embedded Lists
SELECT route.sourceairport, route.destinationairport, schedule.utc
FROM `travel-sample` AS route
UNNEST route.schedule AS schedule
WHERE route.type = 'route' AND schedule.day = 0
ORDER BY route.sourceairport, route.destinationairport, schedule.utc

32. My data’s not flat, why are my queries?
This Photo by Unknown Author is licensed under CC BY-SA

33. Key: route_50490
{
"airline": "SQ",
"airlineid": "airline_4435",
"destinationairport": "ORD",
"distance": 2802.1171926467396,
"equipment": "320",
"id": 50490,
"schedule": [{
"day": 0,
"flight": "SQ279",
"utc": "15:13:00"
}, {
"day": 0,
"flight": "SQ835",
"utc": "21:10:00"
}],
"sourceairport": "LAX",
"stops": 0,
"type": "route"
}
Referenced 1:N Relationship
Key: airport_3484
{
"airportname": "Los Angeles Intl",
"city": "Los Angeles",
"country": "United States",
"faa": "LAX",
"geo": {
"alt": 126,
"lat": 33.942536,
"lon": -118.408075
},
"icao": "KLAX",
"id": 3484,
"type": "airport",
"tz": "America/Los_Angeles"
}

34. Nesting (a.k.a. LINQ GroupJoin)
SELECT
airport.*,
(SELECT RAW r2.destinationairport FROM routes AS r2) AS destinations
FROM `travel-sample` AS airport
INNER NEST `travel-sample` AS routes
ON airport.faa = routes.sourceairport AND routes.type = 'route'
WHERE airport.type = 'airport'
AND airport.airportname LIKE 'Los Angeles%'

35. Nesting (a.k.a. LINQ GroupJoin)
{
"airportname": "Los Angeles Intl",
"city": "Los Angeles",
"country": "United States",
"destinations": ["PHX", "SEA", "MCO", "ATL", "SYD", "YYZ", "LIM", "LHR", "IND", "CLE", "..."],
"faa": "LAX",
"geo": {
"alt": 126,
"lat": 33.942536,
"lon": -118.408075
},
"icao": "KLAX",
"id": 3484,
"type": "airport",
"tz": "America/Los_Angeles"
}

36. Indexing in
Couchbase
Global Secondary Indexes
a.k.a. GSI

37. The Primary Index
CREATE PRIMARY INDEX ON bucket SELECT * FROM bucket

38. Single Attribute Index
CREATE INDEX docsByName
ON bucket (name)
SELECT * FROM bucket
WHERE name LIKE 'A%'
SELECT * FROM bucket
WHERE name >= 'A' AND name < 'N'

39. Multiple Attribute Index
CREATE INDEX docsByNames ON bucket
(lastName, firstName)
SELECT * FROM bucket
WHERE lastName LIKE 'A%'
SELECT * FROM bucket
WHERE lastName = 'Burnett'
AND firstName LIKE ‘B%'

40. Expression Index
CREATE INDEX docsByName ON bucket
(LOWER(lastName), LOWER(firstName))
SELECT * FROM bucket
WHERE LOWER(lastName) LIKE 'a%'
SELECT * FROM bucket
WHERE LOWER(lastName) = 'burnett'
AND LOWER(firstName) LIKE 'b%'

41. Filtered Index
CREATE INDEX custsByName ON bucket
(LOWER(lastName), LOWER(firstName))
WHERE type = 'customer'
SELECT * FROM bucket
WHERE LOWER(lastName) LIKE 'a%'
AND type = 'customer'
SELECT * FROM bucket
WHERE LOWER(lastName) = 'burnett'
AND LOWER(firstName) LIKE 'b%'
AND type = 'customer'

42. Array Index
CREATE INDEX custsByNickName ON bucket
(DISTINCT ARRAY p FOR p IN nickNames END)
WHERE type = 'customer’
SELECT * FROM bucket
WHERE ANY p IN nickNames SATISFIES p = 'Buzz' END
AND type = 'customer'
CREATE INDEX custsByNickName ON bucket
(DISTINCT ARRAY LOWER(p) FOR p IN nickNames END)
WHERE type = 'customer’
SELECT * FROM bucket
WHERE ANY p IN nickNames SATISFIES LOWER(p) = 'buzz' END
AND type = 'customer'

43. Index Nodes
Node B
Index Architecture
Data Nodes
Node A
DCP
DCP
Index 1
Index 2
Replica
Index 3
Index 1
Replica
Index 2 Index 4

44. Deferring Index Build
CREATE INDEX docsByName
ON bucket (name)
WITH {"defer_build": true}
CREATE INDEX docsByNames
ON bucket (lastName, firstName)
WITH {"defer_build": true}
BUILD INDEX ON bucket
(docsByName, docsByNames)

45. Replicated Index
CREATE INDEX custsByName ON bucket
(LOWER(lastName), LOWER(firstName))
WHERE type = 'customer'
WITH {"num_replica": 1}
SELECT * FROM bucket
WHERE LOWER(lastName) LIKE 'a%’
AND type = 'customer'
SELECT * FROM bucket
WHERE LOWER(lastName) = 'burnett'
AND LOWER(firstName) LIKE 'b%’
AND type = 'customer'

46. Partitioned Index
CREATE INDEX custsByName ON bucket
(LOWER(lastName), LOWER(firstName))
WHERE type = 'customer'
PARTITION BY hash(tenantId)
WITH {"num_replica": 1}
SELECT * FROM bucket
WHERE LOWER(lastName) LIKE 'a%’
AND type = 'customer'
SELECT * FROM bucket
WHERE LOWER(lastName) = 'burnett'
AND tenantId = 123456
AND type = 'customer'

47. Query Optimization

48. Index Selection Criteria
• All predicates on the index must be included in
the query
• The first index expression must be in the
predicate
• Chooses the index with the most matching
expressions
• If more than one option, chooses one at random
for load balancing
• Does not use statistics for optimization (yet…)

49. Query Node
Query Process (a simplified subset)
Data Nodes
Index Node
1. Incoming Query 7. Query Result
2. Query Plan
7. Filter, Sort, Agg, etc

50. Live Demo!
This should be interesting…
This Photo by Unknown Author is licensed under CC BY-NC-SA

51. Nested Loop vs Hash Join in C#
Nested Loop Join
IEnumerable<RouteAirlines> Join(
IList<Route> routes, IList<Airline> airlines)
{
foreach (var route in routes)
{
var routeAirlines = new RouteAirlines
{
Route = route,
Airlines = new List<Airline>()
};
foreach (var airline in airlines)
{
if (airline.Iata == route.Airline) {
routeAirlines.Add(airline);
}
}
yield return routeAirlines;
}
}
Hash Join
IEnumerable<RouteAirlines> Join(
IList<Route> routes, IList<Airline> airlines)
{
var hashTable = airlines.ToLookup(p => p.Iata);
foreach (var route in routes)
{
var routeAirlines = new RouteAirlines
{
Route = route,
Airlines = hashTable[route.Airline].ToList()
};
yield return routeAirlines;
}
}

52. N1QL Hash Join
SELECT route.sourceairport, route.destinationairport, airline.name
FROM `travel-sample` AS route
INNER JOIN `travel-sample` AS airline USE HASH(build)
ON route.airline = airline.iata AND airline.type = 'airline'
WHERE route.type = 'route'
ORDER BY route.sourceairport, route.destinationairport, airline.name

53. Key Optimization Takeaways
Make sure fetch
is no larger than
necessary
1
Design covering
indexes where
possible
2
Watch out for
pagination
3
Consider USE
HASH where
applicable
4
Keep joins to a
minimum
5

54. Thanks for Coming!

55. Questions?
This Photo by Unknown Author is licensed under CC BY-NC-ND