Visitar URL original
Add Graph Analytics and Time Series guides with tests (#388) · humemai/arcadedb-embedded-python@334bf0f · GitHub
Skip to content

Commit 334bf0f

Browse files
tae898claude
andcommitted
Add Graph Analytics and Time Series guides with tests (#388)
Two DSL-first guides with runnable db.query/db.command examples, each statement checked to appear in its page and asserted by a test on the 26.10.2 engine. Adds the mkdocs nav entries, the java-api-coverage rows, the testing.md rows and the AGENTS.md rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 5b6ed21 commit 334bf0f

9 files changed

Lines changed: 1185 additions & 1 deletion

File tree

‎bindings/python/AGENTS.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,8 @@ A change to behaviour updates every place that describes it. Grep these for the
2929
| API (database, schema, transactions, queries, results, records, async, exceptions, type conversion, JVM) | `bindings/python/src/arcadedb_embedded/`, `bindings/python/tests/` | `bindings/python/docs/api/`, `bindings/python/docs/guide/core/database.md`, `bindings/python/docs/guide/core/queries.md`, `bindings/python/docs/guide/core/transactions.md` |
3030
| Vectors | same | `bindings/python/docs/guide/vectors.md`, `bindings/python/docs/api/vector.md` |
3131
| Graphs and bulk loading | same | `bindings/python/docs/guide/graphs.md`, `bindings/python/docs/api/graph_batch.md` |
32+
| Graph analytics (Graph Analytical Views, `algo.*`) | `bindings/python/tests/test_guide_graph_analytics.py` | `bindings/python/docs/guide/graph-analytics.md` |
33+
| Time series (`TIMESERIES` types, `ts.*`, `append_samples`) | `bindings/python/tests/test_guide_timeseries.py`, `bindings/python/tests/test_timeseries_sql.py` | `bindings/python/docs/guide/timeseries.md` |
3234
| Import and export | same | `bindings/python/docs/guide/import.md`, `bindings/python/docs/api/importer.md`, `bindings/python/docs/api/exporter.md` |
3335
| Server mode, operations | same | `bindings/python/docs/guide/server.md`, `bindings/python/docs/api/server.md`, `bindings/python/docs/guide/operations.md` |
3436
| Which Java API is wrapped | same | `bindings/python/docs/java-api-coverage.md`, `bindings/python/docs/api-access-methods.md` |

‎bindings/python/docs/development/testing.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,8 @@ asserts, so the list below stops at the file.
9393
| `test_graph_api.py` | Vertex, edge, and document wrappers and traversal helpers |
9494
| `test_graph_algorithms_sql.py` | `shortestPath`, `dijkstra`, and `astar` |
9595
| `test_geo_predicate_sql.py` | `geo.within` and `geo.intersects` (a boundary point returns a boolean; which one is not asserted) |
96+
| `test_guide_graph_analytics.py` | Every example in the Graph Analytics guide: Graph Analytical View create, state, update modes, rebuild and drop, and the `algo.*` calls, each statement checked to appear in the page |
97+
| `test_guide_timeseries.py` | Every example in the Time Series guide: type declaration, insert forms, `append_samples`, `ts.*` functions, buckets, continuous aggregates, each statement checked to appear in the page |
9698
| `test_timeseries_sql.py` | `CREATE TIMESERIES TYPE`, range queries, bucketing, tag filters, and `COMPACT` |
9799
| `test_materialized_view_sql.py` | Materialized view create, refresh, alter, and drop |
98100
| `test_restore_sql.py` | `RESTORE DOCUMENT` and `RESTORE VERTEX`: the record count agrees with a full scan, and the record comes back intact |
Lines changed: 257 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,257 @@
1+
# Graph Analytics Guide
2+
3+
!!! warning "Known engine issues"
4+
Open ArcadeDB bugs can return a wrong answer or refuse a lookup; see
5+
[Known Engine Issues](known-issues.md).
6+
7+
ArcadeDB ships two things for whole-graph questions such as "who matters most" or
8+
"which accounts form one cluster". The first is the `algo.*` procedures, dozens of
9+
them, which you call from openCypher. The second is the **Graph Analytical View** (GAV),
10+
an in-memory compressed copy of the graph (a CSR, compressed sparse row) that the
11+
procedures and some traversals read instead of walking records one at a time.
12+
13+
Neither one has a Python method. Both are reached with `db.command()` (the DDL) and
14+
`db.query("opencypher", ...)` (the procedures), which is the
15+
[DSL-first](../api-access-methods.md) path.
16+
17+
Every example below runs against the graph in the next section, and the test that
18+
runs them is `tests/test_guide_graph_analytics.py`. They were measured on engine
19+
26.10.2.
20+
21+
## The example graph
22+
23+
Six people and six `Follows` edges. `ada`, `bob`, `cy` and `dee` follow each other in a
24+
loop; `eve` follows `fay` and is cut off from the rest.
25+
26+
```python
27+
import arcadedb_embedded as arcadedb
28+
29+
db = arcadedb.create_database("./analytics_db")
30+
31+
db.command("sql", "CREATE VERTEX TYPE Person")
32+
db.command("sql", "CREATE EDGE TYPE Follows")
33+
db.command("sql", "CREATE PROPERTY Follows.weight DOUBLE")
34+
35+
edges = [
36+
("ada", "bob", 1.0),
37+
("bob", "cy", 1.0),
38+
("ada", "cy", 5.0),
39+
("cy", "dee", 1.0),
40+
("dee", "ada", 1.0),
41+
("eve", "fay", 1.0),
42+
]
43+
with db.transaction():
44+
for name in ["ada", "bob", "cy", "dee", "eve", "fay"]:
45+
db.command("sql", "CREATE VERTEX Person SET name = ?", name)
46+
for src, dst, weight in edges:
47+
db.command("sql", "CREATE EDGE Follows FROM (SELECT FROM Person WHERE name = ?) TO (SELECT FROM Person WHERE name = ?) SET weight = ?", src, dst, weight)
48+
```
49+
50+
## The procedures work without a view
51+
52+
`algo.*` procedures run on any graph. Without a view they read the vertices and edges
53+
from storage; with a view that covers the graph they read the CSR. The answers are the
54+
same, so you can start without a view and add one when a call gets slow.
55+
56+
### PageRank
57+
58+
```python
59+
rows = db.query("opencypher", "CALL algo.pagerank() YIELD node, score RETURN node.name AS name, round(score * 1000) / 1000.0 AS score ORDER BY score DESC, name").to_list()
60+
# [{'name': 'cy', 'score': 0.259}, {'name': 'dee', 'score': 0.254},
61+
# {'name': 'ada', 'score': 0.25}, {'name': 'bob', 'score': 0.14},
62+
# {'name': 'fay', 'score': 0.063}, {'name': 'eve', 'score': 0.034}]
63+
```
64+
65+
Options go in one map. `tolerance`, `weightProperty` and `direction` (`OUT`, `IN` or
66+
`BOTH`; default `OUT`) are the other keys.
67+
68+
```python
69+
db.query("opencypher", "CALL algo.pagerank({dampingFactor: 0.9, maxIterations: 50}) YIELD node, score RETURN node.name AS name ORDER BY score DESC, name LIMIT 1").to_list()
70+
# [{'name': 'cy'}]
71+
```
72+
73+
### Connected components
74+
75+
`algo.wcc()` ignores edge direction; `algo.scc()` respects it.
76+
77+
```python
78+
db.query("opencypher", "CALL algo.wcc() YIELD node, componentId RETURN componentId, collect(node.name) AS members").to_list()
79+
# two components: ada, bob, cy, dee and eve, fay
80+
db.query("opencypher", "CALL algo.scc() YIELD node, componentId RETURN componentId, collect(node.name) AS members").to_list()
81+
# three: ada, bob, cy, dee and eve and fay
82+
```
83+
84+
The order of the `members` list and the numbering of `componentId` are not defined;
85+
sort them in Python before you compare.
86+
87+
### Breadth-first search
88+
89+
`algo.bfs(start, relationshipTypes, direction, maxDepth)` takes a vertex, so bind it
90+
with `MATCH` first.
91+
92+
```python
93+
db.query("opencypher", "MATCH (a:Person {name: 'ada'}) CALL algo.bfs(a, 'Follows', 'OUT', 3) YIELD node, depth RETURN node.name AS name, depth ORDER BY depth, name").to_list()
94+
# [{'name': 'bob', 'depth': 1}, {'name': 'cy', 'depth': 1}, {'name': 'dee', 'depth': 2}]
95+
```
96+
97+
The start vertex is not in the output.
98+
99+
### Weighted shortest path
100+
101+
`algo.dijkstra(start, end, relationshipType, weightProperty, direction)` returns the
102+
path and its total weight. The direct `ada -> cy` edge costs 5, the route through `bob`
103+
costs 2, so the cheap route wins.
104+
105+
```python
106+
db.query("opencypher", "MATCH (a:Person {name: 'ada'}), (d:Person {name: 'dee'}) CALL algo.dijkstra(a, d, 'Follows', 'weight', 'OUT') YIELD path, weight RETURN [n IN nodes(path) | n.name] AS names, weight").to_list()
107+
# [{'names': ['ada', 'bob', 'cy', 'dee'], 'weight': 3.0}]
108+
```
109+
110+
### Degree and triangles
111+
112+
```python
113+
db.query("opencypher", "CALL algo.degree() YIELD node, inDegree, outDegree RETURN node.name AS name, inDegree, outDegree ORDER BY name").to_list()
114+
# ada 1/2, bob 1/1, cy 2/1, dee 1/1, eve 0/1, fay 1/0 (in/out)
115+
db.query("opencypher", "CALL algo.triangleCount() YIELD node, triangles RETURN node.name AS name, triangles ORDER BY name").to_list()
116+
# ada 2, bob 1, cy 2, dee 1, eve 0, fay 0
117+
```
118+
119+
`algo.triangleCount` counts triangles with edge direction ignored.
120+
121+
### Communities and centrality
122+
123+
```python
124+
db.query("opencypher", "CALL algo.louvain() YIELD node, communityId RETURN communityId, collect(node.name) AS members").to_list()
125+
# eve and fay share a community; none of the other four is in it
126+
db.query("opencypher", "CALL algo.betweenness() YIELD node, score RETURN node.name AS name, score ORDER BY score DESC, name").to_list()
127+
# eve and fay score 0.0; no score is above 0.15 on this graph
128+
```
129+
130+
Also run on this graph: `algo.kcore()` and `algo.labelPropagation()`. The engine's
131+
procedure list is much longer (similarity, centrality, flow, embeddings); see the
132+
engine reference below for the names and arguments, and run an unfamiliar one on a
133+
small graph before you rely on it.
134+
135+
## Graph Analytical Views
136+
137+
A view is a named, persisted definition plus an in-memory CSR that is built from it.
138+
139+
```python
140+
db.command("sql", "CREATE GRAPH ANALYTICAL VIEW social VERTEX TYPES (Person) EDGE TYPES (Follows) PROPERTIES (name) EDGE PROPERTIES (weight) UPDATE MODE SYNCHRONOUS")
141+
# {'operation': 'create graph analytical view', 'name': 'social', 'created': True}
142+
```
143+
144+
| Clause | Meaning |
145+
| --- | --- |
146+
| `VERTEX TYPES (...)`, `EDGE TYPES (...)` | What the view contains. Leave a clause out to include every type. |
147+
| `PROPERTIES (...)` | Vertex properties copied into the view. Omit to copy all, `PROPERTIES ()` to copy none. |
148+
| `EDGE PROPERTIES (...)` | Edge properties copied in, for weighted algorithms. |
149+
| `UPDATE MODE OFF`, `SYNCHRONOUS`, `ASYNCHRONOUS` | How the view follows later writes (below). Default `OFF`. |
150+
| `COMPACTION THRESHOLD n` | How many pending changes the view collects before it folds them into the CSR. |
151+
| `IF NOT EXISTS` | Returns `created: False` instead of failing when the name is taken. |
152+
153+
`CREATE` returns at once and the CSR is built in the background. Read the state from
154+
`schema:graphAnalyticalViews`:
155+
156+
```python
157+
import time
158+
159+
def wait_ready(db, name="social"):
160+
while True:
161+
row = db.query("sql", "SELECT status, nodeCount, edgeCount, updateMode FROM schema:graphAnalyticalViews WHERE name = 'social'").first()
162+
if row.get("status") == "READY":
163+
return row.to_dict()
164+
time.sleep(0.05)
165+
166+
wait_ready(db)
167+
# {'status': 'READY', 'nodeCount': 6, 'edgeCount': 6, 'updateMode': 'SYNCHRONOUS'}
168+
```
169+
170+
A view that is not `READY` does not serve a call: the `OFF` example below shows a
171+
procedure answering correctly from the live graph while its view is `STALE`.
172+
173+
### Update modes
174+
175+
| Mode | After a commit that changes the graph |
176+
| --- | --- |
177+
| `SYNCHRONOUS` | The change is in the view when `commit()` returns. Slows commits. |
178+
| `ASYNCHRONOUS` | The view folds the change in afterwards: `status` shows `BUILDING`, then `READY`. Poll for it as above. |
179+
| `OFF` | The view is marked `STALE` and keeps its old counts. Procedures scan the live graph, so answers stay correct and the speed-up is gone until you rebuild. |
180+
181+
```python
182+
db.command("sql", "ALTER GRAPH ANALYTICAL VIEW social UPDATE MODE OFF")
183+
with db.transaction():
184+
db.command("sql", "CREATE VERTEX Person SET name = 'hal'")
185+
# the view now reports status STALE and still counts the old 7 vertices
186+
db.query("opencypher", "CALL algo.wcc() YIELD node RETURN count(node) AS n").to_list()
187+
# [{'n': 8}] the call scanned the live graph, so it sees 'hal'
188+
db.command("sql", "REBUILD GRAPH ANALYTICAL VIEW social")
189+
# status READY again, nodeCount 8
190+
```
191+
192+
The `ASYNCHRONOUS` form, with the compaction setting:
193+
194+
```python
195+
db.command("sql", "CREATE GRAPH ANALYTICAL VIEW social VERTEX TYPES (Person) EDGE TYPES (Follows) UPDATE MODE ASYNCHRONOUS COMPACTION THRESHOLD 500")
196+
```
197+
198+
`ALTER GRAPH ANALYTICAL VIEW name COMPACTION THRESHOLD n` changes the other setting.
199+
After a stretch in `OFF`, `REBUILD` before you rely on the speed-up again.
200+
201+
### What a ready view serves
202+
203+
Whole-graph `algo.*` calls use the view when it is `READY` and covers **every** vertex
204+
and edge type of the database. A view that leaves a type out (say, a `City` type next
205+
to `Person`) is not used for those calls: they scan and still count the `City`
206+
vertices. For a call that names its edge types (`algo.bfs(a, 'Follows', ...)`) the view
207+
only has to cover those.
208+
209+
The engine also has view-backed paths for some counts and traversals (the SQL
210+
`shortestPath` and `astar` functions, for one), but this page does not rely on them:
211+
`EXPLAIN` of a count prints `COUNT CHAIN PATHS (CSR count-push-down)` with the view and
212+
without it, so the plan text does not show whether a view served the call. Compare
213+
timings on your own graph instead (a laptop timing needs a quiet machine and several
214+
repeats), and expect the same answers either way.
215+
216+
A view that covers only `Person` next to a `City` type:
217+
218+
```python
219+
db.command("sql", "CREATE GRAPH ANALYTICAL VIEW people VERTEX TYPES (Person) EDGE TYPES (Follows)")
220+
db.query("opencypher", "CALL algo.degree() YIELD node RETURN count(node) AS n").to_list()
221+
# counts the City vertices too: the view is not used for this call
222+
```
223+
224+
### Drop
225+
226+
```python
227+
db.command("sql", "DROP GRAPH ANALYTICAL VIEW social")
228+
# {'operation': 'drop graph analytical view', 'name': 'social', 'dropped': True}
229+
db.command("sql", "DROP GRAPH ANALYTICAL VIEW IF EXISTS social") # no error if missing
230+
```
231+
232+
`CREATE GRAPH ANALYTICAL VIEW IF NOT EXISTS social` answers `created: False` for a name
233+
that is taken, where plain `CREATE` raises `ArcadeDBError` ("already exists"). An
234+
unknown mode in `ALTER ... UPDATE MODE` and a `DROP` of a missing view raise too.
235+
236+
## Caveats
237+
238+
- A view is memory. `memoryUsageBytes` in `schema:graphAnalyticalViews` shows its size;
239+
create one for the types you query, not for everything.
240+
- `SYNCHRONOUS` costs write throughput on every commit; `ASYNCHRONOUS` can lag behind
241+
the commit, so do not read your own write through a view without polling.
242+
- `OFF` is the default. A view you create without `UPDATE MODE` goes stale on the
243+
first write.
244+
- Left out of this page because they were not run here: contraction hierarchies
245+
(`CCH (...)` and `algo.cchShortestPath`), what a view serves inside a transaction
246+
that has uncommitted changes, whether a weighted PageRank uses a view, and a view
247+
across a database reopen.
248+
249+
## References
250+
251+
- [ArcadeDB documentation](https://docs.arcadedb.com/) for the full `algo.*` list and
252+
the Graph Analytical View chapter.
253+
- The grammar of the statements:
254+
[`SQLParser.g4`](https://github.com/ArcadeData/arcadedb/blob/main/engine/src/main/antlr4/com/arcadedb/query/sql/grammar/SQLParser.g4)
255+
(search for `GRAPH ANALYTICAL VIEW`).
256+
- [Graph Operations](graphs.md) for building the graph, and
257+
[Queries](core/queries.md) for the two query languages.

‎bindings/python/docs/guide/graphs.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,9 @@ vertex only:
5252
If a type is often queried from its target, declare it two-way (the default): a pattern
5353
from the target of a one-way type pays a scan of the whole edge type.
5454

55+
For whole-graph questions (PageRank, components, shortest paths) and the Graph
56+
Analytical View that speeds them up, see the [Graph Analytics guide](graph-analytics.md).
57+
5558
## Overview
5659

5760
ArcadeDB's graph model consists of:

0 commit comments

Comments
 (0)