This function is our main entry point to linea-based transformations.
It traces the given code, executing it and writing the results to the DB.
It does the following things in order:
-
accepts the source code and writes it to the db after proper encapsulation
-
determines the transformations that need to be done on this source. The additions currently
are mostly dependent on the python runtime version but can be extended to other variables in
the future. These transforms will be run in order with each transformer fixing the ast so
that the final transformer/s (NodeTransformer and others in future) which do most of the
conversion to linea graph are agnostic of python versions or source format changes.
-
parse code and walk the first level of the ast generated from code. Note that since this is only a first
level walk, all transformers are supposed to walk any sub-trees (eg. check out visit_Index
in py38transformer. not visiting the value will result only in first-level ast to be mutated
leaving the children untouched)
-
Finally writes the linea graph generated from this ast to linea DB
It returns the node corresponding to the last statement in the code,
if it exists.
Source code in lineapy/transformer/transform_code.py
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118 | def transform(
code: str, location: SourceCodeLocation, tracer: Tracer
) -> Optional[Node]:
"""
This function is our main entry point to linea-based transformations.
It traces the given code, executing it and writing the results to the DB.
It does the following things in order:
1. accepts the source code and writes it to the db after proper encapsulation
2. determines the transformations that need to be done on this source. The additions currently
are mostly dependent on the python runtime version but can be extended to other variables in
the future. These transforms will be run in order with each transformer fixing the ast so
that the final transformer/s (NodeTransformer and others in future) which do most of the
conversion to linea graph are agnostic of python versions or source format changes.
3. parse code and walk the first level of the ast generated from code. Note that since this is only a first
level walk, all transformers are supposed to walk any sub-trees (eg. check out visit_Index
in py38transformer. not visiting the value will result only in first-level ast to be mutated
leaving the children untouched)
4. Finally writes the linea graph generated from this ast to linea DB
It returns the node corresponding to the last statement in the code,
if it exists.
"""
# create sourcecode object and register source code to db
src = SourceCode(id=get_new_id(), code=code, location=location)
tracer.db.write_source_code(src)
# defaults for executor that were set inside node transformer
# Set __file__ to the pathname of the file
if isinstance(location, Path):
tracer.executor.module_file = str(location)
# initialize the transformer IN ORDER of preference
transformers: List[BaseTransformer] = []
# python 3.7 handler
if sys.version_info < (3, 8):
transformers.append(Py37Transformer(src, tracer))
# python 3.8 handler
if sys.version_info < (3, 9):
transformers.append(Py38Transformer(src, tracer))
# newer conditional transformers
# FIXME these done work so they have been removed for now
# transformers.append(ConditionalTransformer(src, tracer))
# main transformation handler
transformers.append(NodeTransformer(src, tracer))
# parse the usercode in preparation for visits
try:
tree = ast.parse(
code,
str(get_location_path(location).absolute()),
)
except SyntaxError as e:
raise UserException(e, RemoveFrames(2))
if sys.version_info < (3, 8):
from asttokens import ASTTokens
from lineapy.transformer.source_giver import SourceGiver
# if python version is 3.7 or below, we need to run the source_giver
# to add the end_lineno's to the nodes. We do this in two steps - first
# the asttoken lib does its thing and adds tokens to the nodes
# and then we swoop in and copy the end_lineno from the tokens
# and claim credit for their hard work
ASTTokens(code, parse=False, tree=tree)
SourceGiver().transform(tree)
# walk the parsed tree through every transformer in the list
if len(tree.body) > 0:
for stmt in tree.body:
res = None
for trans in transformers:
res = trans.visit(stmt)
# swap the node with the output of previous transformer and use that for further calls
# or some other statement to figure out whether the node is properly processed or not
if res is not None:
stmt = res
# if no transformers can process it - we'll change node transformer to not throw not implemented exception
# so that it can be extended and move it here.
if isinstance(res, ast.AST):
raise NotImplementedError(
f"Don't know how to transform {type(stmt).__name__}"
)
last_statement_result = res
tracer.db.commit()
return last_statement_result
return None
|