Document the refactoring API

[brettcannon]

I'm starting to look for a library to do source transformations and I think parso could handle it via its parso.Grammar.refactor() method, but there's not docs on how to use it.

[davidhalter]

True. We should do that. Note that this method is not going away in the future, the documentation is just not here yet. Happy to have others chime in and write it.

[brettcannon]

If I end up trying to use parso for the source transformation stuff that I'm thinking about I can look at writing the docs. I assume Jedi has the best examples of its use?

[davidhalter]

Probably. :)

[brettcannon]

Looking at the code it seems the refactor() method is meant for 1:1 statement/exression replacements, and I need 1:many for my project. So I don't expect to be using parso and thus probably won't be writing up any docs.

[davidhalter]

Understandable!

What do you mean by 1:many? I'm not sure what such an API would ideally look like. Can you maybe describe your use case?

[brettcannon]

To give context, this is for my syntactic sugar blog posts that I have been writing.

One of the things I'm going to tackle is rewriting for loops in terms of while. That means:

for a in b:
    c

becomes:

_iterator = iter(b)
_done = False
while not _done:
    try:
        a = next(_iterator)
    except StopIteration:
        _done = True
    else:
        c

As you can see, the single for statement becomes 3 statements (two assignments and a while loop). I would need to return a list of statements or something and have it understood that the full sequence of statements replace the old, single statement.

I'm at the point where no one seems to support this and I'm probably just going to have an API where I construct the nodes manually and it is up to people to manually patch their parse trees (it at least makes the coding easier for me 😁 ).

[davidhalter]

If the parso API is not enough for you, I would be really interested what an API would be enough for something like this.

I mean parso's API can do this. It's just a bit annoying. You can essentially do a (pseudocode):

code = '''
_iterator = iter({input})
_done = False
while not _done:
    try:
        {variable} = next(_iterator)
    except StopIteration:
        _done = True
    else: {suite}
'''

children = ...
dct[children[0]] = ""  # for
dct[children[2]] = ""  # in
dct[children[4]] = code.format(input=children[3], variable=children[1], suite=children[5])  # :

grammar.refactor(module, dct)

This should be perfectly fine. Of course then there's the details if you want to get it right. The suite needs to be indented and a for loop can also have an else, but you might not even care about that :)

[brettcannon]

It looks like the thing that wasn't obvious to me was I was going to have to blank out all the parts of the old statement, construct a new statement as pure text, and then let parso figure out that one of the new things inserted is a slew of new statements.

You can see what I'm doing now at https://github.com/brettcannon/desugar/blob/master/desugar/syntax.py. I could probably switch to parso since I ended up just returning a list of nodes to make things work out and I could use source_code() to get back what the code would be from the nodes (based on my understanding of the docs).

[isidentical]

@brettcannon if you actually don't care about comments etc, you could possibly use ast.unparse() and do the mutations over the AST. It is much simpler, and powerful. Instead of replacing the whole file with ast.unparse()'s output I personally do a specialized transformation (like splitting source code into lines, and then replacing the lines[old_node.lineno:new_node.lineno] = ast.unparse(new_node).splitlines() so that it doesn't do unnecesary refactoring)

[brettcannon]

@isidentical as one of the co-authors of the ast module, I almost did that. 😁 But I was trying to avoid being quite that destructive in the transformation in case people actually wanted to use the results in code and thus didn't want to lose comments.

[isidentical]

@isidentical as one of the co-authors of the ast module, I almost did that. grin But I was trying to avoid being quite that destructive in the transformation in case people actually wanted to use the results in code and thus didn't want to lose comments.

It worked ~decent enough (not perfectly) for me, only with an extra collection of comments from the source code (iterating all tokens, fetching COMMENTs within the given range of line numbers) + specialized replace over the only changed lines in case of you are interested.