sdk/python/Language Features/PyTokenize

PyTokenize

Python stdlib module — tokenize. Auto-indexed from CDN. Method-level security roles have not been annotated; rule writers should inspect the source before use.

Other Methods

.ISEOF()Neutral

Signature

ISEOF(x)

ISEOF function.

.ISNONTERMINAL()Neutral

Signature

ISNONTERMINAL(x)

ISNONTERMINAL function.

.ISTERMINAL()Neutral

Signature

ISTERMINAL(x)

ISTERMINAL function.

.any()Neutral

Signature

any(choices)

any function.

.detect_encoding()Neutral

Signature

detect_encoding(readline)

The detect_encoding() function is used to detect the encoding that should

.generate_tokens()Neutral

Signature

generate_tokens(readline)

Tokenize a source reading Python code as unicode strings.

.group()Neutral

Signature

group(choices)

group function.

.lookup()Neutral

Signature

lookup(encoding)

Looks up a codec tuple in the Python codec registry and returns a CodecInfo object.

.main()Neutral

Signature

main()

main function.

.maybe()Neutral

Signature

maybe(choices)

maybe function.

Fully-Qualified Names

FQN	Field
tokenize	fqns[0]

Wrong FQN → 0 findings. Verify with: change fqns to garbage → must produce 0 results.

Import

rule.py

import tokenize

← PreviousPyToken

Next →PyToposort