summaryrefslogtreecommitdiff
path: root/src/tokeniser.rs
Commit message (Collapse)AuthorAge
* Add Tokeniser::eat_if methodBen Bridle2025-03-08
| | | | | This method is used for consuming a multi-character sequence only if that sequence matches a passed string.
* Rename Tokeniser::subtokenise to tokenise_child_spanBen Bridle2025-03-06
| | | | This is a clearer and more descriptive name for the method.
* Make Tokeniser::eat_to_delimiter method more generalBen Bridle2025-03-06
| | | | | | | | | The eat_to_delimiter method on Tokeniser has been renamed to eat_until, and it now consumes characters until a predicate returns true, not just until a particular character is reached. This means that the method can now be passed a closure that keeps track of nesting depth, and only returns true when a matching delimiter is found at the same level as the opening delimiter.
* Rewrite tokeniserBen Bridle2025-02-27
| | | | | This commit adds a subtokenise method that creates a child tokeniser over all characters between the start and end marks.
* Add tokeniser method to mark previous char as start of tokenBen Bridle2025-02-10
| | | | | | This enables a parsing technique where characters can be marked as the start of a token after they've been eaten, instead of having to determine ahead of time via defensively peeking.
* Tidy codeBen Bridle2025-02-09
|
* Rename Position type to SourcePositionBen Bridle2025-02-09
| | | | | | This better matches the naming convention of the other source locator types, and better differentiates a SourcePosition from other kinds of position.
* Implement source unit compilation, symbol resolution, error reportingBen Bridle2025-02-05
| | | | | | | | | | This library can now carry out all stages of assembly from collecting source fragments to resolving symbols to pruning unused libraries to generating a single compiled source file. Pretty-printing of state has also been implemented in this library. The source tree hierarchy, symbol resolution errors, and file read errors can all be printed in a tidy format.
* Implement a generic source code tokeniserBen Bridle2025-02-04
This is a struct that provides various methods for consuming characters from a character stream and for tracking the provenance of each parsed token.