summaryrefslogtreecommitdiff
path: root/src/tokeniser.rs
Commit message (Collapse)AuthorAge
* Rewrite tokeniserBen Bridle2025-02-27
| | | | | This commit adds a subtokenise method that creates a child tokeniser over all characters between the start and end marks.
* Add tokeniser method to mark previous char as start of tokenBen Bridle2025-02-10
| | | | | | This enables a parsing technique where characters can be marked as the start of a token after they've been eaten, instead of having to determine ahead of time via defensively peeking.
* Tidy codeBen Bridle2025-02-09
|
* Rename Position type to SourcePositionBen Bridle2025-02-09
| | | | | | This better matches the naming convention of the other source locator types, and better differentiates a SourcePosition from other kinds of position.
* Implement source unit compilation, symbol resolution, error reportingBen Bridle2025-02-05
| | | | | | | | | | This library can now carry out all stages of assembly from collecting source fragments to resolving symbols to pruning unused libraries to generating a single compiled source file. Pretty-printing of state has also been implemented in this library. The source tree hierarchy, symbol resolution errors, and file read errors can all be printed in a tidy format.
* Implement a generic source code tokeniserBen Bridle2025-02-04
This is a struct that provides various methods for consuming characters from a character stream and for tracking the provenance of each parsed token.