Unlock The Universal Parsers: A New PicoRuby Compiler

HASUMI Hitoshi

2024-05-16

Unlock The
Universal
Parsers
A New PicoRuby
Compiler
@hasumikin
Monstarlab

[AD] “n-monthly Lambda Note” Vol.4, No.1
Learning How
Programming Languages
Control Electrical Circuits
with PicoRuby
(Japanese only)
1,760 yen (tax included)

CRuby’s parser
parse.y and prism.c: human-writable
parse.c: generated by Bison or Lrama
prism.c: no generator needed
Bison parse.c (retired)
Lrama lrama-generated parse.c
parse.y
prism.c

Prism and Lrama-generated parser
Prism
Newly developed universal Ruby parser
Portable and small footprint
Easy to integrate
Available in form of a gem
Lrama-generated parser (parse.c)
100% compliant with CRuby in any aspect
Not portable by nature

Parsers in Ruby (as of May 2024)
Ruby
Parser
CRuby (default) Lrama-generated
CRuby (experimental) Prism
JRuby Prism
TruffleRuby Prism
mruby by Matz
PicoRuby by hasumikin (me)
IRB(katakata_irb) (gem) Prism
RuboCop (gem) Parser → Prism

Parsers in Ruby (I’m working on)
Ruby
Parser
CRuby (default) Lrama-generated
CRuby (optional) Prism
JRuby Prism
TruffleRuby Prism
mruby ???????
PicoRuby ???????
IRB(katakata_irb) (gem) Prism
RuboCop (gem) Parser → Prism

mruby compiler generates VM code
$ echo 'puts "Hello, World!"'
00000000: 5249 5445 3033 3030
00000010: 3030 3030 4952 4550
00000020: 0000 0036 0000 0004
00000030: 5102 002d 0100 0138
00000040: 656c 6c6f 2c20 576f
00000050: 0470 7574 7300 454e
| bin/mrbc - |
0000 005e 4d41
0000 0042 3033
0000 0000 0000
0169 0001 0000
726c 6421 0000
4400 0000 0008
xxd #
545a
3030
000a
0d48
0100
hexdump
RITE0300...^MATZ
0000IREP...B0300
...6............
Q..-...8.i.....H
ello, World!....
.puts.END.....
$ echo 'puts "Hello, World!"' | bin/mrbc - | bin/mruby
Hello, World!
(PicoRuby compiler does the same)

PicoRuby compiler must be small footprint
Raspberry Pi Pico
133 MHz Arm Cortex-M0+ (dual)
264 KB SRAM
2 MB Flash ROM

PicoRuby compiler must be small footprint
Raspberry Pi Pico
133 MHz Arm Cortex-M0+ (dual)
264 KB SRAM
2 MB Flash ROM
If a universal parser is not small
enough, it won’t be “universal”

$PicoRuby compiler must be small footprint $ valgrind \ --tool=massif \ --stacks=yes \ path/to/mrbc hello_world.rb #=> massif.out.12345 $ ms_print massif.out.12345 | less$

The original mruby Compiler
--------------------------------------------------------------------------------
Command:
bin/mrbc ../mruby-compiler2/fixtures/hello_world.rb
Massif arguments:
--stacks=yes
ms_print arguments: massif.out.29290
--------------------------------------------------------------------------------
KB
129.5^
:
#
|
|
#
|
#
|
#
|
#
|
@#
|
@#
|
@#
|
@#
|
::@@@@@:::@@:::::::@@:@::@:@#:
|
@@@::::::::::::::::::@:@::: @@ @ :::@ ::: :::@@:@::@:@#:
|
@ : :::: ::::: :::: @:@::: @@ @ :::@ ::: :::@@:@::@:@#:
|
@ : :::: ::::: :::: @:@::: @@ @ :::@ ::: :::@@:@::@:@#:
|
@ : :::: ::::: :::: @:@::: @@ @ :::@ ::: :::@@:@::@:@#:
|
@ : :::: ::::: :::: @:@::: @@ @ :::@ ::: :::@@:@::@:@#:
|
@ : :::: ::::: :::: @:@::: @@ @ :::@ ::: :::@@:@::@:@#:
|
@ : :::: ::::: :::: @:@::: @@ @ :::@ ::: :::@@:@::@:@#:
|
@ : :::: ::::: :::: @:@::: @@ @ :::@ ::: :::@@:@::@:@#:
| @@
@ : :::: ::::: :::: @:@::: @@ @ :::@ ::: :::@@:@::@:@#:
0 +----------------------------------------------------------------------->Mi
0
1.030

Existing PicoRuby Compiler
--------------------------------------------------------------------------------
Command:
bin/picorbc ../mruby-compiler2/fixtures/hello_world.rb
Massif arguments:
--stacks=yes
ms_print arguments: massif.out.28004
--------------------------------------------------------------------------------
KB
16.70^
:
#::::::::::@
|
|
#::::::::::@:
|
#::::::::::@:
|
#::::::::::@:
|
#::::::::::@:
|
#::::::::::@::
|
#::::::::::@::
|
#::::::::::@::
|
#::::::::::@::
|
#::::::::::@::
|
#::::::::::@::
|
#::::::::::@::
|
::
#::::::::::@::
|
:: :
#::::::::::@::
|
:: :
#::::::::::@::
|
:: :
#::::::::::@::
|
:: :
#::::::::::@::
|
:::::
: :::::: @@ :::: :
#::::::::::@::
|
:::: ::::::::::::::::::::::: :@ ::::::::@::::@#::::::::::@::
0 +----------------------------------------------------------------------->ki
0
219.6

mrbc_prism
--------------------------------------------------------------------------------
Command:
bin/mrbc_prism ../mruby-compiler2/fixtures/hello_world.rb
Massif arguments:
--stacks=yes
ms_print arguments: massif.out.31071
--------------------------------------------------------------------------------
KB
#
9.219^
|
#
|
#
|
#
|
@@#
|
@ #
|
::
@ #
|
::
@ #
|
:: :
@ #:
|
:: :
@ #:
|
:: :
@ #:
|
:: :
@ #:
|
:: :
@ #:
|
:: :
@ #:
|
:: :
@ #:
|
::::
: @ #:
|
::::
:
:::::
:::
::@ #:
|
@::::::
:::::::: :::: ::@:::::: ::: :::: ::::::@ #::
|
:@::::::::::::::::::::: :::::: @: :
@: ::::::: ::::::@ #::
|
::@:::::::
::::::::: :::::: @: :
@: :: :::: ::::::@ #::
0 +----------------------------------------------------------------------->ki
0
201.0

RAM consumption
Compiler
RAM consumption
mrbc (original) 129.5 KB
picorbc 16.70 KB
mrbc_prism 9.219 KB*
Figures are measured in 64 bit Linux
* mrbc_prism’s small footprint is also due to a new code generator
and possibly going to be bigger with development

$mrbc_prism mrc_irep * compile_ruby_to_mruby_vm_code_with_prism(mrc_ccontext *c) { pm_parser_t parser; uint8_t *string = "puts 'Hello, World!'" pm_parser_init(&parser, string, strlen((const char *)string), NULL); pm_node_t *root = pm_parse(&parser); mrc_irep *irep = mrc_load_exec(c, (mrc_node *)root); pm_node_destroy(&parser, root); pm_parser_free(&parser); return irep; } (For illustrative purposes only and non-functional)$ Prism, really good stuff
Well designed APIs and NODE
Portable
Small footprint
Still, I’m going to implement
Lrama-generated parser.
Why?

Prism, really good stuff
Well designed APIs and NODE
Portable
Small footprint
Still, I’m going to implement
Lrama-generated parser.
Why?

In Taipei, Taiwan last year…
me
＠koic
＠spikeolaf

no longer a Labyrinth
No longer a labyrinth

Why Lrama-generated parser?
Making parse.y readable and maintainable
Integrated error tolerance
Innovative state management of the tokenizer
In short, a new era of LALR parser generator
Computer-science-wide
achievement

OK, let me use both
Prism and Lrama-gen parser
in my PicoRuby compiler

Parsers in Ruby (I’m working on)
Ruby
Parser
CRuby (default) Lrama-generated
CRuby (optional) Prism
JRuby Prism
TruffleRuby Prism
mruby Prism and Lrama-gen
PicoRuby Prism and Lrama-gen
IRB(katakata_irb) (gem) Prism
RuboCop (gem) Parser → Prism

AST of Lrama-generated parser
$ ruby --dump=p -e'puts "Hello, World!"'
###########################################################
## Do NOT use this node dump for any purpose other than ##
## debug and research. Compatibility is not guaranteed. ##
###########################################################
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
@ NODE_SCOPE (id: 3, line: 1, location: (1,0)-(1,20))
+- nd_tbl: (empty)
+- nd_args:
|
(null node)
+- nd_body:
@ NODE_FCALL (id: 0, line: 1, location: (1,0)-(1,20))*
+- nd_mid: :puts
+- nd_args:
@ NODE_LIST (id: 2, line: 1, location: (1,5)-(1,20))
+- as.nd_alen: 1
+- nd_head:
|
@ NODE_STR (id: 1, line: 1, location: (1,5)-(1,20))
|
+- nd_lit: "Hello, World!"
+- nd_next:
(null node)

Version 0.0.0.0.1 of
mrbc_lrama
(A mruby compiler using Lrama-generated parser)

How to integrate Lrama-gen parser
$ cd ruby/ruby
$ ./autogen.sh #=> configure
$ ./configure #=> common.mk etc.
$ make
$ ls -l *.a
-rw-r--r-- 1 145797150 libruby-static.a
An archive file containing all the
object files of the Ruby interpreter

libruby-static.a
libruby-static.a
parse.y
Lrama
parse.c parse.o
node.c node.o
string.c string.o
etc.
libHOGE.so
ruby

$Even CRuby can be embedded // hello_world.c #include <ruby.h> int main(int argc, char **argv) { ruby_init(); rb_eval_string("puts 'Hello, World!'"); ruby_finalize(); return 0; } (For illustrative purposes only and non-functional)$ $Even CRuby can be embedded $ cc -o hello_world \ hello_world.c \ -I/path/to/ruby/include \ -L/path/to/ruby \ -l:libruby-static.a # ^^^^^^^^^^^^^^^^ $ ./hello_world #=> Hello, World!$ Legenday work of embedded CRuby

Let’s see the memory
consumption of mrbc_lrama

Memory consumption of mrbc_lrama
--------------------------------------------------------------------------------
Command:
bin/mrbc_lrama ../mruby-compiler2/fixtures/hello_world.rb
Massif arguments:
--stacks=yes
ms_print arguments: massif.out.14624
--------------------------------------------------------------------------------
MB
9.985^
:
| ::::::::::@:::::::::::@@::::::::::@@::@:::@::::::::::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
0 +----------------------------------------------------------------------->Mi
0
43.25

RAM consumption
Compiler
RAM consumption
mrbc (original) 129.5 KB
picorbc 16.70 KB
mrbc_prism 9.219 KB
mrbc_lrama 9.985 MB

$Why consumes a lot of RAM? mrc_irep * compile_ruby_to_mruby_vm_code_with_lramagen(mrc_ccontext *c) { ruby_init(); // rb_parser_t *parser = rb_ruby_parser_new(); rb_ast_t *ast = rb_ruby_parser_compile_string( parser, "", "puts 'Hello, World!'", 0); mrc_irep *irep = mrc_load_exec(c, (mrc_node *)ast->body.root); ruby_finalize(); return irep; } (For illustrative purposes only and non-functional) ruby_init() prepares the CRuby’s GC. However, if you remove it, you get SEGV$ Dependencies (actual)

DeGC is what we need
DeVALUE
Decouple VALUE that is a C-level representation of a
Ruby object such as RString and RArray
DeIMEMO
Decouple IMEMO that is a C-level object taken care of
by GC
DeGC
After all, we are going to decouple GC from the parser

Relevant work of DeVALUE
Lightning Talks today

$rb_parser_ary_t* instead of RArray enum rb_parser_ary_data_type { PARSER_ARY_DATA_AST_TOKEN, PARSER_ARY_DATA_SCRIPT_LINE }; typedef struct rb_parser_ary { enum rb_parser_ary_data_type rb_parser_ary_data *data; // long len; // long capa; // } rb_parser_ary_t; data_type; typedef of void* current size capacity$ e.g. DeVALUE of tokens
node = RubyVM::AbstractSyntaxTree.
parse('puts "Hello, World!"',
keep_tokens: true) #
node.tokens
#=>
[[0, :tIDENTIFIER, "puts", [1, 0, 1, 4]],
[1, :tSP, " ", [1, 4, 1, 5]],
[2, :tSTRING_BEG, "'", [1, 5, 1, 6]],
[3, :tSTRING_CONTENT, "Hello, World!", [1, 6, 1, 19]],
[4, :tSTRING_END, "'", [1, 19, 1, 20]]]

e.g. DeVALUE of tokens
node = RubyVM::AbstractSyntaxTree.
parse('puts "Hello, World!"',
keep_tokens: true) #
node.tokens
#=>
[[0, :tIDENTIFIER, "puts", [1, 0, 1, 4]],
[1, :tSP, " ", [1, 4, 1, 5]],
[2, :tSTRING_BEG, "'", [1, 5, 1, 6]],
[3, :tSTRING_CONTENT, "Hello, World!", [1, 6, 1, 19]],
[4, :tSTRING_END, "'", [1, 19, 1, 20]]]

$e.g. DeVALUE of tokens // parse.y static void parser_append_tokens(struct parser_params *p,... { VALUE ary = rb_ary_new2(4); // A token (...) rb_ary_push(p->tokens, ary); (...) } (...) static VALUE yycompile0(VALUE arg) { (...) if (p->keep_tokens) { p->ast->node_buffer->tokens = tokens; p->tokens = NULL; } (...) }$ $e.g. DeVALUE of tokens struct parser_params { (...) VALUE tokens; } struct parser_params { (...) rb_parser_ary_t *tokens; }$ e.g. DeVALUE of script_lines
ast->body.script_lines
iseq->variable.script_lines
SCRIPT_LINES__
Three kinds of script_lines

e.g. DeVALUE of script_lines
ast->body.script_lines
iseq->variable.script_lines
SCRIPT_LINES__
Three kinds of script_lines

script_lines in AST
node = RubyVM::AbstractSyntaxTree.
parse("puts 'Hello, World!'",
keep_script_lines: true) #
node.script_lines
#=>
["puts 'Hello, World!'"]

script_lines in ISEQ
iseq = RubyVM::InstructionSequence.
compile("puts 'Hello, World!'",
keep_script_lines: true) #
iseq.script_lines
#=>
["puts 'Hello, World!'"]

$SCRIPT_LINES__ in top-level # hello_world.rb puts 'Hello, World!' SCRIPT_LINES__ = {} require './hello_world' p SCRIPT_LINES__ #=> {"./hello_world"=>["puts 'Hello, World!'"]} debug.rb and tracer.rb used to use SCRIPT_LINES__$ Internally, they were the same RArray
ast->body.script_lines
iseq->variable.script_lines
SCRIPT_LINES__[filename]
Build script_lines as a rb_parser_ary_t
during parsing, then convert it to a VALUE
or when Node#script_lines is called

Internally, they were the same RArray
ast->body.script_lines
iseq->variable.script_lines
SCRIPT_LINES__[filename]
Build script_lines as a rb_parser_ary_t
during parsing, then convert it to a VALUE
or when Node#script_lines is called

IMEMO (Internal MEMOry Object)
IMEMO is a C-level object taken care of by GC
Not a VALUE
Five-word restriction to be GC-ed
The first word has to be the flags
You can only use the five words in total

$e.g. DeIMEMO of AST (before) typedef struct rb_ast_struct { VALUE flags; // To be removed node_buffer_t *node_buffer; struct { const NODE *root; VALUE *script_lines; // To be deVALUEd signed int frozen_string_literal:2; signed int coverage_enabled:2; } body; } rb_ast_t; Five words$ $e.g. DeIMEMO of AST (after) typedef struct rb_ast_struct { node_buffer_t *node_buffer; struct { const NODE *root; rb_parser_ary_t *script_lines; int line_count; // Separeted from script_lines signed int frozen_string_literal:2; signed int coverage_enabled:2; } body; #ifdef UNIVERSAL_PARSER const rb_parser_config_t *config; // Moved from node_buffer_t #endif } rb_ast_t; …Still five words?$ $e.g. DeIMEMO of AST (after) typedef struct rb_ast_struct { node_buffer_t *node_buffer; struct { const NODE *root; rb_parser_ary_t *script_lines; int line_count; // One word in 32 bit signed int frozen_string_literal:2; signed int coverage_enabled:2; } body; #ifdef UNIVERSAL_PARSER const rb_parser_config_t *config; // Moved from node_buffer_t #endif } rb_ast_t; Six words in 32 bit$ DeGC
DeVALUE and DeIMEMO
automatically complete DeGC

DeGC
DeVALUE and DeIMEMO
automatically complete DeGC

$Amount of diff in ruby/ruby by me $ git log --author=hasumikin \ --since=2024-01-01 --until=2024-4-30 \ --oneline ddd8da4b6b 9ea77cb351 55a402bb75 2244c58b00 9b1e97b211 f5e387a300 8aa8fce320 ce544f8dbd 40ecad0ad7 9a19cfd4cd b95e2cdca7 54f27549e2 c4bd6da298 f0a46c6334 15b53e901c [Universal parser] Improve AST structure Remove unnecessary assignment to ast->body.line_count Add line_count field to rb_ast_body_t [Universal parser] Decouple IMEMO from rb_ast_t [Universal parser] DeVALUE of p->debug_lines and ast... Separate SCRIPT_LINES__ from ast.c Fix return-type warning in compile.c [ruby/prism] [Compatibility] Improve printf format [Universal parser] Fix -Wsuggest-attribute=format warnings [Universal Parser] Reduce dependence on RArray in parse.y [ruby/prism] Additional fix of adding `x` prefix after ... [ruby/prism] Chage some names [ruby/prism] Make alloc interface replaceable [ruby/prism] Include unistd.h before cheching _POSIX_... [ruby/prism] Use `_POSIX_MAPPED_FILES` and `_WIN32` to ...$ $Amount of diff in ruby/ruby by me $ git log --author=hasumikin \ --since=2024-01-01 --until=2024-4-30 \ --pretty=tformat: --numstat \ | awk '{add += $1; del += $2} END {print "++:", add, "\n--:", del}' ++: 1321 --: 818 Despite 2000+ lines of changes, CRuby does not alter its behavior.$ $Let’s try valgrind --tool=massif again mrc_irep * compile_ruby_to_mruby_vm_code_with_lramagen(mrc_ccontext *c) { // ruby_init(); Remove this rb_parser_t *parser = rb_ruby_parser_new(); rb_ast_t *ast = rb_ruby_parser_compile_string( parser, "", "puts 'Hello, World!'", 0); mrc_irep *irep = mrc_load_exec(c, (mrc_node *)ast->body.root); // ruby_finalize(); Remove this return irep; } (For illustrative purposes only and non-functional) Now we can remove ruby_init() and ruby_finalize()$

RAM consumption of Lrama-gen parser
--------------------------------------------------------------------------------
Command:
bin/mrbc_lrama ../mruby-compiler2/fixtures/hello_world.rb
Massif arguments:
--stacks=yes
ms_print arguments: massif.out.26697
--------------------------------------------------------------------------------
KB
#
39.91^
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@@
@#:
|
@ :
@#:
|
@ ::::
:@#:
0 +----------------------------------------------------------------------->ki
0
747.9

RAM consumption of Lrama-gen parser
Compiler
RAM consumption
mrbc (original) 129.5 KB
picorbc 16.70 KB
mrbc_prism 9.219 KB
mrbc_lrama 9.985 MB
96% reduction!!!
(Ready to fight with Prism)
39.91 KB

Remaining issues of Lrama-gen parser
Make NODE(AST) design compatible with Prism
Fine-tuning of the RAM consumption
39.91 KB is the result of just deGCed
There is still more that can be done
ROM size and cross compilation

ROM size also matters
$ ls -lSr picoruby/bin/
-rwxr-xr-x
851648 picorbc
-rwxr-xr-x 2512528 mrbc_prism
-rwxr-xr-x 3386824 mrbc # original
-rwxr-xr-x 25080800 mrbc_lrama # 24 MB
picorbc solely consists of mruby compiler (ascii only)
mrbc_prism contains all encodings (can be omitted)
mrbc contains mruby-core
mrbc_lrama contains CRuby (miniruby)

To fix, we need cross compilation
$ cd ruby
$ ./autogen.sh
$ ./configure
#=> .ext/include/x86_64-linux/ruby/config.h
$ ./configure --host=arm-linux-eabi
#=> .ext/include/arm-linux-eabi/ruby/config.h
What is ruby/config.h?

ruby/config.h is platform-specific config
// .ext/include/arm-linux-eabi/ruby/config.h
...
#define RUBY_ABI_VERSION 0
#define HAVE_STDIO_H 1
...
#define SIZEOF_INT 4
#define SIZEOF_SHORT 2
...
#define SIZEOF_VOIDP 4 //
32 bit
#define SIZEOF_FLOAT 4
#define SIZEOF_DOUBLE 8
...

Target triplets [CPU]-[ambiguous]-[ambiguous]
x86-64-linux-gnu:
x86 64 bit Linux with GNU libc
aarch64-linux-gnu:
Arm 64 bit Linux with GNU libc
arm-linux-gnueabi(hf):
Arm 32 bit Linux with GNU libc for EABI (hard-float)
EABI: Embedded Application Binary Interface
arm-none-eabi:
Arm 32 bit bare-metal with newlib

arm-none-eabi
RP2040 (Raspberry Pi Pico)
133 MHz Arm Cortex-M0+ (dual)

Configurable with arm-none-eabi? No
$ cd ruby
$ ./autogen.sh #=> configure
$ ./configure --host=arm-none-eabi
#=> error
Why?
Because “none-eabi” doesn’t support
things like stdio.h that CRuby requires

The next step is to make an effective build
Configurable against bare-metal
Arm, ESP32, and PIC32, etc.
Compile only what we need instead of making
libruby-static.a
parse.o + node.o + parser_st.o + ?.o
Resolving the lack of dependencies is another challenge

Functions to be deVALUEd
VALUE
VALUE
VALUE
VALUE
VALUE
VALUE
VALUE
VALUE
VALUE
VALUE
VALUE
VALUE
VALUE
rb_node_file_path_val(const NODE *node);
rb_node_str_string_val(const NODE *node);
rb_node_line_lineno_val(const NODE *node);
rb_node_encoding_val(const NODE *node);
rb_node_integer_literal_val(const NODE *n);
rb_node_float_literal_val(const NODE *n);
rb_node_rational_literal_val(const NODE *n);
rb_node_imaginary_literal_val(const NODE *n);
rb_node_regx_string_val(const NODE *node);
rb_node_sym_string_val(const NODE *node);
rb_str_new_parser_string(rb_parser_string_t *str);
rb_str_new_mutable_parser_string(rb_parser_string_t *str);
rb_sym2id(VALUE sym);
Further deVALUE is needed

Build with dummy functions (PoC)
# Linking libruby-static.a
-rwxr-xr-x 2512528 mrbc_prism
-rwxr-xr-x 25080800 mrbc_lrama
# Linking dummy functions
-rwxr-xr-x 2512528 mrbc_prism
-rwxr-xr-x 1927440 mrbc_lrama
Now mrbc_lrama is barely able to compile
puts 'Hello, World!'

Working in progress…
https://github.com/picoruby/mruby-compiler2
https://github.com/picoruby/mruby-bin-mrbc2

Also, PicoRuby appears in Lightning Talks

This guy may give you a Raspi Pico
https://twitter.com/hanachin_/status/1790713159906681259
Catch him at the venue

Lrama-gen univ-parser is almost there!!!
Embed libruby-static.a in PicoRuby compiler
→ RAM consumption: 10 MB
→ ROM size: 24 MB
→ DeVALUE, DeIMEMO, DeGC
→ RAM consumption: 40 KB
→ Improve build process
→ ROM size: 1.9 MB

Wrap-up
PicoRuby compiler will integrate
both Prism and Lrama-gen parser
Rest of the work
Cross compile
DeVALUE (a little more)
Symbol table (ID)
Fine-tuning
github.com/picoruby/picoruby

$PicoRuby compiler must be small footprint $ valgrind \ --tool=massif \ --stacks=yes \ path/to/mrbc hello_world.rb #=> massif.out.12345 $ ms_print massif.out.12345 | less$

Description

Presentation slide for RubyKaigi 2024

Text

Page: 1

Unlock The
Universal
Parsers
A New PicoRuby
Compiler
@hasumikin
Monstarlab

Page: 2

[AD] “n-monthly Lambda Note” Vol.4, No.1
Learning How
Programming Languages
Control Electrical Circuits
with PicoRuby
(Japanese only)
1,760 yen (tax included)

Page: 3

CRuby’s parser
parse.y and prism.c: human-writable
parse.c: generated by Bison or Lrama
prism.c: no generator needed
Bison parse.c (retired)
Lrama lrama-generated parse.c
parse.y
prism.c

Page: 4

Prism and Lrama-generated parser
Prism
Newly developed universal Ruby parser
Portable and small footprint
Easy to integrate
Available in form of a gem
Lrama-generated parser (parse.c)
100% compliant with CRuby in any aspect
Not portable by nature

Page: 5

Parsers in Ruby (as of May 2024)
Ruby
Parser
CRuby (default) Lrama-generated
CRuby (experimental) Prism
JRuby Prism
TruffleRuby Prism
mruby by Matz
PicoRuby by hasumikin (me)
IRB(katakata_irb) (gem) Prism
RuboCop (gem) Parser → Prism

Page: 6

Prism in RuboCop

Page: 7

Parsers in Ruby (I’m working on)
Ruby
Parser
CRuby (default) Lrama-generated
CRuby (optional) Prism
JRuby Prism
TruffleRuby Prism
mruby ???????
PicoRuby ???????
IRB(katakata_irb) (gem) Prism
RuboCop (gem) Parser → Prism

Page: 8

mruby compiler generates VM code
$ echo 'puts "Hello, World!"'
00000000: 5249 5445 3033 3030
00000010: 3030 3030 4952 4550
00000020: 0000 0036 0000 0004
00000030: 5102 002d 0100 0138
00000040: 656c 6c6f 2c20 576f
00000050: 0470 7574 7300 454e
| bin/mrbc - |
0000 005e 4d41
0000 0042 3033
0000 0000 0000
0169 0001 0000
726c 6421 0000
4400 0000 0008
xxd #
545a
3030
000a
0d48
0100
hexdump
RITE0300...^MATZ
0000IREP...B0300
...6............
Q..-...8.i.....H
ello, World!....
.puts.END.....
$ echo 'puts "Hello, World!"' | bin/mrbc - | bin/mruby
Hello, World!
(PicoRuby compiler does the same)

Page: 9

PicoRuby compiler must be small footprint
Raspberry Pi Pico
133 MHz Arm Cortex-M0+ (dual)
264 KB SRAM
2 MB Flash ROM

Page: 10

PicoRuby compiler must be small footprint
Raspberry Pi Pico
133 MHz Arm Cortex-M0+ (dual)
264 KB SRAM
2 MB Flash ROM
If a universal parser is not small
enough, it won’t be “universal”

Page: 11

PicoRuby compiler must be small footprint
$ valgrind \
--tool=massif \
--stacks=yes \
path/to/mrbc hello_world.rb
#=> massif.out.12345
$ ms_print massif.out.12345 | less

Page: 12

The original mruby Compiler
--------------------------------------------------------------------------------
Command:
bin/mrbc ../mruby-compiler2/fixtures/hello_world.rb
Massif arguments:
--stacks=yes
ms_print arguments: massif.out.29290
--------------------------------------------------------------------------------
KB
129.5^
:
#
|
|
#
|
#
|
#
|
#
|
@#
|
@#
|
@#
|
@#
|
::@@@@@:::@@:::::::@@:@::@:@#:
|
@@@::::::::::::::::::@:@::: @@ @ :::@ ::: :::@@:@::@:@#:
|
@ : :::: ::::: :::: @:@::: @@ @ :::@ ::: :::@@:@::@:@#:
|
@ : :::: ::::: :::: @:@::: @@ @ :::@ ::: :::@@:@::@:@#:
|
@ : :::: ::::: :::: @:@::: @@ @ :::@ ::: :::@@:@::@:@#:
|
@ : :::: ::::: :::: @:@::: @@ @ :::@ ::: :::@@:@::@:@#:
|
@ : :::: ::::: :::: @:@::: @@ @ :::@ ::: :::@@:@::@:@#:
|
@ : :::: ::::: :::: @:@::: @@ @ :::@ ::: :::@@:@::@:@#:
|
@ : :::: ::::: :::: @:@::: @@ @ :::@ ::: :::@@:@::@:@#:
| @@
@ : :::: ::::: :::: @:@::: @@ @ :::@ ::: :::@@:@::@:@#:
0 +----------------------------------------------------------------------->Mi
0
1.030

Page: 13

Existing PicoRuby Compiler
--------------------------------------------------------------------------------
Command:
bin/picorbc ../mruby-compiler2/fixtures/hello_world.rb
Massif arguments:
--stacks=yes
ms_print arguments: massif.out.28004
--------------------------------------------------------------------------------
KB
16.70^
:
#::::::::::@
|
|
#::::::::::@:
|
#::::::::::@:
|
#::::::::::@:
|
#::::::::::@:
|
#::::::::::@::
|
#::::::::::@::
|
#::::::::::@::
|
#::::::::::@::
|
#::::::::::@::
|
#::::::::::@::
|
#::::::::::@::
|
::
#::::::::::@::
|
:: :
#::::::::::@::
|
:: :
#::::::::::@::
|
:: :
#::::::::::@::
|
:: :
#::::::::::@::
|
:::::
: :::::: @@ :::: :
#::::::::::@::
|
:::: ::::::::::::::::::::::: :@ ::::::::@::::@#::::::::::@::
0 +----------------------------------------------------------------------->ki
0
219.6

Page: 14

mrbc_prism
--------------------------------------------------------------------------------
Command:
bin/mrbc_prism ../mruby-compiler2/fixtures/hello_world.rb
Massif arguments:
--stacks=yes
ms_print arguments: massif.out.31071
--------------------------------------------------------------------------------
KB
#
9.219^
|
#
|
#
|
#
|
@@#
|
@ #
|
::
@ #
|
::
@ #
|
:: :
@ #:
|
:: :
@ #:
|
:: :
@ #:
|
:: :
@ #:
|
:: :
@ #:
|
:: :
@ #:
|
:: :
@ #:
|
::::
: @ #:
|
::::
:
:::::
:::
::@ #:
|
@::::::
:::::::: :::: ::@:::::: ::: :::: ::::::@ #::
|
:@::::::::::::::::::::: :::::: @: :
@: ::::::: ::::::@ #::
|
::@:::::::
::::::::: :::::: @: :
@: :: :::: ::::::@ #::
0 +----------------------------------------------------------------------->ki
0
201.0

Page: 15

RAM consumption
Compiler
RAM consumption
mrbc (original) 129.5 KB
picorbc 16.70 KB
mrbc_prism 9.219 KB*
Figures are measured in 64 bit Linux
* mrbc_prism’s small footprint is also due to a new code generator
and possibly going to be bigger with development

Page: 16

mrbc_prism
mrc_irep *
compile_ruby_to_mruby_vm_code_with_prism(mrc_ccontext *c)
{
pm_parser_t parser;
uint8_t *string = "puts 'Hello, World!'"
pm_parser_init(&parser,
string, strlen((const char *)string),
NULL);
pm_node_t *root = pm_parse(&parser);
mrc_irep *irep = mrc_load_exec(c, (mrc_node *)root);
pm_node_destroy(&parser, root);
pm_parser_free(&parser);
return irep;
}
(For illustrative purposes only and non-functional)

Page: 17

Prism, really good stuff
Well designed APIs and NODE
Portable
Small footprint
Still, I’m going to implement
Lrama-generated parser.
Why?

Page: 18

In Taipei, Taiwan last year…
me
＠koic
＠spikeolaf

Page: 19

A Labyrinth
parse.y is a labyrinth

Page: 20

@ydah_

Page: 21

@junk0612

Page: 22

no longer a Labyrinth
No longer a labyrinth

Page: 23

Why Lrama-generated parser?
Making parse.y readable and maintainable
Integrated error tolerance
Innovative state management of the tokenizer
In short, a new era of LALR parser generator
Computer-science-wide
achievement

Page: 24

OK, let me use both
Prism and Lrama-gen parser
in my PicoRuby compiler

Page: 25

Parsers in Ruby (I’m working on)
Ruby
Parser
CRuby (default) Lrama-generated
CRuby (optional) Prism
JRuby Prism
TruffleRuby Prism
mruby Prism and Lrama-gen
PicoRuby Prism and Lrama-gen
IRB(katakata_irb) (gem) Prism
RuboCop (gem) Parser → Prism

Page: 26

AST of Lrama-generated parser
$ ruby --dump=p -e'puts "Hello, World!"'
###########################################################
## Do NOT use this node dump for any purpose other than ##
## debug and research. Compatibility is not guaranteed. ##
###########################################################
#
#
#
#
#
#
#
#
#
#
#
#
#
#
#
@ NODE_SCOPE (id: 3, line: 1, location: (1,0)-(1,20))
+- nd_tbl: (empty)
+- nd_args:
|
(null node)
+- nd_body:
@ NODE_FCALL (id: 0, line: 1, location: (1,0)-(1,20))*
+- nd_mid: :puts
+- nd_args:
@ NODE_LIST (id: 2, line: 1, location: (1,5)-(1,20))
+- as.nd_alen: 1
+- nd_head:
|
@ NODE_STR (id: 1, line: 1, location: (1,5)-(1,20))
|
+- nd_lit: "Hello, World!"
+- nd_next:
(null node)

Page: 27

Version 0.0.0.0.1 of
mrbc_lrama
(A mruby compiler using Lrama-generated parser)

Page: 28

How to integrate Lrama-gen parser
$ cd ruby/ruby
$ ./autogen.sh #=> configure
$ ./configure #=> common.mk etc.
$ make
$ ls -l *.a
-rw-r--r-- 1 145797150 libruby-static.a
An archive file containing all the
object files of the Ruby interpreter

Page: 29

libruby-static.a
libruby-static.a
parse.y
Lrama
parse.c parse.o
node.c node.o
string.c string.o
etc.
libHOGE.so
ruby

Page: 30

Even CRuby can be embedded
// hello_world.c
#include <ruby.h>
int
main(int argc, char **argv)
{
ruby_init();
rb_eval_string("puts 'Hello, World!'");
ruby_finalize();
return 0;
}
(For illustrative purposes only and non-functional)

Page: 31

Even CRuby can be embedded
$ cc -o hello_world \
hello_world.c \
-I/path/to/ruby/include \
-L/path/to/ruby \
-l:libruby-static.a
#
^^^^^^^^^^^^^^^^
$ ./hello_world
#=> Hello, World!

Page: 32

Legenday work of embedded CRuby

Page: 33

Let’s see the memory
consumption of mrbc_lrama

Page: 34

Memory consumption of mrbc_lrama
--------------------------------------------------------------------------------
Command:
bin/mrbc_lrama ../mruby-compiler2/fixtures/hello_world.rb
Massif arguments:
--stacks=yes
ms_print arguments: massif.out.14624
--------------------------------------------------------------------------------
MB
9.985^
:
| ::::::::::@:::::::::::@@::::::::::@@::@:::@::::::::::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
| :::::::::: @:::: :::: :@@:: :: ::::@ ::@ : @:::: :: ::::::::@:#::::@:::
0 +----------------------------------------------------------------------->Mi
0
43.25

Page: 35

9.985 MB

Page: 36

RAM consumption
Compiler
RAM consumption
mrbc (original) 129.5 KB
picorbc 16.70 KB
mrbc_prism 9.219 KB
mrbc_lrama 9.985 MB

Page: 37

Why consumes a lot of RAM?
mrc_irep *
compile_ruby_to_mruby_vm_code_with_lramagen(mrc_ccontext *c)
{
ruby_init(); //
rb_parser_t *parser = rb_ruby_parser_new();
rb_ast_t *ast = rb_ruby_parser_compile_string(
parser, "", "puts 'Hello, World!'", 0);
mrc_irep *irep = mrc_load_exec(c, (mrc_node *)ast->body.root);
ruby_finalize();
return irep;
}
(For illustrative purposes only and non-functional)
ruby_init() prepares the CRuby’s GC.
However, if you remove it, you get SEGV

Page: 38

Dependencies (actual)

Page: 39

Dependencies (goal)

Page: 40

DeGC is what we need
DeVALUE
Decouple VALUE that is a C-level representation of a
Ruby object such as RString and RArray
DeIMEMO
Decouple IMEMO that is a C-level object taken care of
by GC
DeGC
After all, we are going to decouple GC from the parser

Page: 41

Relevant work of DeVALUE
Lightning Talks today

Page: 42

rb_parser_ary_t* instead of RArray
enum rb_parser_ary_data_type {
PARSER_ARY_DATA_AST_TOKEN,
PARSER_ARY_DATA_SCRIPT_LINE
};
typedef struct rb_parser_ary {
enum rb_parser_ary_data_type
rb_parser_ary_data *data; //
long len;
//
long capa;
//
} rb_parser_ary_t;
data_type;
typedef of void*
current size
capacity

Page: 43

e.g. DeVALUE of tokens
node = RubyVM::AbstractSyntaxTree.
parse('puts "Hello, World!"',
keep_tokens: true) #
node.tokens
#=>
[[0, :tIDENTIFIER, "puts", [1, 0, 1, 4]],
[1, :tSP, " ", [1, 4, 1, 5]],
[2, :tSTRING_BEG, "'", [1, 5, 1, 6]],
[3, :tSTRING_CONTENT, "Hello, World!", [1, 6, 1, 19]],
[4, :tSTRING_END, "'", [1, 19, 1, 20]]]

Page: 44

e.g. DeVALUE of tokens
// parse.y
static void parser_append_tokens(struct parser_params *p,...
{
VALUE ary = rb_ary_new2(4); // A token
(...)
rb_ary_push(p->tokens, ary);
(...)
}
(...)
static VALUE yycompile0(VALUE arg)
{
(...)
if (p->keep_tokens) {
p->ast->node_buffer->tokens = tokens;
p->tokens = NULL;
}
(...)
}

Page: 45

e.g. DeVALUE of tokens
struct parser_params {
(...)
VALUE tokens;
}
struct parser_params {
(...)
rb_parser_ary_t *tokens;
}

Page: 46

e.g. DeVALUE of script_lines
ast->body.script_lines
iseq->variable.script_lines
SCRIPT_LINES__
Three kinds of script_lines

Page: 47

script_lines in AST
node = RubyVM::AbstractSyntaxTree.
parse("puts 'Hello, World!'",
keep_script_lines: true) #
node.script_lines
#=>
["puts 'Hello, World!'"]

Page: 48

script_lines in ISEQ
iseq = RubyVM::InstructionSequence.
compile("puts 'Hello, World!'",
keep_script_lines: true) #
iseq.script_lines
#=>
["puts 'Hello, World!'"]

Page: 49

SCRIPT_LINES__ in top-level
# hello_world.rb
puts 'Hello, World!'
SCRIPT_LINES__ = {}
require './hello_world'
p SCRIPT_LINES__
#=>
{"./hello_world"=>["puts 'Hello, World!'"]}
debug.rb and tracer.rb used to use SCRIPT_LINES__

Page: 50

Internally, they were the same RArray
ast->body.script_lines
iseq->variable.script_lines
SCRIPT_LINES__[filename]
Build script_lines as a rb_parser_ary_t
during parsing, then convert it to a VALUE
or when Node#script_lines is called

Page: 51

IMEMO (Internal MEMOry Object)
IMEMO is a C-level object taken care of by GC
Not a VALUE
Five-word restriction to be GC-ed
The first word has to be the flags
You can only use the five words in total

Page: 52

e.g. DeIMEMO of AST (before)
typedef struct rb_ast_struct {
VALUE flags;
// To be removed
node_buffer_t *node_buffer;
struct {
const NODE *root;
VALUE *script_lines; // To be deVALUEd
signed int frozen_string_literal:2;
signed int coverage_enabled:2;
} body;
} rb_ast_t;
Five words

Page: 53

e.g. DeIMEMO of AST (after)
typedef struct rb_ast_struct {
node_buffer_t *node_buffer;
struct {
const NODE *root;
rb_parser_ary_t *script_lines;
int line_count;
// Separeted from script_lines
signed int frozen_string_literal:2;
signed int coverage_enabled:2;
} body;
#ifdef UNIVERSAL_PARSER
const rb_parser_config_t *config; // Moved from node_buffer_t
#endif
} rb_ast_t;
…Still five words?

Page: 54

e.g. DeIMEMO of AST (after)
typedef struct rb_ast_struct {
node_buffer_t *node_buffer;
struct {
const NODE *root;
rb_parser_ary_t *script_lines;
int line_count;
// One word in 32 bit
signed int frozen_string_literal:2;
signed int coverage_enabled:2;
} body;
#ifdef UNIVERSAL_PARSER
const rb_parser_config_t *config; // Moved from node_buffer_t
#endif
} rb_ast_t;
Six words in 32 bit

Page: 55

DeGC
DeVALUE and DeIMEMO
automatically complete DeGC

Page: 56

Amount of diff in ruby/ruby by me
$ git log --author=hasumikin \
--since=2024-01-01 --until=2024-4-30 \
--oneline
ddd8da4b6b
9ea77cb351
55a402bb75
2244c58b00
9b1e97b211
f5e387a300
8aa8fce320
ce544f8dbd
40ecad0ad7
9a19cfd4cd
b95e2cdca7
54f27549e2
c4bd6da298
f0a46c6334
15b53e901c
[Universal parser] Improve AST structure
Remove unnecessary assignment to ast->body.line_count
Add line_count field to rb_ast_body_t
[Universal parser] Decouple IMEMO from rb_ast_t
[Universal parser] DeVALUE of p->debug_lines and ast...
Separate SCRIPT_LINES__ from ast.c
Fix return-type warning in compile.c
[ruby/prism] [Compatibility] Improve printf format
[Universal parser] Fix -Wsuggest-attribute=format warnings
[Universal Parser] Reduce dependence on RArray in parse.y
[ruby/prism] Additional fix of adding `x` prefix after ...
[ruby/prism] Chage some names
[ruby/prism] Make alloc interface replaceable
[ruby/prism] Include unistd.h before cheching _POSIX_...
[ruby/prism] Use `_POSIX_MAPPED_FILES` and `_WIN32` to ...

Page: 57

Amount of diff in ruby/ruby by me
$ git log --author=hasumikin \
--since=2024-01-01 --until=2024-4-30 \
--pretty=tformat: --numstat \
| awk '{add += $1; del += $2} END
{print "++:", add, "\n--:", del}'
++: 1321
--: 818
Despite 2000+ lines of changes,
CRuby does not alter its behavior.

Page: 58

Let’s try valgrind --tool=massif again
mrc_irep *
compile_ruby_to_mruby_vm_code_with_lramagen(mrc_ccontext *c)
{
// ruby_init();
Remove this
rb_parser_t *parser = rb_ruby_parser_new();
rb_ast_t *ast = rb_ruby_parser_compile_string(
parser, "", "puts 'Hello, World!'", 0);
mrc_irep *irep = mrc_load_exec(c, (mrc_node *)ast->body.root);
// ruby_finalize();
Remove this
return irep;
}
(For illustrative purposes only and non-functional)
Now we can remove
ruby_init() and ruby_finalize()

Page: 59

RAM consumption of Lrama-gen parser
--------------------------------------------------------------------------------
Command:
bin/mrbc_lrama ../mruby-compiler2/fixtures/hello_world.rb
Massif arguments:
--stacks=yes
ms_print arguments: massif.out.26697
--------------------------------------------------------------------------------
KB
#
39.91^
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@#:
|
@@
@#:
|
@ :
@#:
|
@ ::::
:@#:
0 +----------------------------------------------------------------------->ki
0
747.9

Page: 60

RAM consumption of Lrama-gen parser
Compiler
RAM consumption
mrbc (original) 129.5 KB
picorbc 16.70 KB
mrbc_prism 9.219 KB
mrbc_lrama 9.985 MB
96% reduction!!!
(Ready to fight with Prism)
39.91 KB

Page: 61

Finally, DeGCed!!!

Page: 62

Remaining issues of Lrama-gen parser
Make NODE(AST) design compatible with Prism
Fine-tuning of the RAM consumption
39.91 KB is the result of just deGCed
There is still more that can be done
ROM size and cross compilation

Page: 63

ROM size also matters
$ ls -lSr picoruby/bin/
-rwxr-xr-x
851648 picorbc
-rwxr-xr-x 2512528 mrbc_prism
-rwxr-xr-x 3386824 mrbc # original
-rwxr-xr-x 25080800 mrbc_lrama # 24 MB
picorbc solely consists of mruby compiler (ascii only)
mrbc_prism contains all encodings (can be omitted)
mrbc contains mruby-core
mrbc_lrama contains CRuby (miniruby)

Page: 64

mrbc_lrama: 24 MB (on x86-64)

Page: 65

To fix, we need cross compilation
$ cd ruby
$ ./autogen.sh
$ ./configure
#=> .ext/include/x86_64-linux/ruby/config.h
$ ./configure --host=arm-linux-eabi
#=> .ext/include/arm-linux-eabi/ruby/config.h
What is ruby/config.h?

Page: 66

ruby/config.h is platform-specific config
// .ext/include/arm-linux-eabi/ruby/config.h
...
#define RUBY_ABI_VERSION 0
#define HAVE_STDIO_H 1
...
#define SIZEOF_INT 4
#define SIZEOF_SHORT 2
...
#define SIZEOF_VOIDP 4 //
32 bit
#define SIZEOF_FLOAT 4
#define SIZEOF_DOUBLE 8
...

Page: 67

Target triplets [CPU]-[ambiguous]-[ambiguous]
x86-64-linux-gnu:
x86 64 bit Linux with GNU libc
aarch64-linux-gnu:
Arm 64 bit Linux with GNU libc
arm-linux-gnueabi(hf):
Arm 32 bit Linux with GNU libc for EABI (hard-float)
EABI: Embedded Application Binary Interface
arm-none-eabi:
Arm 32 bit bare-metal with newlib

Page: 68

arm-none-eabi
RP2040 (Raspberry Pi Pico)
133 MHz Arm Cortex-M0+ (dual)

Page: 69

Configurable with arm-none-eabi? No
$ cd ruby
$ ./autogen.sh #=> configure
$ ./configure --host=arm-none-eabi
#=> error
Why?
Because “none-eabi” doesn’t support
things like stdio.h that CRuby requires

Page: 70

The next step is to make an effective build
Configurable against bare-metal
Arm, ESP32, and PIC32, etc.
Compile only what we need instead of making
libruby-static.a
parse.o + node.o + parser_st.o + ?.o
Resolving the lack of dependencies is another challenge

Page: 71

Still some deVALUE to do

Page: 72

Functions to be deVALUEd
VALUE
VALUE
VALUE
VALUE
VALUE
VALUE
VALUE
VALUE
VALUE
VALUE
VALUE
VALUE
VALUE
rb_node_file_path_val(const NODE *node);
rb_node_str_string_val(const NODE *node);
rb_node_line_lineno_val(const NODE *node);
rb_node_encoding_val(const NODE *node);
rb_node_integer_literal_val(const NODE *n);
rb_node_float_literal_val(const NODE *n);
rb_node_rational_literal_val(const NODE *n);
rb_node_imaginary_literal_val(const NODE *n);
rb_node_regx_string_val(const NODE *node);
rb_node_sym_string_val(const NODE *node);
rb_str_new_parser_string(rb_parser_string_t *str);
rb_str_new_mutable_parser_string(rb_parser_string_t *str);
rb_sym2id(VALUE sym);
Further deVALUE is needed

Page: 73

Build with dummy functions (PoC)
# Linking libruby-static.a
-rwxr-xr-x 2512528 mrbc_prism
-rwxr-xr-x 25080800 mrbc_lrama
# Linking dummy functions
-rwxr-xr-x 2512528 mrbc_prism
-rwxr-xr-x 1927440 mrbc_lrama
Now mrbc_lrama is barely able to compile
puts 'Hello, World!'

Page: 74

Working in progress…
https://github.com/picoruby/mruby-compiler2
https://github.com/picoruby/mruby-bin-mrbc2

Page: 75

FYI, PicoRuby talk at #rubykaigiC 16:40

Page: 76

Also, PicoRuby appears in Lightning Talks

Page: 77

This guy may give you a Raspi Pico
https://twitter.com/hanachin_/status/1790713159906681259
Catch him at the venue

Page: 78

Lrama-gen univ-parser is almost there!!!
Embed libruby-static.a in PicoRuby compiler
→ RAM consumption: 10 MB
→ ROM size: 24 MB
→ DeVALUE, DeIMEMO, DeGC
→ RAM consumption: 40 KB
→ Improve build process
→ ROM size: 1.9 MB

Page: 79

Wrap-up
PicoRuby compiler will integrate
both Prism and Lrama-gen parser
Rest of the work
Cross compile
DeVALUE (a little more)
Symbol table (ID)
Fine-tuning
github.com/picoruby/picoruby