A goal of mine has been rewriting terraform-mode. The current implemenation has some minor bugs that are annoying. It depends on hcl-mode which introduces complexity in being able to rewrite it. Another goal of mine is to have it extensively tested to prevent regressions.

The end result of this exercise will be a terraform-mode version 2.0.

Identifying Comments​

We start off by identfying our comments. We use the b style comment for single line comments and we use a style comments for multiline.

(defvar terraform-mode-syntax-table
  (let ((table (make-syntax-table)))
    ;; # and // are line comments (style b); /* */ are block comments (style a)
    (modify-syntax-entry ?# "< b" table)
    (modify-syntax-entry ?\n "> b" table)
    (modify-syntax-entry ?/ ". 124b" table)
    (modify-syntax-entry ?* ". 23" table)
    ...
    table)
    "Syntax table for `terraform-mode'.")

For the # style comments < b means this is a single characeter comment starter for b type comments.

For \n the > b means this is what terminates a single line comment.

For / the . 124b compacts a lot of information.

• The . means it's a punctuation character class.
• The 1 means it's the start of a two character start comment sequence.
• The 2 means its the second character of a two character start comment sequence.
• The 4 means its the last character of a two character end comment sequence.
• The b applies to the second time / shows up making it a b style comment.

For * the . 23 finsihes the multiline comment support.

• The . serves the same purpose as before.
• The 2 means the same thing as well.
• The 3 means it's the first character in a 2 character end sequence.

Brackets​

(defvar terraform-mode-syntax-table
  (let ((table (make-syntax-table)))
    ...
    (modify-syntax-entry ?{ "(}" table)
    (modify-syntax-entry ?} "){" table)
    (modify-syntax-entry ?\[ "(]" table)
    (modify-syntax-entry ?\] ")[" table)
    (modify-syntax-entry ?\( "()" table)
    (modify-syntax-entry ?\) ")(" table)
    table)
    "Syntax table for `terraform-mode'.")

For the open curly brace ?{ "(}" says { is an open brace and its matching closing brace is }.

For the close curly brace ?} "){" says } is a close brace and its matching open brace is {.

I won't explain the rest of the table modifications beacuse it's the same thing over and over for the difference braces.

Creating the mode​

In order to create the mode is derived from prog-mode.

;;;###autoload
(define-derived-mode terraform-mode prog-mode "Terraform"
  "Major mode for editing Terraform files."
  :syntax-table terraform-mode-syntax-table
  (setq-local comment-start "#")
  (setq-local comment-end "")
  (setq-local font-lock-defaults '(nil nil nil)))

;;;###autoload
(add-to-list 'auto-mode-alist '("\\.tf\\'" . terraform-mode))
;;;###autoload
(add-to-list 'auto-mode-alist '("\\.tfvars\\'" . terraform-mode))

(provide 'terraform-mode)

At this point when we open a Terraform file it no highlighting works outside of comments.

First Keyword​

Now we want to introduce syntax highlighting for the terraform builtin word. In order to do this we craft a regexp using the rx macro:

(defconst terraform-mode--block-builtins-no-type-or-name
  (rx line-start (zero-or-more space) (group "terraform")))

Then we configure the table we'll pass to font-lock-defaults:

(defconst terraform-mode--font-lock-keywords
  `((,terraform-mode--block-builtins-no-type-or-name 1 font-lock-builtin-face)))

The , in ,terraform-mode--block-builtins-no-type-or-name exapnds the macro within the quoted list. The 1 says to apply face font-lock-builtin-face to the first capture group.

Then we update our font-lock-defaults from above to use our table:

(setq-local font-lock-defaults '(terraform-mode--font-lock-keywords nil nil))

Handling Variables​

When writing terraform we use _ a lot as part of variable names. We need to treat this as part of words instead of as punction. This is achieved by:

(modify-syntax-entry ?_ "w" table)

Then we can write our regexp to capture assignment statements:

(defconst terraform-mode--variable
  (rx line-start (zero-or-more space) (group (one-or-more word)) (zero-or-more space) "="))

Then we add this to our font-lock-keywords table and assignment statements are properly highlighted.

We'll continue iterating on this in the next post.

I want to start off by stating that all prose is written by a human. I use Claude to help me build tools and to insert links into posts I write. Claude does not generate my posts.

It's been nearly 10 years since I last wrote a blog post. This blog used to be a Blogger site. I took it down when I noticed a bunch of spam comments that needed to be cleaned up. But then in the shuffle of life I never migrated it anywhere.

I recently dabbed a Claude Code subscription and after refactoring a few tools I recently built I wanted to see if I could get my blog back up and running using Claude.

I backed my blog using Blogger's export capabilties. This provided me an atom feed of my blog posts. Using Claude I setup a plan to take the existing blog and migrate it to a Docusaurus site.

Claude wrote a Python script to extract the post text and convert it from HTML to Markdown using markdownify. After converting them to markdown I had Claude review to identify any broken links or missing iamges. Claude identified a few dead links and some images that were not downloaded as part of the backup since they expired from Blogger's CDN.

I then used Claude to help me build out the menu. This was really helpful since I haven't done frontend work in a long time. It did take serveral iterations to get the menu where I wanted it.

Once I had my working Docusaurus site, I setup a Github Pages website and configured Giscus to be able to support comments.

Being able to use Claude to accelerate the more tedius parts of the migration was super helpful.

Continuing with the variety of files related to the Arcane Forge, we explored arc_frg_eqp.ds.

This is a list of items equipped to you char from the Arcane forge.

The file is 8 lines long. The odd lines can be either 0, 1, or 2. This maps to the equipped items Ninby's Grace, Herberk's Pendant, Murderer's Ring respectively.

I believe the even number is the item this turns into when you put it back in the inventory.

After hacking away at the character safe information, I wanted to take a crack at understanding the weapons system.

In ~/Libaray/Application Support/com.yoyogames.macyoyorunner there’s a grip of files realted to the Arcane Forge.

In order to give ourselves items we want to look at arc_frg_itm.ds.

This is a list of numbers which corresponds to various weapons. The breakdown is:

When supplying numbers outside this range you can crash the game. I could not get consistent behavior on the crash. One time it was an index out of range, another time it was indexing an object that wasn’t indexible.

When it handles bad values correctly it defaults them to the Barbarian’s Axe.

We notice there’s some interesting characters mentioned that aren’t in the game. There’s the Viking, Viper, Rogue, Knight, Archmage, Paladin, Raven, Destroyer, Duelist, and Mechanic. Their default weapons can be seen below with a bunch of Murderer’s Rings.

From what I can tell it looks like they will be an upgraded version of the base version of each character.

I’ve been playing a game called Dungeon Souls. It’s a rouge like that difficult at first, but then becomes extremely fun once you get the hang of it.

After playing for a bit, I started messing around with Bit-Slicer to give myself massive HP regen and a bunch of damage, I wanted to do something a bit more fun. I wanted to edit my save.

A quick Google search for the save location came up empty. I needed to watch which files the process interacted with during so I could find them.

To do this I ran sudo fs_usage | grep Mac | less. I then launched and exited the game which generated a nice list of file system usage. While browsing through the results a file called dungeonsouls.ds caught my eye.

Upon opening the file, you are greeted with 51 lines of information. It was time to trace these values back to the game.

Time to boot up the game and see if we can find anything that looks like these values.

Oh hey look at that, 4 stars and 27275 experience. I think we found our Barbarian!

Woot passives and gold are found.

And our stats…

After poking around, the save file breaks down as follows:

Unfortunately, I couldn’t figure out what lines 1, 2, 10, and 12 are for.

Passives are capped at 10 and setting them past that doesn’t work.

Based on previous behaviors I saw during my time with Bit-Slicer, setting the experience past the current amount needed for the current level results in one level increase and the experience being reset to 0.

The unlock toggle is 0 for locked 1 for unlocked. It does not work for the Barbarian, Thief, and Archer as they are the default chars.

Do not set the class level outside 0 - 5. You will cause an index out of bounds error and crash the game.

If stats require a numeric value, but receive a string, they are set to 0 in the game.

Just having some funsies with highest level text:

My previous posts about getting started with epgsql helped me get familiar so I could store my Makov chain data in a database to help Markov-lang learn how to talk.

Today we’re going to discuss how we generate the Markov chain data that will go into that database. I’m going to forgo the OTP portion of this application. Though the full version can be found on my github.

For those unfamiliar, Markov chains describe how likely you are to transition from one state to the next. Read about them here and here. Or you can just Google it.

I wanted to model text as a chain. This chain represents the transition from a prefix of two words to a one word suffix.

this is a string, this is a string, this is a ending

start of sentence -> this
start of sentence this -> is
this is -> a
is a -> string,
a string, -> this
string, this -> is
this is -> a
is a -> string,
a string, -> this
string, this -> is
this is -> a
is a -> ending
a ending -> end of sentence

To start this we will represent start of sentence as two spaces [" ", " "]. We will tokenize the string. Then we will represent end of sentence as a special string ["<<< undefined>>>"]. (Please note the space in <<< undefined>>> is there because of the markdown).

The string:

"This is a string, this is a string, this is a ending"

Turns into:

[" ", " ", "This", "is", "a", "string", " this", "is", "a", "string", " this", "is", "a", "ending", "<<< undefined>>>"]

We will write a splitString function to take our incoming string, remove any unprintable chars, and return the list of tokens.

%% markov.erl

We pad it with two blank space to represent the start of a sentence. Then we append "<<< undefined>>>" so that we know when we’ve reached the end of a sentence.

Now that we have a list of the words, we need to turn this into the chains we described above. In order to represent these I created a chain record:

%% records.hrl

Now lets turn our list of tokens into a table of chains.

%% markov.erl

We will discuss the call to reduceTable shortly. genTable/1 takes in a list of tokens and then calls itself with the list, the tail of the list and the tail of the tail of the list. We keep accumulating chains until we run out of suffixes.

When we reach tl(L3) == [] the accumulator acc looks like:

1 [  
2   {chain, "   ", "this"},  
3   {chain, " this", "is"},  
4   {chain, "this is", "a"},      
5   {chain, "is a", "string,"},  
6   {chain, "a string,", "this"},  
7   {chain, "string, this", "is"},  
8   {chain, "this is", "a"},      
9   {chain, "is a", "string,"},  
10  {chain, "a string,", "this"},  
11  {chain, "string, this", "is"},  
12  {chain, "this is", "a"},      
13  {chain, "is a", "ending"},  
14  {chain, "a ending", "<<<undefined>>>"}  
15 ]

You’ll notice that some chains such as lines 4 and 8 have the same prefix and suffix. Also, you’ll notice chains such as lines 5, 9, and 13 all share the same prefix, but line 13 has a different suffix. In order to weight chains properly I came up with 2 more records.

%% records.hrl  
-record(reducedChain, {prefix="", suffixes=[]}).  
-record(suffix, {word="", count=0}).

The reducedChain record represents a prefix and a list of suffixes. The suffixes in the list are of the suffix record type which keeps track of the suffix and how many times it has appeared after the prefix for the reducedChain.

The call to reduceTable, told you we would get to this, takes the list of chains and turns it into a list of reducedChains.

%% markov.erl  
reduceTable(Table) ->  
    reduceTable(Table, []).  
reduceTable([], Acc) ->  
    Acc;  
reduceTable(Table, Acc) ->  
    Chain = hd(Table),  
    Prefix = Chain#chain.prefix,  
    ChainsSamePrefix = gatherChainsWithPrefix(Prefix, Table),  
    Suffixes = genSuffixes(ChainsSamePrefix),  
    reduceTable(Table -- ChainsSamePrefix, Acc ++ [factories:reducedChainFactory(Prefix, Suffixes)]).

Our first step is to get all the chains with the same prefix:

hasPrefix(Prefix, #chain{prefix=Prefix, suffix=_}) ->  
    true;  
hasPrefix(_,_) ->  
    false.  
gatherChainsWithPrefix(Prefix, Table) ->  
    lists:filter(fun(Chain) ->  
             hasPrefix(Prefix, Chain)  
         end, Table).

If we gather the chains for the prefix "is a" we would get the following output:

1 [  
2   {chain, "is a", "string,"},  
3   {chain, "is a", "string,"},  
4   {chain, "is a", "ending"}  
5 ]

The next step is to create the suffixes for our reducedChain.

hasSuffix(Suffix, #chain{prefix=_, suffix=Suffix}) ->  
    true;  
hasSuffix(_,_) ->  
    false.  
  
gatherChainsWithSuffix(Suffix, Table) ->  
    lists:filter(fun(Chain) ->  
            hasSuffix(Suffix, Chain)  
         end, Table).  
  
countSameSuffix(Suffix, Chains) ->  
    lists:foldl(fun(Chain, NumSame) ->  
         case hasSuffix(Suffix, Chain) of  
            true -> NumSame + 1;  
            false -> NumSame  
         end  
      end, 0, Chains).  
  
genSuffixes([], Acc) ->  
    Acc;  
genSuffixes(Chains, Acc) ->  
    Chain = hd(Chains),  
    Suffix = Chain#chain.suffix,  
    SameSuffixes = gatherChainsWithSuffix(Suffix, Chains),  
    NumSameSuffix = countSameSuffix(Suffix, Chains),  
    genSuffixes(Chains -- SameSuffixes, Acc ++ [factories:suffixFactory(Suffix, NumSameSuffix)]).

We would get the following list of suffixes:

[  
    {suffix, "string,", 2},  
    {siffix, "ending", 1}     
]

Now we have everything we need to create a reducedChain. We do this for all items and we end up with the following reducedTable:

[  
    {reducedChain, "   ", [{suffix, "this", 1}],  
    {reducedChain, " this", [{suffix, "is", 1}],  
    ...  
    {reducedChain, "is a", [{suffix, "string,", 2}, {suffix, "ending", 1}],  
    ...  
    {reducedChain, "this is", [{suffix, "a", 3}],  
    ...  
]

Now we have generated out weighted chains and can use these to generate sentences. I will go over this in my next blog post when I cover how we attach the database.

A version of Markov-lang that can generate sentences without the database exists at this commit. Just feed the output of genTable into genSentence

In the last post we got epgsql running inside our Erlang REPL.  Now we are going to start using epgsql.

Start your Postgres server and open up your REPL.  Make sure epgsql is loaded by running:

m(epgsql).

If it's there you will see the module's information:

To connect to our database we run:

{ok, Connection} = epgsql:connect("<hostname>", "<username>", "<password>", [{database, "<dbname>"}]).

This will return a connection that we can then use to interact with the database.

To create a table by running the following Simple Query:

{ok, [], []} = epgsql:squery(Connection, "CREATE TABLE foo (id SERIAL PRIMARY KEY, prefix TEXT, suffix TEXT);").

Now to insert data into our database we run:

{ok, 1} = epgsql:squery(Connection, "INSERT INTO foo (prefix, suffix) VALUES ('foo', 'bar');").

This returns {ok, N}, where N is the number of rows inserted.  Lets go head and add two more items into out database:

{ok, 2} = epgsql:squery(Connection, "INSERT INTO foo  (prefix, suffix) VALUES ('one', 'two'), ('three', 'four');").

In order to query our database we can use a simple query

{ok, Cols, Rows} = epgsql:squery(Connection, "SELECT * FROM foo;").

This will return all the data in the row as binary data:

In order to get data returned typed correctly we need to use an extended query:

{ok, Cols, Rows} = epgsql:equery(Connection, "SELECT * FROM foo;", []).

As you can see, the id column is an integer now, the strings are still binary.  However, if we had booleans they would be returned as boolean values, etc.

That's how you can connect to and get data in and out of a Postgres database using Erlang.

Now lets close the connection by running:

epgsql:close(Connection).

I've been trying to get epgsql so I can build out the database for Markov-lang.  Getting this running took some time.

First we needed to install rebar3.  You do this by running:

brew install homebrew/versions/rebar3

Once rebar3 is installed, you have to configure it to work with hex.pm.  You do this by running the following:

mkdir -p ~/.config/rebar3 

Then create the following file ~/.config/rebar3/rebar.config with the following contents:

 {plugins, [rebar3_hex]}.

(EDIT: According to Reddit user mononcqc this is only necessary if you want to publish packages not fetch them)

Then run:

rebar3 update

Now, in the directory for you project that will use postgres, create a file /Path/To/App/rebar.config with the following content:

{deps, [epgsql]}.

Now you can access epgsql by running:

rebar3 shell

It should compoile epgsql. In the REPL started by the previous command you can run:

m(epgsql).

To access this from within Emacs, start the Erlang REPL in Emacs with the following flag:

-pz _build/default/deps/epgsql/ebin 

Part 2 can be found here.

For a while I've wanted to start writing in a functional language. However, I never really knew how to start or what to do. I read most of the way through Learn You A Haskell. Then I read about a functional language called Erlang which looked really cool. I read a bit of Learn You Some Erlang. Things made a bit more sense, but I still didn't know what to do.

Yesterday, I finally wrote my first program in Haskell. It's called Python Class Creator. Python Class Creator is a pretty simple program. It takes in two strings and then dumps out a basic python class implementation. Check it out on my Github page.

The GPS addon I use with for my RaspberryPi starts sending information to the RPi through the GPIO the second it tuns on. The RPi reads this as input to the serial console and this interrupts UBoot's autoboot process.

In order to fix this you had to use set autoboot to only stop when a certain string is typed in during the autoboot countdown. The following changes are made to UBoot/include/configs/rpi.h.

First we enable autoboot by adding #define CONFIG_BOOT_DELAY N, where N is a number of seconds. Then we enable the advanced autoboot controls by adding #define CONFIG_AUTOBOOT_KEYED.

Next we need to print out a prompt to the user to let them know that we are waiting for autoboot. To do this we define #define CONFIG_AUTOBOOT_PROMPT str,bootdelay, where "str" is a string that takes the format of a call to printf() and allows variable input. I set my str to "autoboot in %d seconds\ntype 'stop' to get to UBoot\n",bootdelay.

Finally we define the string that must be typed in after the prompt appears in order to get into UBoot. This is achieved by adding #define CONFIG_AUTOBOOT_STOP_STR str, where str is the string that needs to be typed in.

Now when I turn on my RaspberryPi I still have serial access if I need it. Now I have to configure loader.conf to not listen to the serial port.