Sunday, February 12, 2023

Magic

Ever wonder what the type of a particular binary file is? Or wonder how a program knows that a particular binary file is in a compatible file format? One way is to look at the magic number used by the file format in question. You can see some examples in a list of file signatures.

The libmagic library commonly supports the file command on Unix systems, other than Apple macOS which has its own implementation, and uses magic numbers and other techniques to identify file types. You can see how it works through a few examples:

$ file vm/factor.hpp
vm/factor.hpp: C++ source text, ASCII text

$ file Factor.app/Contents/Info.plist 
Factor.app/Contents/Info.plist: XML  document text

$ file factor
factor: Mach-O 64-bit executable x86_64

$ file factor.image
factor.image: data

Wrapping the C library

I am going to show how to wrap a C library using the alien vocabulary which provides an FFI capability in Factor. The man pages for libmagic show us some of the functions available in magic.h.

The libmagic library needs to be made available to the Factor instance:

"magic" {
    { [ os macosx? ] [ "libmagic.dylib" ] }
    { [ os unix? ] [ "libmagic.so" ] }
} cond cdecl add-library 

We start by defining an opaque type for magic_t:

TYPEDEF: void* magic_t

Some functions are available for opening, loading, and then closing the magic_t:

FUNCTION: magic_t magic_open ( int flags )

FUNCTION: int magic_load ( magic_t magic, c-string path )

FUNCTION: void magic_close ( magic_t magic )

It is convenient to wrap the close function as a destructor for use in a with-destructors form.

DESTRUCTOR: magic_close

A function that "returns a textual description of the contents of the filename argument", which gives us the file command ability above:

FUNCTION: c-string magic_file ( magic_t magic, c-string path )

That should be everything we need to continue...

Using the C library

Now that we have the raw C library made available as Factor words, we can create a simpler interface by wrapping some of the words into a simple word that guesses the type of a file:

: guess-file ( path -- result )
    [
        normalize-path
        0 magic_open &magic_close
        [ f magic_load drop ]
        [ swap magic_file ] bi
    ] with-destructors ;

And we can then try it on a few files:

IN: scratchpad "vm/factor.hpp" guess-file .
"C++ source, ASCII text"

IN: scratchpad "Factor.app/Contents/Info.plist" guess-file .
"XML 1.0 document, Unicode text, UTF-8 text"

IN: scratchpad "factor" guess-file .
"symbolic link to Factor.app/Contents/MacOS/factor"

IN: scratchpad "factor.image" guess-file .
"data"

This has been available for awhile in the magic vocabulary with improved error checking and some options to guess the MIME type of files.

Wednesday, February 8, 2023

Hipku

Once upon a time, there was a Javascript project called Hipku. The original post that described it was lost somewhere in the series of tubes, but thankfully the "full documentation and a working demo" was saved by the Wayback Machine. It is also still available on npm for installation.

Hipku is a small javascript library to encode IP addresses as haiku. It can express any IPv4 or IPv6 address as a Western-style 5/7/5 syllable haiku.

An implementation in Python was created called PyHipku. It is still available on PyPi for installation, but the website associated with it was also lost to history and not even the Great Wayback Machine seems able to recover it. I think of programming as aspiring to a kind of poetic result — and wonder what kind of language could run Waka Waka Bang Splat — well, the haiku style caught my interest, so I ported the hipku algorithm to the Factor programming language.

At it's core, we encode an IPv4 address or IPv6 address into a series of numerical values and then make a poem by looking up each word from a word list. Some symbols are defined to help us know to start a sentence with an uppercase letter or end a sentence with a period:

SYMBOLS: Octet octet octet. ;

For example, an IPv4 key specifies the word lists to use for each octet and an IPv4 schema specify how the octets form into a hipku — an f indicates a newline:

CONSTANT: ipv4-key ${
    animal-adjectives animal-colors animal-nouns animal-verbs
    nature-adjectives nature-nouns plant-nouns plant-verbs
}

CONSTANT: ipv4-schema ${
    "The" octet octet octet f
    octet "in the" octet octet. f
    Octet octet.
}

To create the hipku, we iterate across the key, choosing words numerically by looking up the octet value, and then composing them into the ordering specified by the schema.

You can see a couple examples below:

IN: scratchpad "127.0.0.1" >hipku print
The hungry white ape
aches in the ancient canyon.
Autumn colors crunch.

IN: scratchpad "2001:db8:3333:4444:5555:6666:7777:8888" >hipku print
Chilled apes and blunt seas
clap dear firm firm grim grim gnomes.
Large holes grasp pained mares.

We support both encoding into a hipku as well as decoding back into an IPv4/IPv6 address. This is available as the hipku vocabulary in a recent nightly build.

Tuesday, February 7, 2023

Proquint

A few days ago, Ciprian Dorin Craciun wrote a binary to text encoding blog post about the "state of the art and missed opportunities" in various encoding schemes. In that post, I was introduced to the Proquint encoding which stands for "PRO-nouncable QUINT-uplets".

In the Factor programming language, we have enjoyed implementing many encoding/decoding methods including: base16, base24, base32, base32hex, base32-crockford, base36, base58, base62, base64, base85, base91, uu, and many others. I thought it would be fun to add a quick implementation of Proquint.

Like other encodings, it makes use of an alphabet — grouped as consonants and vowels:

CONSTANT: consonant "bdfghjklmnprstvz"

CONSTANT: vowel "aiou"

Numbers are grouped into 5-character blocks representing a 16-bit number, with alternating consonants representing 4 bits and vowels representing 2 bits:

: >quint16 ( m -- str )
    5 [
        even? [
            [ -4 shift ] [ 4 bits consonant nth ] bi
        ] [
            [ -2 shift ] [ 2 bits vowel nth ] bi
        ] if
    ] "" map-integers-as reverse nip ;

Encoding a 32-bit number is made by joining two 16-bit blocks:

: >quint32 ( m -- str )
    [ -16 shift ] keep [ 16 bits >quint16 ] bi@ "-" glue ;

Decoding numbers looks up each consonant or vowel, skipping separators:

: quint> ( str -- m )
    0 [
        dup $[ consonant alphabet-inverse ] nth [
            nip [ 4 shift ] [ + ] bi*
        ] [
            dup $[ vowel alphabet-inverse ] nth [
                nip [ 2 shift ] [ + ] bi*
            ] [
                CHAR: - assert=
            ] if*
        ] if*
    ] reduce ;

We can use this to make a random password that might be more memorable — but perhaps more secure if using more random-bits:

: quint-password ( -- quint )
    32 random-bits >quint32 ;

And we could use our ip-parser vocabulary to make IPv4 addresses more memorable:

: ipv4>quint ( ipv4 -- str )
    ipv4-aton >quint32 ;

: quint>ipv4 ( str -- ipv4 )
    quint> ipv4-ntoa ;

You can see how this might work by building a test suite to show roundtrips work:

{ t } [
    {
        { "127.0.0.1"       "lusab-babad" }
        { "63.84.220.193"   "gutih-tugad" }
        { "63.118.7.35"     "gutuk-bisog" }
        { "140.98.193.141"  "mudof-sakat" }
        { "64.255.6.200"    "haguz-biram" }
        { "128.30.52.45"    "mabiv-gibot" }
        { "147.67.119.2"    "natag-lisaf" }
        { "212.58.253.68"   "tibup-zujah" }
        { "216.35.68.215"   "tobog-higil" }
        { "216.68.232.21"   "todah-vobij" }
        { "198.81.129.136"  "sinid-makam" }
        { "12.110.110.204"  "budov-kuras" }
    } [
        [ quint>ipv4 = ] [ swap ipv4>quint = ] 2bi and
    ] assoc-all?
] unit-test

This is now available as the proquint vocabulary in a recent nightly build.

Tuesday, January 24, 2023

Semantic Versioning

Semantic Versioning (or "semver" for short) is a specification for handling version numbers, and providing a way to sort and specify compatibility using a MAJOR.MINOR.PATCH structure with optional "pre-release" and "build" information.

Some examples of semantic version numbers:

  • 1.0.0-alpha
  • 1.0.0-beta+win32
  • 1.0.0-rc.1
  • 1.0.0

For a long time, I thought it might be funny to follow the upcoming release of Factor version 0.99 with version 0.100. Well, if we wanted to be consistent with "semver", it might instead have to be something like 0.100.0-joke+haha.

There is now a semver vocabulary that provides some words for sorting and working with semantic version numbers. Here's an example using it:

IN: scratchpad USE: semver

IN: scratchpad "0.99.0" >semver bump-alpha semver.
0.99.1-alpha.0

IN: scratchpad "0.99.0" >semver bump-preminor bump-rc semver.
0.100.0-rc.0

IN: scratchpad "0.99.0" "0.100.0" semver<=> .
+lt+

IN: scratchpad "0.100.0-joke+haha" >semver bump-major semver.
1.0.0

Reading the Semantic Versioning 2.0.0 specification, it suggests using the version numbers to represent compatibility with previous versions. And many languages have package managers that use these compatibility guarantees with "semver ranges" to manage project dependencies.

Saturday, January 21, 2023

Five Questions

Many years ago, there was a blog post containing five programming problems every software engineer should be able to solve in less than 1 hour. I had bookmarked it at the time and didn't notice the controversy it created on Reddit. The original link seems to be down — you can view a copy of it on the Wayback Machine — but there are various solutions posted online, including a solution in Python.

I finally got around to looking at it and writing up some solutions to the problems listed. Apparently, instead of solving this in 1 hour in Factor, it took me almost 8 years:

IN: scratchpad 2015 05 09 <date> days-since >integer .
2814

Problem 1

Write three functions that compute the sum of the numbers in a given list using a for-loop, a while-loop, and recursion.

In idiomatic Factor, this is just sum, and we would typically use sequence combinators, but instead here are a few solutions using lexical variables.

Using a for-loop, iterating forwards:

:: solve-1 ( seq -- n )
    0 seq length [ seq nth-unsafe + ] each-integer ;

Using a while-loop, iterating forwards:

:: solve-1 ( seq -- n )
    0 0 seq length :> n
    [ dup n < ] [
        [ seq nth-unsafe + ] [ 1 + ] bi
    ] while drop ;

Using recursion, iterating backwards:

:: (solve-1) ( accum i seq -- accum' )
    accum i [
        1 - seq [ nth-unsafe + ] 2keep (solve-1)
    ] unless-zero ;

: solve-1 ( seq -- n )
    0 swap [ length ] keep (solve-1) ;

Some test cases to confirm behavior:

{ 0 } [ { } solve-1 ] unit-test

{ 1 } [ { 1 } solve-1 ] unit-test

{ 6 } [ { 1 2 3 } solve-1 ] unit-test

Problem 2

Write a function that combines two lists by alternatively taking elements. For example: given the two lists [a, b, c] and [1, 2, 3], the function should return [a, 1, b, 2, c, 3].

We can alternately choose items from each list:

: solve-2 ( seq1 seq2 -- newseq )
    [ min-length 2 * ] 2keep '[
        [ 2/ ] [ even? ] bi _ _ ? nth-unsafe
    ] { } map-integers-as ;

Some test cases to confirm behavior:

{ { "a" 1 "b" 2 "c" 3 } } [
    { "a" "b" "c" } { 1 2 3 } solve-2
] unit-test

{ { "a" 1 "b" 2 "c" 3 } } [
    { "a" "b" "c" "d" } { 1 2 3 } solve-2
] unit-test

Problem 3

Write a function that computes the list of the first 100 Fibonacci numbers. By definition, the first two numbers in the Fibonacci sequence are 0 and 1, and each subsequent number is the sum of the previous two.

There are many approaches, including using memoization, but instead we'll just iterate from the starting values and use replicate to build up an output array.

: solve-3 ( n -- seq )
    [ 0 1 ] dip [ dup rot [ + ] keep ] replicate 2nip ;

Some test cases to confirm behavior:

{ { } } [ 0 solve-3 ] unit-test

{ { 0 } } [ 1 solve-3 ] unit-test

{ { 0 1 } } [ 2 solve-3 ] unit-test

{ { 0 1 1 2 3 5 8 13 21 34 } } [ 10 solve-3 ] unit-test

{ 573147844013817084100 } [ 100 solve-3 sum ] unit-test

Problem 4

Write a function that given a list of non negative integers, arranges them such that they form the largest possible number. For example, given [50, 2, 1, 9], the largest formed number is 95021.

We can try each-permutation of the input numbers, looking for their largest numerical value when the digits are concatenated.

: solve-4 ( seq -- n )
    0 swap [ number>string ] map
    [ concat string>number max ] each-permutation ;

Some test cases to confirm behavior:

{ 95021 } [ { 50 2 1 9 } solve-4 ] unit-test

{ 5523 } [ { 52 5 3 } solve-4 ] unit-test

Problem 5

Write a program that outputs all possibilities to put + or - or nothing between the numbers 1, 2, ..., 9 (in this order) such that the result is always 100. For example: 1 + 2 + 34 – 5 + 67 – 8 + 9 = 100.

This one is more complicated than the previous ones, but we can build it up piece by piece, using a test-case on each step to show how it works.

First, we want a word to interleave numbers amongst operators using solve-2.

: insert-numbers ( operators -- seq )
    [ length [1..b] ] [ solve-2 ] [ length 1 + suffix ] tri ;

{ { 1 f 2 + 3 f 4 } } [ { f + f } insert-numbers ] unit-test

Next, we want a word that will join adjacent digits — separated by f:

GENERIC: digits, ( prev intermediate -- next )
M: number digits, [ 10 * ] [ + ] bi* ;
M: object digits, [ [ , 0 ] dip , ] when* ;

: join-digits ( seq -- seq )
    [ [ ] [ digits, ] map-reduce , ] { } make ;

{ { 12 + 34 } } [ { 1 f 2 + 3 f 4 } join-digits ] unit-test

Since Factor is a kind of Reverse Polish notation, we'll want to swap from infix to postfix:

: swap-operators ( seq -- seq )
    dup rest-slice 2 <groups> [ 0 1 rot exchange ] each ;

{ { 12 34 + } } [ { 12 + 34 } swap-operators ] unit-test

The solution, then, is to use all-selections of addition, subtraction, and adjacency — interleaving the numbers, joining adjacent digits, swapping operators, and then calling each sequence as a quotation, filtering for the ones that return 100:

: solve-5 ( -- solutions )
    { + - f } 8 all-selections
    [ insert-numbers join-digits swap-operators ] map
    [ >quotation call( -- x ) 100 = ] filter ;

We can print the formula out by swapping the operators back to infix and printing them out:

: print-formula ( solutions -- )
    [ present ] map swap-operators " " join print ;

: solve-5. ( -- )
    solve-5 [ print-formula ] each ;

Spoilers! The printed solutions:

IN: scratchpad solve-5.
1 + 2 + 3 - 4 + 5 + 6 + 78 + 9
1 + 2 + 34 - 5 + 67 - 8 + 9
1 + 23 - 4 + 5 + 6 + 78 - 9
1 + 23 - 4 + 56 + 7 + 8 + 9
12 + 3 + 4 + 5 - 6 - 7 + 89
12 + 3 - 4 + 5 + 67 + 8 + 9
12 - 3 - 4 + 5 - 6 + 7 + 89
123 + 4 - 5 + 67 - 89
123 + 45 - 67 + 8 - 9
123 - 4 - 5 - 6 - 7 + 8 - 9
123 - 45 - 67 + 89

Thursday, January 19, 2023

GetPercentageRounds

There was a funny post on Twitter a couple of days ago about a recent event where the "Dutch government was forced to release the source code of their DigiD digital authentication iOS app" with this piece of C# code:

Some very funny discussions continued, with comments about how good or bad this code is, and how one might rewrite it in various ways. I thought it would be a fun opportunity to show a few variations of this simple function in Factor.

Implementations

A direct translation of this code, might use cond which is basically a sequence of if statements:

: get-percentage-rounds ( percentage -- str )
    {
        { [ dup 0.0 <= ] [ drop "⚪⚪⚪⚪⚪⚪⚪⚪⚪⚪" ] }
        { [ dup 0.0 0.1 between? ] [ drop "🔵⚪⚪⚪⚪⚪⚪⚪⚪⚪" ] }
        { [ dup 0.1 0.2 between? ] [ drop "🔵🔵⚪⚪⚪⚪⚪⚪⚪⚪" ] }
        { [ dup 0.2 0.3 between? ] [ drop "🔵🔵🔵⚪⚪⚪⚪⚪⚪⚪" ] }
        { [ dup 0.3 0.4 between? ] [ drop "🔵🔵🔵🔵⚪⚪⚪⚪⚪⚪" ] }
        { [ dup 0.4 0.5 between? ] [ drop "🔵🔵🔵🔵🔵⚪⚪⚪⚪⚪" ] }
        { [ dup 0.5 0.6 between? ] [ drop "🔵🔵🔵🔵🔵🔵⚪⚪⚪⚪" ] }
        { [ dup 0.6 0.7 between? ] [ drop "🔵🔵🔵🔵🔵🔵🔵⚪⚪⚪" ] }
        { [ dup 0.7 0.8 between? ] [ drop "🔵🔵🔵🔵🔵🔵🔵🔵⚪⚪" ] }
        { [ dup 0.8 0.9 between? ] [ drop "🔵🔵🔵🔵🔵🔵🔵🔵🔵⚪" ] }
        [ drop "🔵🔵🔵🔵🔵🔵🔵🔵🔵🔵" ]
    } cond ;

But since this is a series of if statements checked sequentially, you can just check the upper bounds. And since we only care about the argument for the comparison, we can use cond-case:

: get-percentage-rounds ( percentage -- str )
    {
        { [ 0.0 <= ] [ "⚪⚪⚪⚪⚪⚪⚪⚪⚪⚪" ] }
        { [ 0.1 <= ] [ "🔵⚪⚪⚪⚪⚪⚪⚪⚪⚪" ] }
        { [ 0.2 <= ] [ "🔵🔵⚪⚪⚪⚪⚪⚪⚪⚪" ] }
        { [ 0.3 <= ] [ "🔵🔵🔵⚪⚪⚪⚪⚪⚪⚪" ] }
        { [ 0.4 <= ] [ "🔵🔵🔵🔵⚪⚪⚪⚪⚪⚪" ] }
        { [ 0.5 <= ] [ "🔵🔵🔵🔵🔵⚪⚪⚪⚪⚪" ] }
        { [ 0.6 <= ] [ "🔵🔵🔵🔵🔵🔵⚪⚪⚪⚪" ] }
        { [ 0.7 <= ] [ "🔵🔵🔵🔵🔵🔵🔵⚪⚪⚪" ] }
        { [ 0.8 <= ] [ "🔵🔵🔵🔵🔵🔵🔵🔵⚪⚪" ] }
        { [ 0.9 <= ] [ "🔵🔵🔵🔵🔵🔵🔵🔵🔵⚪" ] }
        [ drop "🔵🔵🔵🔵🔵🔵🔵🔵🔵🔵" ]
    } cond-case ;

One suggestion was to generate a substring based on the input — with the somewhat negative aspect that it allocates memory for the returned string when called:

: get-percentage-rounds ( percentage -- str )
    10 * 10 swap - >integer dup 10 +
    "🔵🔵🔵🔵🔵🔵🔵🔵🔵🔵⚪⚪⚪⚪⚪⚪⚪⚪⚪⚪" subseq ;

But another way would be to just index into the possible results, using quoted words to reduce the amount of tokens involved — resulting in this fairly aesthetic result:

: get-percentage-rounds ( percentage -- str )
    10 * ceiling >integer qw{
        ⚪⚪⚪⚪⚪⚪⚪⚪⚪⚪
        🔵⚪⚪⚪⚪⚪⚪⚪⚪⚪
        🔵🔵⚪⚪⚪⚪⚪⚪⚪⚪
        🔵🔵🔵⚪⚪⚪⚪⚪⚪⚪
        🔵🔵🔵🔵⚪⚪⚪⚪⚪⚪
        🔵🔵🔵🔵🔵⚪⚪⚪⚪⚪
        🔵🔵🔵🔵🔵🔵⚪⚪⚪⚪
        🔵🔵🔵🔵🔵🔵🔵⚪⚪⚪
        🔵🔵🔵🔵🔵🔵🔵🔵⚪⚪
        🔵🔵🔵🔵🔵🔵🔵🔵🔵⚪
        🔵🔵🔵🔵🔵🔵🔵🔵🔵🔵
    } nth ;

It's always fun to see different ways to solve problems. In the Twitter thread, that includes using binary search, building the output character-by-character, generating solutions using ChatGPT, one-liners in Python, pattern matching, unit testing, and discussions of edge cases and naming conventions.

Monday, January 16, 2023

Project Gemini

Project Gemini is a neat modern take on the Gopher protocol. You can read the Gemini FAQ or the Gemini specification to learn more details, but the home page has a nice summary:

Gemini is a new internet protocol which

  • Is heavier than gopher
  • Is lighter than the web
  • Will not replace either
  • Strives for maximum power to weight ratio
  • Takes user privacy very seriously

There are some nice Gemini clients implemented in various languages, for both the command-line and with nice user interfaces. I happen to enjoy using AV-98 and Lagrange, but many others are also great.

In a similar manner to my Gopher implementation in Factor, I recently implemented the Gemini protocol as well as a Gemini server and a Gemini user interface:

Instead of going into how the protocol or the user interface is implemented, I wanted to go over the Gemini command-line interface. In the spirit of Python's cmd module, I contributed the command-loop vocabulary to support generic line-oriented command interpreters.

We start by making a sequence of commands that our Gemini interpreter will support:

CONSTANT: COMMANDS {
    T{ command
        { name "back" }
        { quot [ drop gemini-back ] }
        { help "Go back to the previous Gemini URL." }
        { abbrevs { "b" } } }
    T{ command
        { name "forward" }
        { quot [ drop gemini-forward ] }
        { help "Go forward to the next Gemini URL." }
        { abbrevs { "f" } } }
    T{ command
        { name "history" }
        { quot [ drop gemini-history ] }
        { help "Display recently viewed Gemini URLs." }
        { abbrevs { "h" "hist" } } }
    T{ command
        { name "less" }
        { quot [ drop gemini-less ] }
        { help "View the most recent Gemini URL in a pager." }
        { abbrevs { "l" } } }
    T{ command
        { name "ls" }
        { quot [ gemini-ls ] }
        { help "List the currently available links." }
        { abbrevs f } }
    T{ command
        { name "go" }
        { quot [ gemini-go ] }
        { help "Go to a Gemini URL" }
        { abbrevs { "g" } } }
    T{ command
        { name "gus" }
        { quot [ drop "gemini://gus.guru/search" gemini-go ] }
        { help "Submit a query to the GUS search engine." }
        { abbrevs f } }
    T{ command
        { name "up" }
        { quot [ drop gemini-up ] }
        { help "Go up one directory from the recent Gemini URL." }
        { abbrevs { "u" } } }
    T{ command
        { name "url" }
        { quot [ drop gemini-url ] }
        { help "Print the most recent Gemini URL." }
        { abbrevs f } }
    T{ command
        { name "reload" }
        { quot [ drop gemini-reload ] }
        { help "Reload the most recent Gemini URL." }
        { abbrevs { "r" } } }
    T{ command
        { name "root" }
        { quot [ drop gemini-root ] }
        { help "Navigate to the most recent Gemini URL's root." }
        { abbrevs f } }
    T{ command
        { name "shell" }
        { quot [ gemini-shell ] }
        { help "'cat' the most recent Gemini URL through a shell." }
        { abbrevs { "!" } } }
    T{ command
        { name "quit" }
        { quot [ drop gemini-quit ] }
        { help "Quit the program." }
        { abbrevs { "q" "exit" } } }
}

And then we define a custom command-loop that will allow us to number the links on a Gemini page, and then by typing a number we can navigate to one of the links by detecting a "missing command":

TUPLE: gemini-command-loop < command-loop ;

M: gemini-command-loop missing-command
    over string>number [ 1 - LINKS ?nth ] [ f ] if* [
        gemini-go 3drop
    ] [
        call-next-method
    ] if* ;

You can see it in action:

$ ./factor -run=gemini.cli
Welcome to Project Gemini!
GEMINI> go gemini.circumlunar.space/news/

Official Project Gemini news feed

[1] Atom feed

2023 News

[2] 2023-01-14 - Tidying up gemini.circumlunar.space user capsules
[3] 2023-01-08 - Changing DNS server

2022 News

[4] 2022-06-20 - Three years of Gemini!
[5] 2022-01-30 - Minor specification update (0.16.1)
[6] 2022-01-22 - Mailing list archives, Atom feed for official news
[7] 2022-01-16 - Mailing list downtime, official news feed