Today In History

For every day of the year, Wikipedia maintains a list of famous events, births, and deaths in history. For example, if you look at today in history, you will see that in 1843, Charles Dickens novel, A Christmas Carol first went on sale.

Wouldn't it be great if you could use Factor to parse and display these famous historical events?

USING: accessors calendar formatting http.client io kernel
sequences xml xml.traversal ;

Using our calendar vocabulary, we will build a word that converts a timestamp into a Wikipedia URL:

: historical-url ( timestamp -- url )
    [ month-name ] [ day>> ] bi
    "http://en.wikipedia.org/wiki/%s_%s" sprintf ;

If you look at the source for these Wikipedia pages, you will find that it contains a series of unordered lists. In particular, we are interested in the second list containing "Events". The third list contains "Births" and the fourth contains "Deaths".

: historical-events ( timestamp -- events )
    historical-url http-get nip string>xml
    "ul" deep-tags-named second children-tags
    [ deep-children>string ] map ;

Sometimes it is a nice convention to build a word that produces output:

: historical-events. ( timestamp -- )
    historical-events [ print ] each ;

You can try this out:

IN: scratchpad today historical-events.
211 – Publius Septimius Geta, co-emperor of Rome, is lured
to come without his bodyguards to meet his brother Marcus
Aurelius Antoninus (Caracalla), to discuss a possible
reconciliation. When he arrives the Praetorian Guard murders
him and he dies in the arms of his mother Julia Domna.
324 – Licinius abdicates his position as Roman Emperor.
1154 – Henry II of England is crowned at Westminster Abbey.
1490 – Anne, Duchess of Brittany, is married to Maximilian I,
Holy Roman Emperor by proxy.
...

The code for this is on my GitHub, along with words to output historical births and deaths and formatted output that includes links to Wikipedia.

New Logo

For a long time now, the Factor logo has been a cool looking velociraptor designed in 2007 by Elie Chaftari. As part of supporting retina displays, we need new higher resolution icons. Unfortunately, Elie doesn't have the original artwork anymore to produce them.

With the help of 99designs, we ran a contest to create a new logo and icon for the Factor programming language (to capture the increase in speed and functionality gained in the last few years).

This is our new logo:

And incorporating the logo into a new application icon:

The website has been updated and the latest development version includes new icon files.

Retina Displays

I recently purchased a MacBook Pro with Retina Display. Shortly after receiving it, I realized that Factor did not support rendering on its Retina Display. Magnified (and blurry) text looks pretty terrible next to the crisp native fonts on these HiDPI displays.

Well, after a series of commits, the latest development version of Factor now supports it! There are still a few minor rendering issues with our newly supported "resolution independence", but we plan on fixing those in the near future.

To use the development version, follow the directions from the README.

The 3n+1 Problem

Yesterday, a lengthy blog post from the Racket community discussed the "3n+1" problem (also known as the Collatz conjecture):

Consider a positive number n. The cycle length of n is the number of times we repeat the following, until we reach n=1:

• If n is odd, then n = 3n+1
• If n is even, then n = n/2

For example, given n=22, we see the following sequence: 22 11 34 17 52 26 13 40 20 10 5 16 8 4 2 1. The cycle length of 22 is, therefore, 16, since it took 16 repetitions to get to 1.

Given a definition of cycle length of a number, here’s the rest of the problem: given any two numbers i and j, compute the maximum cycle length over all numbers between i and j, inclusive.

Solution

Solving this in Factor could start by defining a word to take a number and compute its next value in the cycle. Since this is an internal word, we won't add the check for the end condition of n=1:

: next-cycle ( x -- y )
    dup odd? [ 3 * 1 + ] [ 2 / ] if ;

Creating the "cycle sequence" for a given number can be done by:

: cycles ( n -- seq )
    [ dup 1 > ] [ [ next-cycle ] keep ] produce swap suffix ;

We could always find the "cycle length" by calling cycles length, but a slightly faster way would not create an intermediate sequence:

: cycle-length ( n -- m )
    1 [ over 1 > ] [ [ next-cycle ] [ 1 + ] bi* ] while nip ;

We can find the maximum cycle (number and it's cycle length) for any given sequence of numbers:

: max-cycle ( seq -- elt )
    [ dup cycle-length ] { } map>assoc [ second ] supremum-by ;

For convenience, we can make some helper words to find either the number and/or the maximum cycle length:

: max-cycle-value ( seq -- n ) max-cycle first ;

: max-cycle-length ( seq -- m ) max-cycle second ;

Testing

Factor strives to be a concise language, some of which is natural for a concatenative style. You can see this in the simple approach to unit testing:.

{
    { 22 11 34 17 52 26 13 40 20 10 5 16 8 4 2 1 }
} [ 22 cycles ] unit-test

{ 1  } [ 1 cycle-length ] unit-test
{ 2  } [ 2 cycle-length ] unit-test
{ 6  } [ 5 cycle-length ] unit-test
{ 16 } [ 22 cycle-length ] unit-test

{ 20  } [ 1 10 [a,b] max-cycle-length ] unit-test
{ 125 } [ 100 200 [a,b] max-cycle-length ] unit-test
{ 89  } [ 201 210 [a,b] max-cycle-length ] unit-test
{ 174 } [ 900 1000 [a,b] max-cycle-length ] unit-test

Performance

The original post discussed performance. Since many cycles have numbers in common, we could change our approach to a recursive one, with memoization:

MEMO: fast-cycle-length ( n -- m )
    dup 1 > [ next-cycle fast-cycle-length 1 + ] [ drop 1 ] if ;

Indeed, this takes roughly 15-20% of the time that our previous approach took to compute the number of cycles for the numbers 1 to 100,000 with a cold cache.

Main

Like the original solution, we can add a "main" to allow running from the command line:

: run-cycles ( -- )
    [
        [ blank? ] split-when harvest first2
        [ string>number ] bi@ 2dup [a,b] max-cycle-length
        "%s %s %s\n" printf
    ] each-line ;

MAIN: run-cycles

And using the original sample file, produce the same output:

$ cat sample-data.txt 
1 10
100 200
201 210
900 1000

$ factor cycles.factor < sample-data.txt
1 10 20
100 200 125
201 210 89
900 1000 174

The code for this is available on my Github.

Select Random Lines

Ned Batchelder made an interesting blog post on selecting randomly from an unknown sequence. His Python version uses a "random replacement" strategy to choose each line with a 1/n probability, where n is the number of elements seen so far:

import random

def random_element(seq):
    """Return an element chosen at random from `seq`."""
    it = None
    for n, elem in enumerate(seq):
        if random.randint(0, n) == 0:
            it = elem
    return it

random-line

I wanted to show how a similar approach in Factor could be used to select a random line from a file, or any stream that supports a character-based stream protocol.

We can make a variant of each-line that calls a quotation with each line and its "line number" (indexed from 1 to n, the number of lines in the stream):

: each-numbered-line ( ... quot: ( ... line number -- ... ) -- ... )
    [ 1 ] dip '[ swap [ @ ] [ 1 + ] bi ] each-line drop ; inline

Then, it is pretty easy to select a random line (replacing each line with chance of 1/K probability where K is the current line number.

: random-line ( -- line/f )
    f [ random zero? [ nip ] [ drop ] if ] each-numbered-line ;

random-lines

To efficiently select more than one random line from a stream, we will be using a technique called reservoir sampling. The idea is to select the first n elements, then randomly replace them with weighted probability n/K, where K is the current line number:

:: random-lines ( n -- lines )
    V{ } clone :> accum
    [| line line# |
        line# n <= [
            line accum push
        ] [
            line# random :> r
            r n < [ line r accum set-nth ] when
        ] if
    ] each-numbered-line accum ;

Note: we used local variables to implement this.

This is available in the io.random vocabulary in the Factor development branch.

Parsing IPv4 Addresses

I came across an interesting question on StackOverflow asking about parsing IPv4 addresses. Typically, IPv4 addresses are specified with four components (e.g., something like 192.164.1.55). As the top answer points out, you might be suprised to see that ping interprets addresses a bit oddly:

C:\>ping 1
Pinging 0.0.0.1 with 32 bytes of data:

C:\>ping 1.2
Pinging 1.0.0.2 with 32 bytes of data:

C:\>ping 1.2.3
Pinging 1.2.0.3 with 32 bytes of data:

C:\>ping 1.2.3.4
Pinging 1.2.3.4 with 32 bytes of data:

C:\>ping 1.2.3.4.5
Ping request could not find host 1.2.3.4.5. Please check the name and try again.

C:\>ping 255
Pinging 0.0.0.255 with 32 bytes of data:

C:\>ping 256
Pinging 0.0.1.0 with 32 bytes of data:

In fact, you can reach google.com, using the IP address specified as dotted decimal (74.125.226.4), flat decimal (1249763844), dotted octal (0112.0175.0342.0004), flat octal (011237361004), dotted hex (0x4A.0x7D.0xE2.0x04), flat hex (0x4A7DE204), or even something of each (74.0175.0xe2.4).

Implementation

Of course, my first thought was that Factor should have a parser that works similarly (especially since I implemented support for the ping protocol awhile ago). We want a parse-ipv4 word taking a string representing the address and returning an IPv4 address string that has the typical four components.

First, we need to have words to split a string into numbered parts and a word to join them back together:

: split-components ( str -- array )
    "." split [ string>number ] map ;

: join-components ( array -- str )
    [ number>string ] map "." join ;

Then, we can parse the address simply:

: parse-ipv4 ( str -- ip )
    split-components dup length {
        { 1 [ { 0 0 0 } prepend ] }
        { 2 [ first2 [| A D | { A 0 0 D } ] call ] }
        { 3 [ first3 [| A B D | { A B 0 D } ] call ] }
        { 4 [ ] }
    } case join-components ;

Extras

If we want to support octal addresses, we can convert an octal number like 0112 to something Factor can easily parse (0o112) in our splitting code:

: cleanup-octal ( str -- str )
    dup { [ "0" head? ] [ "0x" head? not ] } 1&&
    [ 1 tail "0o" prepend ] when ;

: split-components ( str -- array )
    "." split [ cleanup-octal string>number ] map ;

And if we want to support the "carry propagation" which allows 256 to mean 0.0.1.0, we need to "bubble" the array before joining:

: bubble ( array -- array' )
    reverse 0 swap [ + 256 /mod ] map reverse nip ;

: join-components ( array -- str )
    bubble [ number>string ] map "." join ;

This (along with some error handling) has been committed to the Factor repository in the ip-parser vocabulary. If it proves useful, it might be nice to change the io.sockets to use this when resolving IPv4 addresses...

COLOR: <TAB>

As a diversion today, I added "color tab completion" to the Factor environment. I even made the color names render with their assigned color value:

For the curious, I extended the tools.completion vocabulary to implement a "colors-matching" word and then added a "colors-completion" type to the UI.

Faster Tables!

We've known for awhile that the Factor user interface is a little bit slow when displaying tables with many rows. I wasn't sure this was a big problem until a detailed bug report was filed (thanks @k7f!) that pointed out the opengl.gl vocabulary (with over 2,000 symbols in it) was "virtually unscrollable".

After poking around at the profiler output, I noticed that some UI gadgets had "baseline alignment" support which was used in the layout process. Some of these calculations were quite expensive and could easily be cached.

After a few changes, and introducing the concept of an "aligned gadget" which provides a mechanism to cache these alignment calculations, the original test case is almost 10 times as responsive as before!

This is available in the master branch on Github.

Watching Words

Function "annotations" are a feature that many languages support. They may take various forms such as decorators in Python or class and method annotations in Java -- having direct (like Python) or no direct (like Java) effect on the code it annotates. Factor has some similar features that I'll demonstrate below.

We can define a simple word to use in this example (that adds 2 to its input):

: foo ( x -- n ) 2 + ;

Watch

Using the tools.annotations vocabulary, we can attach a "watch" annotation that prints the inputs and outputs of a word when it is called:

IN: scratchpad \ foo watch

If you print the word definition, you can see how it was modified:

IN: scratchpad \ foo see
USING: math tools.annotations.private ;
IN: scratchpad
: foo ( x -- n ) \ foo entering 2 + \ foo leaving ;

You can call this word (either directly, or indirectly by calling another word which calls it) and see its inputs and outputs. A nice feature of this is that the UI lets you click on these values and see more detail (particularly useful if they are tuples or more complex objects):

IN: scratchpad 3 foo
--- Entering foo
x 3
--- Leaving foo
n 5

Reset

You can stop watching a word by calling "reset" on it (or right-clicking on its definition in the UI and choosing "reset") to remove all of its annotations. Currently, a word can only be annotated once.

IN: scratchpad \ foo reset

Timing

In addition to "watching", you can also use annotations to track the total running time of a word:

IN: scratchpad \ foo add-timing

IN: scratchpad 0 10,000 [ foo ] times .
20000

IN: scratchpad word-timing.
foo 0.000594456

Other

It also supports arbitrary annotations, such as adding "before" and "after" logic:

IN: scratchpad \ foo [ "hi" print ] [ "bye" print ]
               [ surround ] 2curry annotate

IN: scratchpad 3 foo .
hi
bye
5

These annotations can be pretty powerful and were even used to build our code coverage tool.

Literate Programming

Donald Knuth pioneered Literate programming as a technique for writing structured programs. The literate programming world typically has more descriptive text than code, so rather than "comment out" the descriptive text, they "comment in" the code.

It is very popular in some communities, for example Haskell, where even many blog posts are written in a Literate Haskell style. This is done by creating a .lhs file instead of a .hs file. In it, all lines starting with > are interpreted as code, everything else is considered a comment.

Graham Telfer asked a question on the mailing list if Factor supported literate programming. I thought it might be fun to implement some of the ideas quickly, below I'll share how it works.

The Lexer

Factor uses a lexer object to turn a stream of text into tokens that are used by the parser to turn tokens into Factor objects and definitions. We can extend the lexer system to create a version that skips over any lines that are not prefixed by a > character.

First, we define our literate-lexer sub-class:

TUPLE: literate-lexer < lexer ;

: <literate-lexer> ( text -- lexer ) literate-lexer new-lexer ;

Second, we implement the skip-blank word to skip over all lines that are just comments:

M: literate-lexer skip-blank
    dup column>> zero? [
        dup line-text>> [
            "> " head?
            [ [ 2 + ] change-column call-next-method ]
            [ [ nip length ] change-lexer-column ]
            if
        ] [ drop ] if*
    ] [ call-next-method ] if ;

We can then create a quick syntax word that looks for an end token and parses all the lines between:

SYNTAX: <LITERATE
    "LITERATE>" parse-multiline-string string-lines [
        <literate-lexer> (parse-lines) append!
    ] with-nested-compilation-unit ;

Try It

Now, you can do something like this:

IN: scratchpad <LITERATE
               This is a big wall of text with no Factor code...
               Does this work?
               1 1 + .

               I bet it didn't... maybe the following works:
               > 8675309 .
               Yay!

               You can create functions that span multiple lines
               > : foo ( -- x )
               We interrupt this program to bring you this:
               > 12 ;

               Now, we can run foo:
               > foo .
               LITERATE>
8675309
12

It might be nice to automatically support .lfactor files, but this is a quick prototype to see if it makes sense. Not bad for ten or so lines of code?

It is available now in the development branch of Factor, in the literate vocabulary.

Factor 0.95 now available

"The reports of my death are greatly exaggerated" - Mark Twain

I'm very pleased to announce the release of Factor 0.95!

OS/CPU	Windows	Mac OS X	Linux
x86	0.95	0.95	0.95
x86-64	0.95	0.95	0.95

Source code: 0.95

Note: it looks like the Mac OS X binaries unintentionally require 10.8 or greater, due to an upgrade of our build farm. If you want to use it on 10.6 or 10.7 before we make a fix, you can build from source:

git clone https://github.com/slavapestov/factor.git && cd factor && ./build-support/factor.sh update

This release is brought to you with over 2,500 commits by the following individuals:

Alex Vondrak, Alfredo Beaumont, Andrew Pennebaker, Anton Gorenko, Brennan Cheung, Chris Double, Daniel Ehrenberg, Doug Coleman, Eric Charlebois, Eungju Park, Hugo Schmitt, Joe Groff, John Benediktsson, Jon Harper, Keita Haga, Maximilian Lupke, Philip Searle, Philipp Brüschweiler, Rupert Swarbrick, Sascha Matzke, Slava Pestov, @8byte-jose, @yac, @otoburb, @rien

In addition to lots (and lots!) of bug fixes and improvements, I want to highlight the following features:

• GTK-based UI backend
• Native image loaders using Cocoa, GTK, and GDI+
• Sampling profiler replacing counting profiler
• Code coverage tool to improve unit tests
• VM and application-level signal handlers
• ICMP support and an IPv4 "ping" implementation
• DNS client and "host" implementation
• Support frexp and log of really big numbers
• Cross-platform open URL in webbrowser
• Cross-platform send file to trash (Mac OS X, Linux, Windows)
• Speedup bignum GCD and ratio operations
• Speedup in thread yield performance on Mac OS X
• CSV library is 3x faster
• XML library is 2x faster
• JSON library is 2-3x faster
• Many stability and performance enhancements

Some possible backwards compatibility issues:

• Change alien references from "<int>" to "int <ref>" and "*int" to "int deref"
• Removed Windows CE, BSD, and Solaris platform support
• Natively support binary (0b), octal (0o), and hexadecimal (0x) number syntax
• Unify "( -- )" and "(( -- ))" stack effect syntax
• Change prepend to return type of first sequence to match append behavior
• Change ".factor-rc" to be ".factor-rc" on all platforms
• Cleanup specialized array syntax to be more generic and consistent
• Change to quadratic probing instead of linear probing in hashtables
• Allow dispatching on anonymous intersections/unions

What is Factor

Factor is a concatenative, stack-based programming language with high-level features including dynamic types, extensible syntax, macros, and garbage collection. On a practical side, Factor has a full-featured library, supports many different platforms, and has been extensively documented.

The implementation is fully compiled for performance, while still supporting interactive development. Factor applications are portable between all common platforms. Factor can deploy stand-alone applications on all platforms. Full source code for the Factor project is available under a BSD license.

New Libraries

• 99-bottles: texting "99 bottles of beer" (Doug Coleman)
• anagrams: words for working with anagrams (John Benediktsson)
• atk: bindings to GNOME Accessibility Toolkit (Anton Gorenko)
• audio-gadget: play button (Joe Groff)
• bitcoin.client (Chris Double)
• classes.maybe: union of classes and f (Doug Coleman)
• clutter: bindings to Clutter (Anton Gorenko)
• code-arrays (Doug Coleman)
• colors.hex (John Benediktsson)
• combinators.random: random versions of combinators (Jon Harper)
• core-foundation.launch-services bindings to OS X Launch Services API (Doug Coleman)
• dice: rolling random dice (John Benediktsson)
• dns: implements DNS protocol (Doug Coleman)
• echo-server: simple TCP echo server (John Benediktsson)
• fastcgi: support for FastCGI standard (Brennan Cheung)
• furnace.auth.providers.couchdb: CouchDB authentication (Alex Drummond)
• gdk: bindings to GDK (Anton Gorenko)
• gio: bindings to GIO (Anton Gorenko)
• gmodule: bindings to GModule (Anton Gorenko)
• gobject: bindings to GObject (Anton Gorenko)
• gobject-introspection: bindings to GObject Introspection (Anton Gorenko)
• google.charts: render Google charts in the listener (John Benediktsson)
• google.search: search Google from the listener (John Benediktsson)
• google.translate: translate languages from the listener (Doug Coleman)
• graphviz: bindings to Graphviz library (Alex Vondrak)
• gtk: bindings to GTK (Anton Gorenko)
• hamurabi: port of HAMURABI.BAS game (John Benediktsson)
• hashtables.sequences (John Benediktsson)
• help.search: allow searching within help article text (John Benediktsson)
• ini-file: parsing INI-style configuration files (John Benediktsson)
• interval-sets: efficient sets of integer intervals (Daniel Ehrenberg)
• io.binary.fast: fast endian conversion (Doug Coleman)
• io.files.temp: better cross-platform temp files (Joe Groff)
• io.files.trash: cross platform send-to-trash (John Benediktsson)
• io.sockets.icmp: ICMP support (John Benediktsson)
• io.streams.256color: xterm-256color formatted stream (John Benediktsson)
• io.streams.peek: peekable I/O streams (Doug Coleman)
• io.streams.throwing: I/O streams that throw on EOF (Doug Coleman, Joe Groff)
• lint: Factor "lint" tool (Doug Coleman)
• llvm.clang: Clang FFI bindings (Erik Charlebois)
• math.approx: approximating rationals (John Benediktsson)
• math.distances: distance functions (John Benediktsson)
• math.extras: assorted math functions (John Benediktsson)
• math.similarity: similary functions (John Benediktsson)
• math.transforms.fft: fast Fourier transform (John Benediktsson)
• memcached: query memcached databases (John Benediktsson)
• ntp: client for NTP protocol (John Benediktsson)
• ping: implements IPv4 ping client (John Benediktsson)
• progress-bars: displaying textual progress bars (Doug Coleman)
• random.data: generate randomized strings (Doug Coleman)
• readline-listener: with tab completion! (Eric Charlebois)
• readline: libreadline bindings (Eric Charlebois)
• resolv-conf: parser for resolv.conf files (Doug Coleman)
• rosetta-code: examples from Rosetta Code website (many anonymous authors)
• semantic-versioning: implements Semantic Versioning specification (Maximilian Lupke)
• sequences.extras: assorted math functions (Doug Coleman, John Benediktsson)
• sequences.windowed: virtual rolling sequence (Doug Coleman)
• slots.macros: macro-based accessors (Joe Groff)
• sorting.extras: assorted words (John Benediktsson)
• tnetstrings: reader/writer for "tagged netstrings" (John Benediktsson)
• tools.coverage: measure code coverage (Doug Coleman)
• tools.deploy.embed: self-executing images (Joe Groff)
• tools.dns (Doug Coleman)
• tools.profiler.sampling: sampling profiler (Joe Groff)
• tools.ps: process listing tool (Doug Coleman)
• tools.test.fuzz: quickcheck-style probabilistic testing (Joe Groff, Andrew Pennebaker)
• txon: reader/writer for TXON format (John Benediktsson)
• units.reduction: reduce units to convenient forms (Doug Coleman)
• unix.signals: signal handlers (Joe Groff)
• wake-on-lan: support for Wake-on-LAN protocol (John Benediktsson)
• webbrowser: open URL's in your webbrowser (John Benediktsson)
• windows.gdiplus: bindings to GDI+ (Joe Groff)
• windows.iphlpapi: bindings to IP Helper (Doug Coleman)
• windows.privileges: bindings to Privileges API (Doug Coleman)
• windows.registry: bindings to Registry API (Doug Coleman)
• windows.streams: IStream interface (Joe Groff)
• wolfram-alpha: query Wolfram|Alpha from the listener (John Benediktsson)
• xkcd: viewer for XKCD comics (John Benediktsson)
• xmode.highlight: syntax highlighting using formatted streams (John Benediktsson)

Improved Libraries:

• alien.endian: speedup byte-reverse (Doug Coleman)
• alien.fortran: various improvements (Joe Groff)
• assocs: performance improvements (John Benediktsson)
• classes.struct: support for packed structures (John Benediktsson)
• combinators.smart: various improvements (Doug Coleman)
• compiler: various improvements (Slava Pestov)
• command-line: pass arguments to "-run" vocabs (John Benediktsson)
• core-foundation.run-loop: greatly speed up yield (John Benediktsson)
• csv: performance improvements (John Benediktsson)
• curses: various improvements (Philipp Brüschweiler)
• fuel: various improvements (Doug Coleman, Eungju Park, John Benediktsson)
• generalizations: adding smart-with (John Benediktsson)
• graphviz: unit tests and improvements (Alex Vondrak)
• help.apropos: fixes and improvements (otoburb, John Benediktsson)
• http.client: added OPTIONS, TRACE and HEAD HTTP methods (otoburb)
• images.png: various fixes (Philip Searle)
• infix: support slice notation (John Benediktsson)
• io: non-copying read operations (Joe Groff)
• io.backend: various improvements (Slava Pestov, Joe Groff, Erik Charlebois)
• io.binary: performance improvements (John Benediktsson)
• io.encodings.string: performance improvements (Joe Groff)
• io.encodings.utf8: various improvements (John Benediktsson)
• io.files.windows: fix handling of UNC paths (8byte-jose)
• io.pathnames: support "~" pathname expansion (otoburb)
• io.serial: fixes (Doug Coleman)
• io.sockets: implement a <raw> port (Doug Coleman)
• json: performance improvements (John Benediktsson)
• lexer/parser: various improvements (Doug Coleman)
• make: support building assocs (John Benediktsson)
• math.combinatorics: various improvements (John Benediktsson)
• math.complex: syntax checking (Doug Coleman)
• math.floating-point: added double>ratio (John Benediktsson)
• math.functions: various improvements (Doug Coleman, John Benediktsson)
• math.matrices: various improvements (Doug Coleman, John Benediktsson)
• math.polynomials: speedup polyval (Doug Coleman)
• math.primes: various improvements (Samuel Tardieu)
• math.primes.erato: bug fixes (Rupert Swarbrick)
• math.statistics: many new words (Doug Coleman, John Benediktsson)
• math.vectors: adding v^, v^n, n^v (John Benediktsson)
• memoize: speedup no-argument memoized words (John Benediktsson)
• mongodb: various improvements (Sascha Matzke)
• namespaces: speedup global access (Joe Groff)
• path-finding: Dijkstra's algorithm and fixes (Samuel Tardieu, Alfredo Beaumont)
• random: many improvements (Doug Coleman, John Benediktsson)
• sequences: performance improvements (John Benediktsson)
• sequences.repeating: various improvements (John Benediktsson)
• sets: performance improvements (John Benediktsson)
• slots: improved slot type checking and coercion (Doug Coleman)
• sorting.human: case-insensitive comparisons (Doug Coleman)
• specialized-arrays: various cleanups (John Benediktsson)
• splitting: speedup string-lines, add split* (John Benediktsson)
• system-info.macosx: various improvements (John Benediktsson)
• tools.completion: performance improvements (John Benediktsson)
• tools.crossref: performance improvements (John Benediktsson)
• ui.gadgets.table: improve hook behavior (Hugo Schmitt)
• ui.tools: Ctrl-Shift-F for "toggle fullscreen" (John Benediktsson)
• ui.tools.browser: support font resizing (John Benediktsson)
• ui.tools.listener: better exception handling (John Benediktsson)
• ui.tools.listener.completion: improved word completion (John Benediktsson)
• ui.tools.operations: better watch/reset behavior (John Benediktsson)
• unicode.breaks: speedup >words (John Benediktsson)
• xml: performance improvements (John Benediktsson)

Echo Server

Factor has nice cross-platform support for network programming. As is fairly typical, I am going to use an "echo server" to demonstrate how the libraries work.

The basic logic for an "echo server" is to read input from a client, write it back to them, and repeat until the client disconnects. Using the input and output stream abstraction, we can build a word which does this in a general manner, recursing until f is read indicating end-of-stream:

: echo-loop ( -- )
    1024 read-partial [ write flush echo-loop ] when* ;

Our "echo server" will use TCP (i.e., connection-oriented networking) and the general server abstraction that comes with Factor with a binary encoding. The server framework uses the name "echo.server" to automatically log client-related messages (such as connect and disconnect events as well as errors) to $FACTOR/logs/echo.server. We specify the echo-loop quotation as the handler for clients:

: <echo-server> ( port -- server )
    binary <threaded-server>
        swap >>insecure
        "echo.server" >>name
        [ echo-loop ] >>handler ;

We can start an echo server on, for example, port 12345:

IN: scratchpad 12345 <echo-server> start-server

Testing this is pretty easy using Netcat or a similar client:

$ nc localhost 12345
Hello, Factor
Hello, Factor

This is available as the echo-server vocabulary.

xterm-256color

For those of you using terminals, you are probably aware of ANSI escape codes. As a refresher, it allows formatting text with foreground and background colors, underlines, italics, blinking, among other characteristics.

Factor has support for formatted output streams that can apply styles to textual content. It is used extensively by the help system to produce documentation in both the UI and command-line, export to HTML or PDF, and even syntax highlighting.

This morning, I wanted to implement a formatted output stream that can be used in the terminal to produce colored text output easily.

You can look at documentation for the xterm-256color ANSI extensions, but basically you wrap your text with format codes, like so:

Setting foreground to blue:

Setting background to red:

There are varying methods of checking if a terminal supports the 256 color extension, but a simple way is to look at the TERM environment variable:

: 256color-terminal? ( -- ? )
    "TERM" os-env "-256color" tail? ;

The specification should tell us how to map RGB color codes to their "256color" code:

RGB Colors

A value between 16 and 231 is used for RGB colors with each color having 6 intensities. Red has a value between 0 and 5 multiplied by 36, Green has a value between 0 and 5 multiplied by 6, and Blue has a value between 0 and 5.

8 to 24 bit color conversion

To display xterm RGB colors as 24 bit RGB colors the following values are suggested for the 6 intensities: 0x00, 0x5F, 0x87, 0xAF, 0xD7, and 0xFF.

Following that guideline, we can build our map of "256color" codes. I also added system colors and grayscale colors, which is left as an exercise for the reader (or you can read the source code):

CONSTANT: intensities { 0x00 0x5F 0x87 0xAF 0xD7 0xFF }

CONSTANT: 256colors H{ }

! Add the RGB colors
intensities [| r i |
    intensities [| g j |
        intensities [| b k |
            i 36 * j 6 * + k + 16 +
            r g b 3array
            256colors set-at
        ] each-index
    ] each-index
] each-index

Given a RGBA Color, we want to convert it to an array of its red, green, and blue components, and then find the "256color" with the smallest "distance" (in this case Euclidean distance) from the desired value.

: color>rgb ( color -- rgb )
    [ red>> ] [ green>> ] [ blue>> ] tri
    [ 255 * round >integer ] tri@ 3array ;

: color>256color ( color -- 256color )
    color>rgb '[ _ distance ]
    256colors [ keys swap infimum-by ] [ at ] bi ;

Converting a color to either a foreground or a background ANSI escape code is now pretty easy:

: color>foreground ( color -- str )
    color>256color "\u00001b[38;5;%sm" sprintf ;

: color>background ( color -- str )
    color>256color "\u00001b[48;5;%sm" sprintf ;

With a basic implementation of the formatted stream protocol, we can now take something like this:

And easily (and automatically!) display it as something like this:

To see more colorful implementation of words:

Or even experiment with various background colors:

This has been committed to Factor's development branch and is available now.

Wake-on-LAN

Wake-on-LAN is an ethernet standard that allows a computer to be turned on or woken up by a network message. This is a kinda-fun-sometimes-useful feature that I thought we should have in Factor.

The way the protocol works is to broadcast a "magic packet" that contains information describing the computer you want to wakeup. From the Wikipedia article:

The magic packet is a broadcast frame containing anywhere within its payload 6 bytes of all 255 (FF FF FF FF FF FF in hexadecimal), followed by sixteen repetitions of the target computer's 48-bit MAC address, for a total of 102 bytes. Since the magic packet is only scanned for the string above, and not actually parsed by a full protocol stack, it may be sent as any network- and transport-layer protocol, although it is typically sent as a UDP datagram to port 7 or 9, or directly over Ethernet as EtherType 0x0842.

First, we need a way to turn a MAC address into an array of bytes:

: mac-address-bytes ( mac-address -- byte-array )
    ":-" split [ hex> ] B{ } map-as ;

Next, we need a way to construct the "magic packet" according to the specification:

: wake-on-lan-packet ( mac-address -- bytearray )
    [ 16 ] [ mac-address-bytes ] bi* <array> concat
    B{ 0xff 0xff 0xff 0xff 0xff 0xff } prepend ;

Now that we have the "magic packet", we can create a datagram port configured to send broadcast packets (using a recent commit that made this easy and cross-platform):

: wake-on-lan ( mac-address broadcast-ip -- )
    [ wake-on-lan-packet ] [ 9 <inet4> ] bi*
    f 0 <inet4> <broadcast> [ send ] with-disposal ;

It's easy enough to test this using Mac OS X since most modern Mac hardware supports Wake-on-LAN. You can configure it via the System Preferences Energy Saver panel, in the Options tab. Marking the "Wake for network access" checkbox enables Wake-on-LAN support. For other platforms, read more about receiving the magic packet.

The code for this is on my Github.