Thursday, October 18, 2012

Parsing IPv4 Addresses

I came across an interesting question on StackOverflow asking about parsing IPv4 addresses. Typically, IPv4 addresses are specified with four components (e.g., something like 192.164.1.55). As the top answer points out, you might be suprised to see that ping interprets addresses a bit oddly:

C:\>ping 1
Pinging 0.0.0.1 with 32 bytes of data:

C:\>ping 1.2
Pinging 1.0.0.2 with 32 bytes of data:

C:\>ping 1.2.3
Pinging 1.2.0.3 with 32 bytes of data:

C:\>ping 1.2.3.4
Pinging 1.2.3.4 with 32 bytes of data:

C:\>ping 1.2.3.4.5
Ping request could not find host 1.2.3.4.5. Please check the name and try again.

C:\>ping 255
Pinging 0.0.0.255 with 32 bytes of data:

C:\>ping 256
Pinging 0.0.1.0 with 32 bytes of data:

In fact, you can reach google.com, using the IP address specified as dotted decimal (74.125.226.4), flat decimal (1249763844), dotted octal (0112.0175.0342.0004), flat octal (011237361004), dotted hex (0x4A.0x7D.0xE2.0x04), flat hex (0x4A7DE204), or even something of each (74.0175.0xe2.4).

Implementation

Of course, my first thought was that Factor should have a parser that works similarly (especially since I implemented support for the ping protocol awhile ago). We want a parse-ipv4 word taking a string representing the address and returning an IPv4 address string that has the typical four components.

First, we need to have words to split a string into numbered parts and a word to join them back together:

: split-components ( str -- array )
    "." split [ string>number ] map ;

: join-components ( array -- str )
    [ number>string ] map "." join ;

Then, we can parse the address simply:

: parse-ipv4 ( str -- ip )
    split-components dup length {
        { 1 [ { 0 0 0 } prepend ] }
        { 2 [ first2 [| A D | { A 0 0 D } ] call ] }
        { 3 [ first3 [| A B D | { A B 0 D } ] call ] }
        { 4 [ ] }
    } case join-components ;

Extras

If we want to support octal addresses, we can convert an octal number like 0112 to something Factor can easily parse (0o112) in our splitting code:

: cleanup-octal ( str -- str )
    dup { [ "0" head? ] [ "0x" head? not ] } 1&&
    [ 1 tail "0o" prepend ] when ;

: split-components ( str -- array )
    "." split [ cleanup-octal string>number ] map ;

And if we want to support the "carry propagation" which allows 256 to mean 0.0.1.0, we need to "bubble" the array before joining:

: bubble ( array -- array' )
    reverse 0 swap [ + 256 /mod ] map reverse nip ;

: join-components ( array -- str )
    bubble [ number>string ] map "." join ;

This (along with some error handling) has been committed to the Factor repository in the ip-parser vocabulary. If it proves useful, it might be nice to change the io.sockets to use this when resolving IPv4 addresses...

Monday, September 24, 2012

COLOR: <TAB>

As a diversion today, I added "color tab completion" to the Factor environment. I even made the color names render with their assigned color value:

For the curious, I extended the tools.completion vocabulary to implement a "colors-matching" word and then added a "colors-completion" type to the UI.

Tuesday, September 18, 2012

Faster Tables!

We've known for awhile that the Factor user interface is a little bit slow when displaying tables with many rows. I wasn't sure this was a big problem until a detailed bug report was filed (thanks @k7f!) that pointed out the opengl.gl vocabulary (with over 2,000 symbols in it) was "virtually unscrollable".

After poking around at the profiler output, I noticed that some UI gadgets had "baseline alignment" support which was used in the layout process. Some of these calculations were quite expensive and could easily be cached.

After a few changes, and introducing the concept of an "aligned gadget" which provides a mechanism to cache these alignment calculations, the original test case is almost 10 times as responsive as before!

This is available in the master branch on Github.

Monday, September 10, 2012

Watching Words

Function "annotations" are a feature that many languages support. They may take various forms such as decorators in Python or class and method annotations in Java -- having direct (like Python) or no direct (like Java) effect on the code it annotates. Factor has some similar features that I'll demonstrate below.

We can define a simple word to use in this example (that adds 2 to its input):

: foo ( x -- n ) 2 + ;

Watch

Using the tools.annotations vocabulary, we can attach a "watch" annotation that prints the inputs and outputs of a word when it is called:

IN: scratchpad \ foo watch

If you print the word definition, you can see how it was modified:

IN: scratchpad \ foo see
USING: math tools.annotations.private ;
IN: scratchpad
: foo ( x -- n ) \ foo entering 2 + \ foo leaving ;

You can call this word (either directly, or indirectly by calling another word which calls it) and see its inputs and outputs. A nice feature of this is that the UI lets you click on these values and see more detail (particularly useful if they are tuples or more complex objects):

IN: scratchpad 3 foo
--- Entering foo
x 3
--- Leaving foo
n 5

Reset

You can stop watching a word by calling "reset" on it (or right-clicking on its definition in the UI and choosing "reset") to remove all of its annotations. Currently, a word can only be annotated once.

IN: scratchpad \ foo reset

Timing

In addition to "watching", you can also use annotations to track the total running time of a word:

IN: scratchpad \ foo add-timing

IN: scratchpad 0 10,000 [ foo ] times .
20000

IN: scratchpad word-timing.
foo 0.000594456

Other

It also supports arbitrary annotations, such as adding "before" and "after" logic:

IN: scratchpad \ foo [ "hi" print ] [ "bye" print ]
               [ surround ] 2curry annotate

IN: scratchpad 3 foo .
hi
bye
5

These annotations can be pretty powerful and were even used to build our code coverage tool.

Wednesday, August 29, 2012

Literate Programming

Donald Knuth pioneered Literate programming as a technique for writing structured programs. The literate programming world typically has more descriptive text than code, so rather than "comment out" the descriptive text, they "comment in" the code.

It is very popular in some communities, for example Haskell, where even many blog posts are written in a Literate Haskell style. This is done by creating a .lhs file instead of a .hs file. In it, all lines starting with > are interpreted as code, everything else is considered a comment.

Graham Telfer asked a question on the mailing list if Factor supported literate programming. I thought it might be fun to implement some of the ideas quickly, below I'll share how it works.

The Lexer

Factor uses a lexer object to turn a stream of text into tokens that are used by the parser to turn tokens into Factor objects and definitions. We can extend the lexer system to create a version that skips over any lines that are not prefixed by a > character.

First, we define our literate-lexer sub-class:

TUPLE: literate-lexer < lexer ;

: <literate-lexer> ( text -- lexer ) literate-lexer new-lexer ;

Second, we implement the skip-blank word to skip over all lines that are just comments:

M: literate-lexer skip-blank
    dup column>> zero? [
        dup line-text>> [
            "> " head?
            [ [ 2 + ] change-column call-next-method ]
            [ [ nip length ] change-lexer-column ]
            if
        ] [ drop ] if*
    ] [ call-next-method ] if ;

We can then create a quick syntax word that looks for an end token and parses all the lines between:

SYNTAX: <LITERATE
    "LITERATE>" parse-multiline-string string-lines [
        <literate-lexer> (parse-lines) append!
    ] with-nested-compilation-unit ;

Try It

Now, you can do something like this:

IN: scratchpad <LITERATE
               This is a big wall of text with no Factor code...
               Does this work?
               1 1 + .

               I bet it didn't... maybe the following works:
               > 8675309 .
               Yay!

               You can create functions that span multiple lines
               > : foo ( -- x )
               We interrupt this program to bring you this:
               > 12 ;

               Now, we can run foo:
               > foo .
               LITERATE>
8675309
12

It might be nice to automatically support .lfactor files, but this is a quick prototype to see if it makes sense. Not bad for ten or so lines of code?

It is available now in the development branch of Factor, in the literate vocabulary.

Thursday, August 16, 2012

Factor 0.95 now available

"The reports of my death are greatly exaggerated" - Mark Twain

I'm very pleased to announce the release of Factor 0.95!

OS/CPUWindowsMac OS XLinux
x86
x86-64

Source code: 0.95

Note: it looks like the Mac OS X binaries unintentionally require 10.8 or greater, due to an upgrade of our build farm. If you want to use it on 10.6 or 10.7 before we make a fix, you can build from source:

git clone https://github.com/slavapestov/factor.git && cd factor && ./build-support/factor.sh update

This release is brought to you with over 2,500 commits by the following individuals:

Alex Vondrak, Alfredo Beaumont, Andrew Pennebaker, Anton Gorenko, Brennan Cheung, Chris Double, Daniel Ehrenberg, Doug Coleman, Eric Charlebois, Eungju Park, Hugo Schmitt, Joe Groff, John Benediktsson, Jon Harper, Keita Haga, Maximilian Lupke, Philip Searle, Philipp Brüschweiler, Rupert Swarbrick, Sascha Matzke, Slava Pestov, @8byte-jose, @yac, @otoburb, @rien

In addition to lots (and lots!) of bug fixes and improvements, I want to highlight the following features:

  • GTK-based UI backend
  • Native image loaders using Cocoa, GTK, and GDI+
  • Sampling profiler replacing counting profiler
  • Code coverage tool to improve unit tests
  • VM and application-level signal handlers
  • ICMP support and an IPv4 "ping" implementation
  • DNS client and "host" implementation
  • Support frexp and log of really big numbers
  • Cross-platform open URL in webbrowser
  • Cross-platform send file to trash (Mac OS X, Linux, Windows)
  • Speedup bignum GCD and ratio operations
  • Speedup in thread yield performance on Mac OS X
  • CSV library is 3x faster
  • XML library is 2x faster
  • JSON library is 2-3x faster
  • Many stability and performance enhancements

Some possible backwards compatibility issues:

  • Change alien references from "<int>" to "int <ref>" and "*int" to "int deref"
  • Removed Windows CE, BSD, and Solaris platform support
  • Natively support binary (0b), octal (0o), and hexadecimal (0x) number syntax
  • Unify "( -- )" and "(( -- ))" stack effect syntax
  • Change prepend to return type of first sequence to match append behavior
  • Change ".factor-rc" to be ".factor-rc" on all platforms
  • Cleanup specialized array syntax to be more generic and consistent
  • Change to quadratic probing instead of linear probing in hashtables
  • Allow dispatching on anonymous intersections/unions

What is Factor

Factor is a concatenative, stack-based programming language with high-level features including dynamic types, extensible syntax, macros, and garbage collection. On a practical side, Factor has a full-featured library, supports many different platforms, and has been extensively documented.

The implementation is fully compiled for performance, while still supporting interactive development. Factor applications are portable between all common platforms. Factor can deploy stand-alone applications on all platforms. Full source code for the Factor project is available under a BSD license.


New Libraries


Improved Libraries:

Monday, August 6, 2012

Echo Server

Factor has nice cross-platform support for network programming. As is fairly typical, I am going to use an "echo server" to demonstrate how the libraries work.

The basic logic for an "echo server" is to read input from a client, write it back to them, and repeat until the client disconnects. Using the input and output stream abstraction, we can build a word which does this in a general manner, recursing until f is read indicating end-of-stream:

: echo-loop ( -- )
    1024 read-partial [ write flush echo-loop ] when* ;

Our "echo server" will use TCP (i.e., connection-oriented networking) and the general server abstraction that comes with Factor with a binary encoding. The server framework uses the name "echo.server" to automatically log client-related messages (such as connect and disconnect events as well as errors) to $FACTOR/logs/echo.server. We specify the echo-loop quotation as the handler for clients:

: <echo-server> ( port -- server )
    binary <threaded-server>
        swap >>insecure
        "echo.server" >>name
        [ echo-loop ] >>handler ;

We can start an echo server on, for example, port 12345:

IN: scratchpad 12345 <echo-server> start-server

Testing this is pretty easy using Netcat or a similar client:

$ nc localhost 12345
Hello, Factor
Hello, Factor

This is available as the echo-server vocabulary.