Planet Parrot

Patrick Michaud: Rakudo Star 2010.08 released

Wed, 01 Sep 2010 12:36:56 +0000

[This announcement was made last week on rakudo.org -- I'm reposting to use.perl.org so it will show up in the various Perl aggregators. --Pm]

On behalf of the Rakudo and Perl 6 development teams, I'm happy to announce the August 2010 release of "Rakudo Star", a useful and usable distribution of Perl 6. The tarball for the August 2010 release is available from http://github.com/rakudo/star/downloads.

Rakudo Star is aimed at "early adopters" of Perl 6. We know that it still has some bugs, it is far slower than it ought to be, and there are some advanced pieces of the Perl 6 language specification that aren't implemented yet. But Rakudo Perl 6 in its current form is also proving to be viable (and fun) for developing applications and exploring a great new language. These "Star" releases are intended to make Perl 6 more widely available to programmers, grow the Perl 6 codebase, and gain additional end-user feedback about the Perl 6 language and Rakudo's implementation of it.

In the Perl 6 world, we make a distinction between the language ("Perl 6") and specific implementations of the language such as "Rakudo Perl". The August 2010 Star release includes release #32 of the Rakudo Perl 6 compiler [1], version 2.7.0 of the Parrot Virtual Machine [2], and various modules, documentation, and other resources collected from the Perl 6 community.

This release of Rakudo Star adds the following features over the previous Star release:
* Nil is now undefined
* Many regex modifiers are now recognized on the outside of regexes
* Mathematic and range operations are now faster (they're still slow, but they're significantly faster than they were in the previous release)
* Initial implementations of .pack and .unpack
* MAIN can parse short arguments
* Removed a significant memory leak for loops and other repeated blocks

This release (temporarily?) omits the Config::INI module that was included in the 2010.07 release, as it no longer builds with the shipped version of Rakudo. We hope to see Config::INI return soon.

There are some key features of Perl 6 that Rakudo Star does not yet handle appropriately, although they will appear in upcoming releases. Thus, we do not consider Rakudo Star to be a "Perl 6.0.0" or "1.0" release. Some of the not-quite-there features include:
* nested package definitions
* binary objects, native types, pack and unpack
* typed arrays
* macros
* state variables
* threads and concurrency
* Unicode strings at levels other than codepoints
* pre and post constraints, and some other phasers
* interactive readline that understands Unicode
* backslash escapes in regex character classes
* non-blocking I/O
* most of Synopsis 9
* perl6doc or pod manipulation tools

In many places we've tried to make Rakudo smart enough to inform the programmer that a given feature isn't implemented, but there are many that we've missed. Bug reports about missing and broken features are welcomed.

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. An updated draft of a Perl 6 book is available as in the release tarball.

The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org mailing list, or join us on IRC channel #perl6 on freenode.

Rakudo Star releases are created on a monthly cycle or as needed in response to important bug fixes or improvements. The next planned release of Rakudo Star will be on September 28, 2010.

[1] http://github.com/rakudo/rakudo
[2] http://parrot.org/

Andrew Whitworth: Parrot Foundation Board

noreply@blogger.com (Whiteknight) — Wed, 01 Sep 2010 00:56:41 +0000

Today at the weekly Parrotsketch meeting there was also a meeting of members of the Parrot Foundation, with the aim of electing board members for the coming year. All but one of the current board members chose not to run for re-election, so it was a pretty open field.

Four candidates were nominated, and today all of them were elected by simple majority: Jerry Gay (particle), Jim Keenan (kid51), Jonathan Leto (dukeleto), and myself.

Here's a brief discussion of what I want to do this year as a Parrot Foundation board member:

Re-read and re-reread the foundation bylaws. I've read through them a while ago back when the foundation was first forming, but don't think I've looked at them since. Plus, I've read through so many sets of bylaws for so many young organizations in my time, the details all run together in my mind.
Money. The foundation doesn't have a whole ton of money, of course I can't think of too many non-profits of this size which do. There are lots of things that can be done with money: grants and bounties being two that immediately come to mind, but there are plenty of other things. Finding ways to raise money and increase the PaFo coffers should probably be a pretty big priority for us.
Create a proper membership committee. Parroteer "smash" has been running the elections and doing a fantastic job, but some problems have been exposed in the membership process. Many people do not know whether they are members or not, and many people are confused by how a person becomes a member. Clearing up confusion in this area will help everybody.
Recruit new people. People are the fuel that runs an open-source project, and you can never have too many people. Parrot is by no means a small open source project, but it is far from being a big one. More developers create more/better software, which attracts more end users, which increases the prestige of the project, which attracts more developers, which.... It's a feedback loop that we should be trying much harder to feed.

These are just four things that I would like to focus on this term, however this is not a definitive list. I am hoping to get, within the next few days, some information from the current board members about what kinds of tasks they have left unfinished, or what kinds of things they would try to accomplish if given another term. Since there is so much new blood on the committee, we need to make sure to thoroughly pick the brains of the current board so that we can get this coming year off to a running start.

parrot.org: PAST Optimization: GSoC is over

Tue, 24 Aug 2010 20:43:29 +0000

The Google Summer of Code pencils-down date was last Monday. GSoC is now over, but I don't plan to stop working on my project.

The initial goals listed in my project proposal were:

A library for PAST traversal.
A framework for PAST optimization and analysis tools.
A regular-expression-like pattern matching library for PASTs.
An optimization to turn tail-calls in PAST into PIR ".tailcall".

Andrew Whitworth: PLA Planning: Next Steps

noreply@blogger.com (Whiteknight) — Mon, 23 Aug 2010 21:00:00 +0000

I did a little bit more infrastructure work on Parrot-Linear-Algebra this weekend. I wasn't able to get a lot done because we spent Saturday with family, but I got some nice groundwork laid for important future developments.

I started thinking about bindings for the LAPACK library, and much of the work I did this weekend was in preparation for that work. LAPACK is particularly tricky because it's written in FORTRAN and unlike the ATLAS library I use for my BLAS implementation, there aren't any good C bindings to LAPACK around that I can find. Yes there is the CLAPACK library, but I am not a particular fan of the way that library is generated and there aren't any premade packages for CLAPACK on Ubuntu. In installing PLA on Linux, I would like users to be able to get all prerequisites from their distribution's package manager and not have to compile these other prerequisites manually. This is a good thing since some of these can be extremely tricky to compile.

I've decided that, in the foreseeable future, we will be linking to the LAPACK library directly. This does mean that we are going to need to write our own bindings and wrappers for the functions we want to call, but that's not a huge issue. Plus, I can start moving forward on something I've wanted to have for a while now: more flexible and configurable interfaces to BLAS and LAPACK. There are several implementations of each of these libraries (BLAS especially), and I want PLA to support as many of them as possible.

To get these kinds of flexible interfaces, we're going to need to support large swaths of conditionally-compiled code. This will get extremely unweildy if we try to keep all the logic in the few monolithic PMC files I've been writing, so I decided that we're going to need to upgrade our build machinery to support linking in additional C library files into the dynpmc build. This weekend I added that machinery, and started the process of factoring out some of the supporting routines from the various PMC types into those files. With common functions defined in one place, I can go through and make sure all my types are making consistent use of them. Plus, I can start writing up complicated library bindings without worrying about making my large PMC files any larger.

BLAS doesn't actually supply too many routines that the PLA matrix types need. The most important, and only one I call so far, is the GEMM routine, which performs the generalized matrix multiply/addition operation:

Y = αAB + βC

Where α and β are scalar values (FLOATVAL in NumMatrix2D and Complex in ComplexMatrix2D), and A, B, and C are matrices of compatible dimension. This is a very powerful operation, and really the only one from BLAS that I need. Other routines from BLAS are more highly optimized versions of this same routine for specific types of matrices (diagonal, triangular, etc), and operations between matrices and vectors which I really don't support yet. Because there is really only this one routine, and because it's so powerful and common, I've included it as a method directly in the PMC types themselves. I may rethink that decision later, but for now it seems like a decent fit where it is.

LAPACK is a little bit different, however. LAPACK contains hundreds of routines, all of which are named in an idiosyncratic and difficult-to-understand way. These routines are broken up into drivers, computational routines, and auxiliary routines. The driver routines perform the most important operations and typically call sequences of the lower-level computational and auxiliary routines to perform their work. Driver routines include calculating linear least squares, finding eigenvalues and eigenvectors, and singular value decomposition.These are the ones that PLA is going to target first, though access to other, lower-level routines might be nice to have as well, later down the road.

I am torn as to how I want to provide LAPACK bindings in Parrot. The obvious thing I could do is write them all up as methods on the PLA matrix types. This is a system that provides the easiest access, but also starts to get bloated and unweildy very quickly. Plus, if you don't need LAPACK routines for your program, I would be forcing you to load the LAPACK library and create Sub PMCs for all those methods. That can get to be a pretty large cost to pay. I really don't like this idea because it adds huge bloat to my PMC files and adds extra runtime overhead of adding in all sorts of extra method PMCs when we load PLA. Plus, it doesn't scale well, any time we want to add a new LAPACK-based routine we need to add a new method to several PMC types present and future.

One alternative is to provide a separate library of functions and dynamically load it using Parrot's NCI tools. I don't know what the current status of the NCI system is, but without significant improvement there, attempting to load a complex library like LAPACK, even if I provided some simplified Parrot-specific wrappers, would be a nightmare. This option does provide a little bit more flexibility because we can load in the methods that we want without having to deal with ones that we don't. Also, I don't think the distutils library that PLA is using supports creating standalone shared libraries that aren't dynpmc or dynops libraries, so I would have to write all that machinery myself. I'm not sure I really like this option either, for all these reasons.

I'm also torn about whether I want to make some prerequisites for PLA optional. Kakapo, for instance, is only used for the test suite. If you don't have Kakapo installed you can still build and use PLA, you just can't run the test suite. BLAS is currently required because it's used to implement the GEMM method but also the various multiply vtables. I suppose I could make it optional, but I think it would be a little sneaky if different builds of PLA supported a different list of VTABLEs and METHODs for core types, including for something as fundamental as multiplication. LAPACK is a little bit different, because even without it we do have a pretty solid amount of core functionality. I am strongly looking into an ability to make LAPACK an optional prerequisite, and only include bindings for it if we have the library installed. If I wrote a dynamic library wrapper for LAPACK and had the user include that file manually to get those routines, you could still use the core library.

I had also considered the idea (though I've been fighting it) of building some kind of PLALibrary singleton PMC type which would control this kind of stuff. In addition to providing information about library version and configuration options, a PLALibrary PMC could allow dynamic loading of methods, and could forcibly inject those methods into the necessary PMC types if requested. This would allow me to build everything together into a single dynamic library without forcing me to define all the methods upfront which I don't need. I may like this idea the best, though I've been resisting creating a library singleton PMC until now.

There's a lot to consider, and I probably won't make any hard decisions about future plans for the next few days. In the interim I do want to put together some proof-of-concept wrappers for some common LAPACK routines, and start playing with those. If I could get routines to calculate Eigenvalues, Eigenvectors, Inverses, and Singular Value Decompositions, I think that would be a pretty great start.

Andrew Whitworth: GSoC Threads: Chandon's Results

noreply@blogger.com (Whiteknight) — Sat, 21 Aug 2010 12:00:06 +0000

The pencil's down date for the 2010 GSoC program has come and gone, and now we're starting the process of sorting through the rubble and seeing what all got accomplished this summer. I have been purposefully trying not to provide too many of my own status updates for Chandon's project this summer, instead wanting him to post his own updates as he went without me getting in the way. It's his project after all, and I wanted him to get all the credit for all the good work he was doing.

The project is now over, and he's posted a nice final review of what he managed to accomplish this summer. While it isn't everything that he mentioned in his proposal, it still is quite a nice step in the right direction and a lot of the necessary framework is now available to bring threading to Parrot in a real and usable way.

So what did he manage to do? In his branch there is now a more-or-less working implementation of green threads, which means Parrot is getting onto the same level of capability as modern Python or Ruby implementations. You can schedule multiple independent tasks to run and, while they cannot run truly in parallel on separate OS threads or even on multiple processor cores, you can get the illusion of concurrency. It also gives you the ability to run multiple tasks together without having to explicitly schedule them one after the other, or having to manually switch back and forth between them. In a later post I may go into some detail about how this all works, and how it can be used by programs.

This is a pretty significant step forward, and once a few of the final nits get worked out I think we will be able to merge this into Parrot trunk and start making use of the new functionality immediately. However, green threads is only one half of the promised "hybrid threads" that Chandon's proposal hoped to achieve. The other half is the ability to run Parrot code in true parallel on separate OS threads. This was the larger part of the project and definitely the more difficult piece. Today I would like to talk a little bit about why it didn't get done, and maybe motivate some people to look into help completing the work as we move forward.

Let's take a look at a small snippet of normal object-oriented code:

foo.bar();
foo.bar();

Extremely simple, we have an object "foo" and we are calling the "bar" method on it. In a very naive staticly-typed language this would be a simple operation. The compiler would determine the type of foo, calculate the location of the bar routine in memory and would simply call that address twice. This would be extremely fast to execute too, which everybody likes. This would basically be converted to this low-level pseudo-code:

call bar(foo)
call bar(foo)

Now let's move up to a slightly more complicated example: a statically-typed language which allows subclassing and method overriding. Now, the compiler doesn't necessarily know which function in memory corresponds to "foo.bar()", since foo could be of type Foo, or of type FooSubclass, or even FooSubSubclass, and the location of the appropriate bar function would change with each different type. However, for each type, the value of bar does not change. It can be overridden by subclasses, but it cannot be changed for the class itself. Now, the compiler needs to ask foo to get the appropriate method first:

method _bar = get_method(typeof(foo), "bar")
call _bar(foo)
call _bar(foo)

Assuming foo does not change type inside the call to _bar, this code works just fine. Next let's look at a more complicated example still, the case of a dynamic language like Perl or Python. In these languages the class MyFooClass is an object of type Class, and foo is an object of type MyFooClass. Also bar is not a compiled-in constant property of the Class, but is instead a mutable property of it. Inside the call to bar, maybe we change the definition of bar itself inside the class object. Likewise, the definition of routines like "find_method" inside Class can change. Accounting for all this dynamicism, our code changes to look like this:

class fooclass = find_class(foo)
method _get_method = find_method(fooclass, "bar")
call _bar(foo)
class fooclass = find_class(foo)
method _get_method = find_method(fooclass, "bar")

call _bar(foo)

Keep in mind that everything can change inside each call to bar. We can:

Modify the inheritance hierachy of MyFooClass to add or remove parent classes
Add or remove methods in MyFooClass, and any item in it's inheritance hierarchy
Add or remove methods on the object foo itself (like in javascript where foo.bar = function() {...})
Change the class of foo

Let's bring this back to the idea of threading before I go too far off topic. Every time we call a method, or attempt to access a field on an object, we need to look those up in the associated Class object first. Those Class objects need to be global, because all code on all threads need to be able to lookup methods on any object, because we can pass objects between threads and we need to be able to work with them.

If classes are global, and need to be accessible from multiple threads, we inevitably run into the idea of contention: If one thread is changing the definition of foo.bar() at the same time that another thread is attempting to look it up, the thread doing the reading may end up with incomplete, corrupted, or incorrect information.

Global data is the enemy of good multithreading. Global data is a necessity for highly dynamic object systems like those supported by Parrot. Ergo, multithreading on Parrot is hard.

Look at all the global data in Parrot: there's the interpreter object itself, the packfiles, the list of Classes (and the methods/properties that they contain), the list of Namespaces (and the global values they contain), the context graph (and the contents of all registers in those contexts), the opcode table and the MMD signature cache. This is a hell of a lot of globally-available data, and we're going to need to find a sane way to limit corruptive concurrent access to these things. Not only that, but if we have to lock every single access to a global value, which means we need to obtain a lock every time we want to do a method call, our performance is going to be terrible.

By convention I think we can all agree that things like the opcode table should really not be modified when we have multiple threads running. But it's harder to make such a convention that class/method definitions should be treated as constant when multiple threads are running.

What we can do to alleviate some of the performance problems is to create short-term caches for things like Classes and Methods in local threads, with the understanding that without explicit synchronization, the exact ordering of read/write operations between threads cannot be guaranteed and can be scheduled for maximum benefit. If the relative execution times of two operations on separate threads cannot be guaranteed, then it makes no difference to functionality for the user whether those operations happen at random times with respect to each other and whether Parrot manually orders them to improve caching.

Think about it this way: We have two threads, one is writing a global value and one is reading it. These threads are operating in true parallel on two separate processor cores. If we run this program a million times, sometimes the read will randomly occur before the write, sometimes the write will occur before the read, and sometimes they will happen at the exact same moment and cause corruption. Depending on a simple matter of random timing, we could get three very different results from our program. Now, if Parrot implements a heuristic that if a write and a read happen within a very short period of time, Parrot will manually order them so that the read occurs before the write. And, if the read always happens before the write, maybe we don't read at all, but instead use a cached version. Then, only when the write has complete, we update the cache.

Another thing to think about is that we could disallow direct lookups of data in global variables, and give each thread a local copy of global values. Considering the principal I mentioned above, we can be a little bit liberal with the way we handle writes and reads. If a thread modifies its copy of a Class, those changes could be propagated to a global reference copy at a convenient time, and then propagated down to the local copies later, only when it's convenient to do so. Remember, if things are happening at random times compared to each other, it makes no substantive difference to the thread whether it's copy of a global variable is exactly up-to-date or whether it's a little bit lagged behind. That is, the reading thread doesn't know whether the writing thread had a legitimate delay before writing, or whether Parrot manually scheduled the write at a different time.

To get a complete Hybrid Threads implementation in Parrot like what Chandon was envisioning, we are going to have a few steps to take:

We have to break the Interpreter structure up into two parts: One which is truly global and contains data which all threads need, and one which is local for each thread and contains data that the thread needs. The local interpreter may contain a cache or a mirror of data in the global interpreter.
We need to come up with a good sane scheme (which will likely consist of both technical solutions and programmer conventions) to manage access to global values
We need to come up with a good sane scheme for sharing values between threads.Creating a local variable and passing a reference to it to another thread essentially turns it into a global variable and opens up problems with contention. We need a good way to manage this without slowing everything down.

All of these problems are solvable, and if we put forward a concerted effort we could have a decent hybrid threading system in Parrot within the next few months. We almost have a complete working Green Threads system ready to merge now, and hopefully that will be enough for our users while we get the rest of the details of the hybrid system worked out and implemented.

Andrew Whitworth: PLA: More New Methods

noreply@blogger.com (Whiteknight) — Fri, 20 Aug 2010 21:00:02 +0000

In a previous post I discussed some of the common behaviors that the matrix types in Parrot-Linear-Algebra implement. Recently, I've added a few more which I think are pretty cool and worth discussing.

First and foremost, I've finally added some methods to convert between type. There are now methods for convert_to_number_matrix, convert_to_complex_matrix, and convert_to_pmc_matrix. Every type implements all of these methods, even when it would be a no-op. This is so you can take a PMC and if it "does 'matrix'" you can cast it to the type you want to deal with without worrying about spending too much expense. These methods always return a new matrix, so you can keep a copy of the matrix in it's original form and also have a new copy to play with. In the case where the matrix is already in the target format, I create a clone.

Unfortunately, these conversion operations can be a little bit expensive when you're actually converting types. The problem is that the data for the matrices is stored internally in a very dense format. For the Number and Complex matrices, the data is stored internally in the format required by the BLAS library routines. For the number matrix, the values are basically stored together in a single large buffer. For complex matrices, the real and imaginary values are stored together also, alternating positions. Converting one to the other is not easy, since I have to allocate a completely new buffer and iterate over each space individually. So, too many conversion operations can get expensive quickly.

Using these new conversion methods, I have updated some previous methods, like gemm(), the routine which performs the GEMM operation from the BLAS library. You can now pass any matrix types to this method, and it will perform conversions internally to ensure the types match. Here's a fun little NQP example that shows some of the capabilities of this library today:

INIT { pir::load_bytecode("pla_nqp.pbc"); }

my $A := NumMatrix2D.new();
$A.initialize_from_args(2, 2, 1, 2.0, "3", 4);
$A.transpose();
pir::say("A: " ~ $A);

my $B := ComplexMatrix2D.new();
$B.initialize_from_array(2, 2, [1, 2.0, "3+3i", "7+5i"]);
$B.conjugate();
pir::say("B: " ~ $B);

my $C := PMCMatrix2D.new();
$C.fill(4.4, 2, 2);
pir::say("C: " ~ $C);

my $D := $B.gemm(0.5, $A, $B, "1+2i", $C);
pir::say("D: " ~ $D);

You can try it yourself, or you can take my word for it that the result is correct. I've verified it this morning in Octave, and the results are the same (though the Octave script to produce this result is considerably shorter).

PLA is finally starting to get the kind of basic functionality and test coverage that I've been hoping for. With a few more finishing touches on this base, I'm going to start adding new functionality like LAPACK bindings. Specifically, I'm hoping to add in support for some common matrix decompositions, matrix reductions, inverses and eigenvalues. I'm also hoping to get started on the module for Rakudo sometime soon.

Andrew Whitworth: PLA Test Suite Redux

noreply@blogger.com (Whiteknight) — Fri, 20 Aug 2010 02:05:29 +0000

I've been doing a hell of a lot of work on the Parrot-Linear-Algebra test suite in the past few days, and the results are quite spectacular. I've completely reworked the way the suite works, and broke a small handful of monolithic test files into a series of smaller, focused test files. Instead of describing it, I'll just show the results:

t/sanity.t .............................................. ok
t/pmc/pmcmatrix2d.t ..................................... ok
t/pmc/nummatrix2d.t ..................................... ok
t/pmc/complexmatrix2d.t ................................. ok
t/methods/nummatrix2d/convert_to_complex_matrix.t ....... ok
t/methods/nummatrix2d/row_scale.t ....................... ok
t/methods/nummatrix2d/get_block.t ....................... ok
t/methods/nummatrix2d/mem_transpose.t ................... ok
t/methods/nummatrix2d/convert_to_number_matrix.t ........ ok
t/methods/nummatrix2d/initialize_from_args.t ............ ok
t/methods/nummatrix2d/row_swap.t ........................ ok
t/methods/nummatrix2d/iterate_function_inplace.t ........ ok
t/methods/nummatrix2d/fill.t ............................ ok
t/methods/nummatrix2d/initialize_from_array.t ........... ok
t/methods/nummatrix2d/iterate_function_external.t ....... ok
t/methods/nummatrix2d/transpose.t ....................... ok
t/methods/nummatrix2d/item_at.t ......................... ok
t/methods/nummatrix2d/gemm.t ............................ ok
t/methods/nummatrix2d/row_combine.t ..................... ok
t/methods/nummatrix2d/resize.t .......................... ok
t/methods/nummatrix2d/convert_to_pmc_matrix.t ........... ok
t/methods/nummatrix2d/set_block.t ....................... ok
t/methods/complexmatrix2d/convert_to_complex_matrix.t ... ok
t/methods/complexmatrix2d/row_scale.t ................... ok
t/methods/complexmatrix2d/get_block.t ................... ok
t/methods/complexmatrix2d/mem_transpose.t ............... ok
t/methods/complexmatrix2d/convert_to_number_matrix.t .... ok
t/methods/complexmatrix2d/initialize_from_args.t ........ ok
t/methods/complexmatrix2d/row_swap.t .................... ok
t/methods/complexmatrix2d/iterate_function_inplace.t .... ok
t/methods/complexmatrix2d/fill.t ........................ ok
t/methods/complexmatrix2d/initialize_from_array.t ....... ok
t/methods/complexmatrix2d/conjugate.t ................... ok
t/methods/complexmatrix2d/iterate_function_external.t ... ok
t/methods/complexmatrix2d/transpose.t ................... ok
t/methods/complexmatrix2d/item_at.t ..................... ok
t/methods/complexmatrix2d/gemm.t ........................ ok
t/methods/complexmatrix2d/row_combine.t ................. ok
t/methods/complexmatrix2d/resize.t ...................... ok
t/methods/complexmatrix2d/convert_to_pmc_matrix.t ....... ok
t/methods/complexmatrix2d/set_block.t ................... ok
t/methods/pmcmatrix2d/convert_to_complex_matrix.t ....... ok
t/methods/pmcmatrix2d/get_block.t ....................... ok
t/methods/pmcmatrix2d/mem_transpose.t ................... ok
t/methods/pmcmatrix2d/convert_to_number_matrix.t ........ ok
t/methods/pmcmatrix2d/initialize_from_args.t ............ ok
t/methods/pmcmatrix2d/iterate_function_inplace.t ........ ok
t/methods/pmcmatrix2d/fill.t ............................ ok
t/methods/pmcmatrix2d/initialize_from_array.t ........... ok
t/methods/pmcmatrix2d/iterate_function_external.t ....... ok
t/methods/pmcmatrix2d/transpose.t ....................... ok
t/methods/pmcmatrix2d/item_at.t ......................... ok
t/methods/pmcmatrix2d/resize.t .......................... ok
t/methods/pmcmatrix2d/convert_to_pmc_matrix.t ........... ok
t/methods/pmcmatrix2d/set_block.t ....................... ok
PASSED 356 tests in 55 files

I factored out all tests to delegate matrix creation and matrix population routines into a series of type-specific factory objects. With these factories, creation of tests goes extremely quickly:

Write tests using a few methods and values from the factory instead
Create a short stub test for each Matrix type, subclassing the common tests and add a proper factory
There is no step 3.

It's really become quite simple, and I'm much happier with the test suite now. Plus, I've added plenty of new features lately (complete with tests!) and I'm going to talk about that plenty in a later post.

parrot.org: Hybrid Threads: GSoC Project Results

Fri, 20 Aug 2010 01:15:25 +0000

I proposed a pretty ambitious Google Summer of Code project this year. Although I didn't manage to do everything I hoped, I did manage to get a useful subset of threading functionality working in the gsoc_threads branch. In this blog post I will describe what I have working and what more needs to be done.

parrot.org: Parrot 2.7.0 "Australian King" Released!

Wed, 18 Aug 2010 04:20:56 +0000

Never worry about theory as long as the machinery does what it's supposed to do. 
    —Robert A. Heinlein

On behalf of the Parrot team, I'm proud to announce Parrot 2.7.0 "Australian King"

Parrot is a virtual machine aimed at running all dynamic languages.

Parrot 2.7.0 is available on Parrot's FTP site, or follow the download instructions. For those who would like to develop on Parrot, or help develop Parrot itself, we recommend using Subversion on our source code repository to get the latest and best Parrot code.

SHA digests for this release are:

Andrew Whitworth: PLA Status Updates

noreply@blogger.com (Whiteknight) — Tue, 17 Aug 2010 11:49:56 +0000

On Friday night we dropped the kid off with his grandparents for a sleepover. With the apartment to ourselves, my wife and I did what we've been wanting to do for months: she went to bed early and I stayed up to program. I started making some real progress in Parrot-Linear-Algebra, and also uncovered some interesting bugs which needed to be fixed.

I added a short file for adding PLA support to programs written in NQP. The file, pla.nqp, can be included into an NQP program like this once it's installed:

INIT { pir::load_bytecode("pla.pbc"); }

That file loads the PLA binary library, stores a global reference to the library, and registers the type names with P6metaclass. That done, you can start writing more natural-looking programs in NQP. I updated the Gaussian Elimination example I wrote a few weeks ago to use this new bootstrap file (and some new methods I added), which means the example can now run without Kakapo or any other dependencies.

Next up, I started prototyping bindings for the library for Rakudo. These are still preliminary, but I also have a working example of using numerical matrices in Rakudo. Within the next few days I will probably create a complete module for Rakudo (Math::LinearAlgebra, or something like that). I'll post more details as I do more work on it.

I added a new item_at method which returns and optionally sets a value at given coordinates in a matrix. This method is common to all my core matrix types. The list of common methods for these matrix types is:

resize() : pre-allocate size for the matrix, growing (never shrinking) it to hold a certain size
fill() : Fill a matrix, or a region of a matrix, with a constant value. Automatically resize if necessary
transpose() : Transpose (swap rows with columns) the matrix lazily
mem_transpose() : Eagerly transpose the actual memory contents
iterate_function_inplace() : Execute a function for every element of the matrix, replacing that element with the function result
iterate_function_external() : Execute a function for every element of the matrix, creating a new matrix with the results. This is similar to Perl's map function.
initialize_from_array() : Insert values into the matrix from an array
initialize_from_args() : Similar to the _from_array variant, but initializes the matrix using elements from a slurpy argument list
get_block() : Return a block, or "submatrix" from the matrix
set_block() : Set a block in the matrix
item_at() : New this weekend, gets or sets a value in the matrix at the specified coordinates

There are maybe a handful of other methods I would like to add, but in terms of the core types, this is the standard API (plus a series of VTABLE calls, which are standard). For mathematics types I also have GEMM (the BLAS-based matrix multiply operation), and elementary matrix operations (add rows, swap rows, scale rows). This is not a shabby interface at all, and can start to be used for some real-world applications.

This weekend I also started seeing some very weird errors in the PLA test suite. Tests were running fine, but I was seeing exceptions (and, in one case, a segfault) occur after all tests had passed. This sounded very similar to another problem I've seen in the past. Here's the test that set it off. Can you spot the problem?

method test_METHOD_iterate_function_inplace_TRANSPOSE() {
    my $m := self.fancymatrix2x2();
    $m.transpose();
    my $n := self.matrix2x2(self.fancyvalue(0) * 2, self.fancyvalue(2) * 2,
                            self.fancyvalue(1) * 2, self.fancyvalue(3) * 2);
    my $sub := -> $matrix, $value, $x, $y {
        return ($value * 2);
    };
    $m.iterate_function_inplace($sub);
    assert_equal($m, $n, "external iteration does not respect transpose");
}

What's maddening is that this test has been a problem for months, but never caused a failure. It was silently wrong, probably since the day I wrote it.

See it yet?

The problem is that return statement. What should happen is this: The pointy block creates an anonymous subroutine, which I pass to the iterate_function_inplace method. The method should loop over ever element in our matrix and call that pointy block for each one. The result should be the same matrix with every element multiplied by two.

What actually happens is this: the iterate_function_inplace method, in order to call the callback, must recurse on the C stack into a new runloop function. The pointy block executes in this inferior runloop. However, where things get weird is that return statement. In NQP, returns are performed by constructing and throwing control exceptions. In the case of the pointy block (and I'm not sure whether or not this is a bug), the return statement jumps to the return handler for the test_METHOD_iterate_function_inplace_TRANSPOSE() function, not the anonymous sub.

The sub executes after having only called the callback once, it never hits the final assertion (which, it turns out, would have failed). Control flow continues inside the inner runloop until the end of the test program, then the C runloop function returns, tries to continue executing from that point, and things go to hell.

The solution is really quite simple. Change this:

my $sub := -> $matrix, $value, $x, $y {
    return ($value * 2);
};

into this:

my $sub := sub ($matrix, $value, $x, $y) {
    return ($value * 2);
};

Problem solved, and now more tests are legitimately passing.

It's been a productive couple of days in PLA, and I'm hoping I can keep this momentum up in the days ahead. I need to finish implementing my GEMM wrapper for ComplexMatrix2D, and then get started on the Linear Algebra module for Rakudo.

Andrew Whitworth: Kakapo Fixed!! (Mostly)

noreply@blogger.com (Whiteknight) — Thu, 12 Aug 2010 11:50:41 +0000

The lovable Austin Hastings has gotten Kakapo mostly working as of yesterday. It's not 100%, but I'm able to use it for some purposes such as getting the PLA test suite passing again.

When I tried to build PLA last night, for the first time in a long time, the build failed. Some of the string handling functions I was using in the get_string VTABLEs were using old, deprecated functions in the string API. I took this as my opportunity to rewrite a lot of the logic to use the new StringBuilder type instead. This should improve performance and bring PLA up to some of the modern best-practices of the Parrot community.

When I tried to run the tests, however, KABLAMO. Kakapo wasn't happy and the tests wouldn't run. I tracked down a small series of problems, and put together a hasty patch. Austin pointed out that this wasn't the real solution, so he didn't want me to push it to the master at Gitorious. Instead, I pushed to my mirror at Github. This fix out of the way, the vast majority of the PLA test suite runs, and most tests were passing. This morning, I pushed another patch to my Kakapo mirror. After that, I'm proud to say, the PLA test suite passes 100% of important tests. That means I can start making preparations for an official release, and then begin development on cool new features.

Some people have asked me why I've gone through all this hassle; if I had just rewritten my testsuite without Kakapo it could have been working a long time ago. The answer to that question is pretty simple: I want the abstraction and insulation that Kakapo provides.

To look at it another way, consider this: NQP has a test suite of it's own that needs to pass before it can make a release. If I'm using NQP, I know that the behavior NQP provides will be a little more stable than the underlying Parrot VM. However, NQP isn't a perfect overlay for Parrot: There are some features that Parrot provides that NQP doesn't cleanly wrap. To get an abstraction layer over those features, I need a framework like Kakapo. This is all not to mention that several other high-profile projects use NQP, so even if it's test suite isn't all-encompassing, the test suites of projects which build on top of NQP probably cover more ground.

Kakapo likewise has a test suite, and that needs to pass before Kakapo gets released. If I'm using Kakapo, I can be certain that the API that it and NQP provide together should be relatively stable and are tested to work before they are released. Since we are talking about tracking Parrot trunk with PLA development, this stability is very welcome.

What we've run into recently is a rare confluence of events where highly disruptive changes in Parrot caused major breakages in Kakapo. When Kakapo got fixed, the PLA test suite ran perfectly without any changes to the suite itself. Considering the magnitude of changes to Parrot recently, the ability for me to continue running the test suite without alterations is a pretty big and important aspect. Sure there was some down-time, but once our dependencies were fixed, our stuff worked again like magic.

Kakapo isn't 100% yet, but Austin is still working on it and I'll be lending a hand. I'm also going to start making some progress in PLA, and see what I can do to help out with some of the performance improvements and other big tasks that are going on in Parrot too. Expect to see many more updates on things in the coming days and weeks.

Khairul: Almost there.

Fri, 06 Aug 2010 20:06:20 +0000

I’ve been spending the past few weeks cleaning up and finalising the interface used to instrument Parrot. So here’s a recap of the major changes:

Refactored Instrument.pmc
Previously, Instrument.pmc serves as the interface to instrument the ops and to attach probes into the various components of Parrot. Refactoring the runcore-related functions out into InstrumentRuncore.pmc allows me to make Instrument.pmc the main interface to create and attach the probes. As of now, it has the following methods:
- instrument_op : Returns an Instrument::Probe instance that can be used to inspect ops being executed.
- instrument_gc: Returns an Instrument::Event::GC instance that can be used to get notifications whenever a GC function is called through the interpreter’s GC_Subsystem interface.
- instrument_class: Returns an Instrument::Event::Class instance that is tied to a particular class allowing instrumenting the class’ VTABLE entries and methods.
- instrument_object: Returns an Instrument::Event::Object instance that is tied to a particular object allowing instrumenting the object’s VTABLE entries and methods.
An example of usage will be as follows:
```
# Instrument the ops.
$P0 = new ['Instrument']
$P1 = $P0.'instrument_op'()
$P1.'callback'('some_sub')
$P1.'inspect'('op_name')
$P0.'attach'($P1)
```
with similar usages for the other methods listed above. Instrument.pmc also takes over the function of actually calling the callbacks, a function previously handled by EventDispatcher. For this task, it has 3 methods:
- register_eventhandler(event, probe)
- remove_eventhandler(event, probe)
- raise_event(event, data)
whose functions are rather self explanatory.
Post execution callbacks
Previously, most of the callbacks are called before execution, with no way to define whether the callback is to be called pre or post execution of the op/function/etc. With this refactor, all callbacks are called pre-execution, but can now return an invokable, be it a Sub or an NCI instance that will be called post-execution. An example will be as follows:
```
.sub pre
    # This is called before execution.
    # ...
    .param pmc data
    .param pmc instrument
    .param pmc probe
    $P0 = get_global 'post'
    .return($P0)
.end

.sub post
    # This is called after execution.
    # ...
    .param pmc data
    .param pmc instrument
    .param pmc probe
.end
```
Along with this change, all callbacks are now passed 3 parameters, with data being a PMC that is an instance of InstrumentOp for op probes or a Hash for other kinds of probes, instrument being the Instrument instance itself and probe the instance that the callback associated with.

With all that done, I’m now focusing on documentation and dealing with the bugs I’ve uncovered in my code, some new, some old and some not yet known.

parrot.org: Moving to multiple OS threads

Thu, 05 Aug 2010 00:12:53 +0000

Now that I have the basic green threads (Task) API basically working, it's time to start mixing in OS threads. I plan to do that in two steps: First I'm going to use OS threads to solve the key issue with the green threads implementation - blocking IO. The result of this will be basically equivilent to CPython threads-with-GIL. The second step - real parallel thread execution - will have to wait for after GSoC pencils down.

parrot.org: PAST Optimization: plans for the final weeks of GSoC

Wed, 04 Aug 2010 02:23:12 +0000

I spent this last week working on Tree::Optimizer (and getting distracted by Rakudo *).

I've got the functionality described in my blog post last week mostly finished. The only thing remaining is making sure that recursive passes correctly handle nulls, which I'm about to work on.

The Google Summer of Code is almost over. Next Monday is the "suggested 'pencils down'" date. The following Monday is the "firm 'pencils down'" date. After that are final evaluations.

Here's my plan for the remainder of GSoC:

Merge the pass-manager branch of my project's repo into master.

parrot.org: The extra pointer has to go.

Fri, 30 Jul 2010 22:37:28 +0000

I reached the midterms mostly as scheduled, and NFG is pretty much feature complete now. There's still some stuff to do here and there, but the 'big ticket' items are done. So, I have been looking at what needs to be done before the gsoc_nfg branch can be merged back into trunk. And that means giving a hard look at all of the places where I've cut corners and see if they can be made better. It's mostly minor stuff, like leaving out a cast, or not paying attention to const mismatches in a few places. Most of it is just a matter of code cleanup. Until you see the extra pointer I added to string headers.

Patrick Michaud: Rakudo Star - an "early adopter" distribution of Perl 6

Thu, 29 Jul 2010 12:28:04 +0000

On behalf of the Rakudo and Perl 6 development teams, I'm happy to announce the July 2010 release of "Rakudo Star", a useful and usable distribution of Perl 6. The tarball for the July 2010 release is available from http://github.com/rakudo/star/downloads.

In the Perl 6 world, we make a distinction between the language ("Perl 6") and specific implementations of the language such as "Rakudo Perl". "Rakudo Star" is a distribution that includes release #31 of the Rakudo Perl 6 compiler [1], version 2.6.0 of the Parrot Virtual Machine [2], and various modules, documentation, and other resources collected from the Perl 6 community. We plan to make Rakudo Star releases on a monthly schedule, with occasional special releases in response to important bugfixes or changes.

Some of the many cool Perl 6 features that are available in this release of Rakudo Star:

Perl 6 grammars and regexes
formal parameter lists and signatures
metaoperators
gradual typing
a powerful object model, including roles and classes
lazy list evaluation
multiple dispatch
smart matching
junctions and autothreading
operator overloading (limited forms for now)
introspection
currying
a rich library of builtin operators, functions, and types
an interactive read-evaluation-print loop
Unicode at the codepoint level
resumable exceptions

nested package definitions
binary objects, native types, pack and unpack
typed arrays
macros
state variables
threads and concurrency
Unicode strings at levels other than codepoints
pre and post constraints, and some other phasers
interactive readline that understands Unicode
backslash escapes in regex <[...]> character classes
non-blocking I/O
most of Synopsis 9
perl6doc or pod manipulation tools

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources.

Rakudo Star also bundles a number of modules; a partial list of the modules provided by this release include:

Blizkost - enables some Perl 5 modules to be used from within Rakudo Perl 6
MiniDBI - a simple database interface for Rakudo Perl 6
Zavolaj - call C library functions from Rakudo Perl 6
SVG and SVG::Plot - create scalable vector graphics
HTTP::Daemon - a simple HTTP server
XML::Writer - generate XML
YAML - dump Perl 6 objects as YAML
Term::ANSIColor - color screen output using ANSI escape sequences
Test::Mock - create mock objects and check what methods were called
Math::Model - describe and run mathematical models
Config::INI - parse and write configuration files
File::Find - find files in a given directory
LWP::Simple - fetch resources from the web

These are not considered "core Perl 6 modules", and as module development for Perl 6 continues to mature, future releases of Rakudo Star will likely come bundled with a different set of modules. Deprecation policies for bundled modules will be created over time, and other Perl 6 distributions may choose different sets of modules or policies. More information about Perl 6 modules can be found at http://modules.perl6.org.

Rakudo Star also contains a draft of a Perl 6 book -- see "docs/UsingPerl6-draft.pdf" in the release tarball.

Rakudo Star releases are created on a monthly cycle or as needed in response to important bug fixes or improvements. The next planned release of Rakudo Star will be on August 24, 2010.

[1] http://github.com/rakudo/rakudo
[2] http://parrot.org/

Andrew Whitworth: Kakapo and Parrot-Linear-Algebra

noreply@blogger.com (Whiteknight) — Wed, 28 Jul 2010 22:00:01 +0000

Parrot's 2.6 release is out and now everybody is holding their breath in anticipation of the new Rakudo Star release tomorrow. I'm still fighting with my own release project: Kakapo.Kakapo got broken pretty severely when Parrot was changed to no longer store methods in namespaces, and I have been unable in several weeks to make it work again. Austin has been busy with other projects during this time too, so it's been me by my lonesome to try and get this thing working again.

The problems are three-fold: First, Austin's plan for Kakapo was ambitious and the framework does a hell of a lot of things, not all of which I fully understand. Second, NQP and P6object have progressed in some significant ways since Kakapo was originally inked out, and much of the workaround code that Austin wrote to bypass some missing features no longer seems necessary. Third, we have this issue with methods not working nicely with NameSpaces anymore, which makes all of Kakapo's method-injection logic break.Without method injection happening, and happening early enough, all the code in Kakapo which relies on the new methods is broken.

Tene put together a patch for NQP which allows methods marked "our" to be stored in NameSpaces again. This should simplify some of the new logic in Kakapo though I've been too busy in the past few days to test it. Hopefully this patch of his represents a large portion of the fix to get Kakapo working again. I doubt it will be everything that's necessary, but it will be a nice start.

Without working Kakapo I haven't done any work whatsoever on Parrot-Linear-Algebra. PLA uses Kakapo to implement all it's unit tests, and I'm happy enough with the new testing framework that I do not want to move it to use anything besides Kakapo. In the worst case I would consider trying to break out a testing sub-framework from Kakapo if we can't fix the entire framework, but I haven't gotten to that point yet and likely won't have the time for it in the near future. I would much rather fix the framework as a whole first, and then maybe talk about splitting out the unit test stuff into a separate sub-library later, if I still feel like that's a beneficial path to take.

What I would really like to be doing is getting back into PLA development. I had hoped to get some wrappers together to make it work with Rakudo Star, but without working tests I haven't cut a suitable release and without a release it won't be included in the Rakudo Star bundle anyway.

My hope is that within the next few days I can get Kakapo working again. At least I need it working enough that I can get the unit tests for PLA going. With Tene's fix, some hard work, and some trial-and-error I think it will be possible.

parrot.org: Documentation and information collection

Wed, 28 Jul 2010 02:14:43 +0000

Tutorial to learn the basics of the PIR syntax and beginner's introduction to Parrot

NQP - Not Quite Perl

The target Lorito

Khairul: Method madness

Tue, 27 Jul 2010 22:06:11 +0000

Well, the past two weeks haven’t been very productive. Other than the usual addition of tests, I’m slightly stuck on how I’m approaching instrumenting methods and this post is meant to be a sounding board.

In Parrot, methods of a class can be defined in a few ways. First is by defining a sub in a namespace with the same name as the class and annotating the sub with :method. Second is through the use of the addmethod op. Lastly, it seems all PMCs are classes and PMCs can define methods in C. So at the very least, methods can be either Sub or NCI instances (or rather, invokables), and through the use of the addmethod op, an invokable can be a method to more than one class. So the past week I added support to instrument methods of a class, raising an event whenever any instances of the class invokes the method. What I did not foresee was that due to the ability to “share” methods, an event would also be raised for another class that did not instrument the “shared” method. And this unintended consequence is further exacerbated when the instances themselves are instrumented (I added the ability to instrument on a per-object basis too, although it seems like it’s not working too well now).

So a step back to reflect is in order. I would need to:

Keep a list of classes and instances that instrument any given method.
When the method is invoked, check the invocant against this list and raise the event if required.
1. The invocant can be found in the CURRENT_CONTEXT (I think)

It sounds like a plan, albeit not very fleshed out. Time to find out if it works.

parrot.org: Cleaning up and speeding up optimizations with Tree::Optimizer

Sun, 25 Jul 2010 04:10:14 +0000

After a false start earlier this week, I've begun implementing something like LLVM's PassManager. It can be found in the pass-manager branch of my project's git repository.

Today, I'm going to talk about adding optimizations to compilers using my GSoC project without Tree::Optimizer and with it. Note that most of the features of Tree::Optimizer that I describe here are not yet implemented.

parrot.org: Green Threads: A Classic Example

Fri, 23 Jul 2010 04:18:40 +0000

Now that I have green threads basically working with the API functionality I discussed in my last blog post, I've been working on coming up with tests/examples that clearly demonstrate both how green threads in Parrot work and that they work as advertised. With that goal in mind, this blog post is about a classic concurrent programming example: The Sieve of Eratosthenes.

parrot.org: Parrot 2.6.0 "Red-rumped" supported release.

Wed, 21 Jul 2010 03:53:39 +0000

What we call the beginning is often the end
And to make an end is to make a beginning.
The end is where we start from.
   —T. S. Eliot, Four Quartets

On behalf of the Parrot team, I'm happy to announce Parrot 2.6.0
"Red-rumped." Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages.

Parrot 2.6.0 is available on Parrot's FTP site, or follow the
download instructions at http://parrot.org/download. For those who would like to
develop on Parrot, or help develop Parrot itself, we recommend using Subversion

parrot.org: The uneventful mid-term week and the future of PAST/Tree Optimization

Sun, 18 Jul 2010 23:07:10 +0000

This has been an uneventful week in my GSoC work.

Parrot's 2.6 release is Tuesday, and I've taken on the task of updating Squaak to modern NQP-rx prior to it. Because of that and my GSoC midterm evaluations, I haven't spent much time hacking on Tree::Pattern or on optimizations using it this week. I plan to finish working on Squaak tonight or tomorrow, commit my fixes to trunk, and finally get back to work on my GSoC project. I'm a little behind schedule now because of that, but I plan to work hard on optimization for the rest of GSoC.

Jonathan Worthington: Last Post

Sun, 18 Jul 2010 02:58:24 +0000

Well, on use.perl.org, anyway. I'd like to thank those who run this place for doing so, as it's been the place I've blogged about my Perl 6 stuff for the last couple of years. However, I've found the user experience of WordPress a lot more enjoyable - it was great for the Perl 6 Advent Calendar - and figure I'll blog more if I more enjoy doing so.

So, from now you'll be able to read my Rakudo news and other Perl 6 related ramblings from my shiny new blog. For those reading through Planet Six, no action required - I'll arrange for my new blog to be aggregated there.

See you over there.

parrot.org: Green Threads: The Task API

Thu, 15 Jul 2010 00:32:45 +0000

Now that I have pre-emptive task switching between green threads basically working, I need to figure out what the API to interact with them wants to look like. Additionally, I need to clean up this scheduler thing that I've mauled in order to make the API and terminology consistent.

Khairul: Week 7: Refactor refactor

Mon, 12 Jul 2010 17:55:23 +0000

Done this week:

Added in generator scripts.
cotto suggested it would be a good idea to put the scripts I used to generate the stubs in tools/build, so that got done. That done, I added a way to get at the parameters passed to the vtable and GC functions. The example below show’s what I mean:
test-invoke.pir

.sub '' :anon :load :init
    load_bytecode 'Instrument/InstrumentLib.pbc'
.end

.sub main :main
    .param pmc args
    $S0 = shift args

    $P0 = new ['Instrument']

    $P2 = get_hll_global ['Instrument';'Event'], 'Class'
    $P1 = $P2.'new'()
    $P1.'callback'('class_handler')
    $P1.'inspect_class'('Sub')
    $P1.'inspect_vtable'('invoke')

    $P0.'attach'($P1)

    $S0 = args[0]
    $P0.'run'($S0, args)
.end

.sub class_handler
    .param pmc data

    $I0 = data['line']
    $P0 = data['parameters']
    $I1 = $P0
    $P1 = $P0[1]

    print 'Line: '
    print $I0
    print ' with '
    print $I1
    say ' parameters.'
    print 'address: '
    say $P1
.end

test-script.pir

.sub main :main
    test()

    say 'Done'
.end

.sub test
    say 'Invoke!'
.end

Which yields the following output:

Line: 3 with 2 parameters.
address: 0
Line: 25 with 2 parameters.
address: 1006b0050
Invoke!
Done

Along the way, I discovered a small bug in src/pmc/pointer.pmc, in line 162:

return Parrot_sprintf_c(INTERP, "%s", PARROT_POINTER(SELF)->pointer);

Subtle, innocuous looking code, until you realise that it’s supposed to print an address, not a string.

Partially implemented Instrument::Event::Class
The code above shows that it is somewhat working. What is missing is instrumenting methods, removing the hooks, and instrumenting dynamically loaded classes.
Massive refactoring of InstrumentGC and InstrumentVtable
Both InstrumentGC and InstrumentVtable work pretty much the same way. So by refactoring the common code out into InstrumentStubBase, most of the remaining code in InstrumentGC and InstrumentVtable are generated by the scripts tools/build/gen_gc_stubs.pl and tools/build/gen_vtable_stubs.pl. Refactoring the code was somewhat in the back of my mind, and cotto also mentioned it in last week’s meeting with him. Trying to complete Instrument::Event::Class was the thing that spurred this refactoring, as I found out more and more how similar both InstrumentGC and InstrumentVtable is. Along with the refactoring, I took a step back and reevaluated how my mappings for various items were done, well, that got changed too. So did the event dispatcher, again.
Not as massive refactoring of EventDispatcher
So this is the second week that EventDispatcher got refactored again. So now it can pass events such as Class::Sub::vtable::main::invoke, so that’s category, class name, vtable, vtable group and the item itself, which is more flexible that the previous case which was to split it into 3 components.

What did not get done was most of last week’s goals. I got sidetracked into refactoring and ended up not doing the goals. Furthermore, I found out that I need to take into account inheritance when instrumenting the vtable. As an example, EventHandler extends Sub, so when I instrument Sub’s vtable, the stub entries got imported into EventHandler, leading to fireworks that I least expected. Ah well, at least while figuring it out I discovered how to use XCode’s debugger on parrot. So long command line gdb! And good riddance.

Goals this week will be to complete last week’s goals, since this week, week 8 is supposed to be code review, and most of that got covered last week.

parrot.org: Tail-call elimination and moving to github

Mon, 12 Jul 2010 06:15:17 +0000

My main goals this week have been getting a tail-call elimination optimization to work in NQP-rx and moving my GSoC project to an external repository to make it easier for other projects to use it. Both of these goals have been successful. You can find the tail-call elimination optimization here. My GSoC project can now be found on github or installed via Plumage with ./plumage install tree-optimizations.

Andrew Whitworth: Lorito: A Design

noreply@blogger.com (Whiteknight) — Sat, 10 Jul 2010 22:42:09 +0000

Cotto has been putting together a really great checklist for Lorito, the new microcoding approach that the Parrot developers are hoping to implement soon. The first step on the checklist, creating a proper compiler for our ops code DSL is already complete. Today at #parrotsketch Allison also mentioned some things about Lorito, and how we should start focusing more effort on it. I, for one, am very happy about that.

In this post I'm going to play architect and spell out a vision for Lorito. At the very least, this will be a good play-excercise. I don't expect everybody to like all the things I say here, but I do expect this post to generate some thought and discussion.

On the ground floor, Lorito needs to be able to do what C code does now. Actually, it really needs to be able to do much of what the underlying hardware machine does now. C is just a useful abstraction over the hardware, one that people are familiar with. What we don't want to be doing is creating our own ABI, or creating new low-level calling conventions or anything like that. Those kinds of problems are already resolved and if we want to be able to tap into the myriad of existing libraries we will want to stay compatible with existing conventions.

We want Lorito to interact with raw integers and pointers, including pointer dereferences and pointer arithmetic. This is important so that we can easily work with C-level arrays and pointers. I can't think of a modern architecture where pointers and integers are different sizes and are treated differently, but I can't say I'm familiar with every architecture in current use. I don't foresee any huge problems with creating an opset that works transparently with integers and pointers. We also want Lorito to be able to call C-level functions, including indirect calls through a function pointer. There's no hard requirement yet stated that Lorito needs to be binary compatible with compiled C code, but I think we can all agree that such would be a major benefit. In fact, if we had a utility that could compile Lorito directly into C code, that would be the best intermediate step we could take. As I will discuss next, such a tool would make the conversion of Parrot to using Lorito easier in the long run. We probably don't want Lorito-to-C conversion to be a permanent part of our build, but it does get us moving now.

The PASM ops are currently written in C. If Lorito does everything that C can do, eventually they could all be rewritten in Lorito instead. That raises the interesting idea that groups of Lorito ops can be composed into more complex higher-level ops. PASM then becomes little more than a set of predefined convenience compositions, though it certainly does not represent the complete set of possible compositions. Dynops would just be custom predefined Lorito compositions and, since they wouldn't be defined in machine-code libraries, their definitions could be included directly in bytecode for later use.

In fact, while we are moving down this though path, we get to the idea that we could identify repeated patterns of Lorito ops at compile time, and define custom compositions on the fly. This allows us to compress packfiles for size without really adding all the overhead of general-purpose compression algorithms. When optimizing code, if we can apply an optimization to any composed op, even those identified on the fly, those optimizations can be immediately used anywhere the composition op is. This allows us to prune branches out of the parse tree when optimizing quickly.

Any language for which a compiler exists to convert it to Lorito can be considered an "overlay" language for Lorito. We can then use any overlay language any place where we can use Lorito. For instance, if NQP is converted to output Lorito (and composition ops) directly, we can then use NQP to define all the PASM ops, all the core PMCs, and several parts of Parrot's core. That would be quite the turnaround from how NQP is used currently.

Conversely, almost any C code can be rewritten in Lorito, so long as we have a step in the build to preprocess Lorito back into valid C code. In fact, I see no reason why we cannot interleave code written in both languages, in a manner analogous to how the Q:PIR construct allows PIR code to be written in NQP source. If we had a preprocessor that scanned code files and converted Lorito ops into equivalent C code line-by-line, we could start slowly rewriting much of Parrot, if not all of it, in Lorito. Then, once we have converted all the code over that we need, we could change the build process to convert those code files to bytecode instead of machine code, and run large parts of Parrot's code through the runcore.

Alternatively, instead of writing core code in Lorito directly, we could write core code in any Lorito overlay language. NQP comes to mind here.

The huge benefit to be had is that if user programs are written in a Lorito overlay, and Parrot itself is written primarily in Lorito or an overlay, our eventual JIT can record traces across the API boundary, and optimize hot sections of the Parrot core on the fly at runtime.

Here ends the part of the blog post concerned with overarching architecture discussions. Let's get down to the nitty-gritty details.

Lorito is going to want some basic arithmetic operations: addition, subtraction, multiplication and division on integers (pointers) and floating point numbers. We're also going to want modulus, address-of (C &) and pointer dereference (C unary *) for integer/pointer types, int-to-float and float-to-int cast operations. Logical operations are probably essential as well: and, or, xor, not. Logical inversion (C unary !) would probably be necessary as well. Assuming we can find a way to share ops for int/pointer and float operands (which I doubt we can do in an elegant way), that's still about 20 ops we're going to want to even perform basic manipulations on numbers. I can't imagine any modern, robust, mature virtual machine which doesn't perform these operations readily.

On top of basic mathematical operations, we're going to need some important operations for calling subroutines: Passing and retrieving arguments, calling subroutines, returning with values. I don't know how low-level Lorito intends to get, but let's face the reality of the world: Hardware machines are stack-based. Every library function we want to call is probably going to be passing arguments on the system stack. If we want to be defiant and call these ops "pass_arg" and "retrieve_arg" to hide the fact that Parrot is using the system stack, that's fine by me. The "stacks are evil" mantra, while occasionally misguided, is definitely firmly ingrained in Parrot culture.

In a minimalist world, the only ops we would really need are ops to call functions with passed arguments. We could implement every other op as a huge library of functions to call. Of course this would be extremely sub-optimal, but it does open a new possible avenue: Any operation which is sufficiently uncommon could be turned into a function call instead of having a dedicated op. We could encapsulate the call in a composite op. There are lots of options here, we have to weigh the desire to have a smaller op set against the need to have ready access to a huge myriad of low-level operations.

Having ops to do basic ops is necessary but not sufficient. I don't think we can sit back on our laurels and be happy with an op set that only does a subset of what C can do. That's worthless. Parrot needs to provide access to it's PMC and STRING types, and also to its interpreter. We want ops to load registers with values from the constants table in the bytecode file. We want to treat PMCs and STRINGs as opaque pointers at the user-level. They aren't like other pointers which can be manipulated, they are GCable objects that Parrot treats specially. We don't, for instance, want to be passing a PMC pointer willy-nilly to an arithmetic operation. We also don't want such a mistake to imply a call to one of the addition VTABLEs.

That said, we want to be able to get the VTABLE structure from a PMC, introspect information about that, and be able to call the various VTABLE interface functions there without having to calculate pointer offsets. This might make a good use of composite ops, where we can still do the pointer derefs in Lorito, but hide the nonsense behind some PASM-levle composite ops. We already have most of these ops already, but I think we're going to want to rework some of them. We definitely need to radically reduce the number of VTABLE interface functions, or else interacting with all of them from Lorito will be extremely ungainly.

So far as we are talking about a major redesign of the system, maybe it's time to start considering an idea chromatic has been talking about with making all vtables into methods so we can share a common lookup mechanism, common dispatch mechanism, common MMD behaviors, common inheritance behaviors, etc. That's not a bad idea, but we either need dramatic improvements to PCC performance, or we need to create a PCC fast-path for calls which are made without constructing a call object and without doing too many marshalling operations to get arguments where they need to be.

In fact, on a tangentially-related topic, I definitely think PCC should have a fast-path for functions which do not participate in MMD and which have fixed argument lists containing only positional args (no :optional, no :slurpy, no :named, no :call_sig, etc). This is, I think, an extremely common case and there are plenty of optimizations to be had here if we want them.

To round out the low-level opset, we need ops to move data between registers, between registers and memory, and maybe even ops to move data between memory locations directly, without moving them to a register first.

That's my conception for Lorito: probably 64 ops or less, a capability to add composite ops, treating PASM as a series of common composite ops, and the ability to write low-level code interchangably in C or Lorito. I think this plan for it provides a great development path without too many abrupt stops or course changes. We are quickly going to find ourselves in desperate need of a robust, modern JIT. We are going to be able to move to a proper precise GC since all the Lorito code will be using Parrot's registers to store working values, not relying on stack space which will need to be traced.

I am highly interested in hearing what other people have to say about Lorito, and what changes Parrot should be considering so long as we are writing this blank check for a massive refactor.

Khairul: Week 6: Vtable madness

Mon, 05 Jul 2010 18:27:56 +0000

Done the past week:

Finished up instrumenting GC.
GC events are now fully instrumented, with the exception of the following 3 functions (is_blocked_mark, is_blocked_sweep, get_gc_info). An example of how to use this is shown below:

gc_do_gc_mark.pir:

.sub '' :anon :init :load
    load_bytecode 'Instrument/InstrumentLib.pbc'
.end

.sub 'main' :main
    .param pmc args
    $S0 = shift args

    # Create an Instrument::Event::GC object.
    $P0 = get_hll_global ['Instrument';'Event'], 'GC'
    $P1 = $P0.'new'()
    $P1.'callback'('do_gc_mark_callback')
    $P1.'inspect'('do_gc_mark')

    $P2 = new ['Instrument']
    $P2.'attach'($P1)

    $S0 = args[0]
    $P2.'run'($S0, args)
.end

.sub 'do_gc_mark_callback'
    .param pmc data

    $I0 = data['line']
    $S0 = data['file']
    $S1 = data['type']

    print '('
    print $S1
    print ') at line '
    print $I0
    print ' in file '
    say $S0
.end

gc_sample.pir:

.sub main :main
    meh()

    sweep 1
    collect

    say "End main"
.end

.sub meh
    $P0 = new ['Hash']

    $I0 = 0
    LOOP:
      $P0 = new ['String']
      inc $I0
    unless $I0 > 1 goto LOOP

    sweep 1
    collect

    say "End"
.end

Running the command: “./parrot gc_do_mark_sweep.pir gc_sample.pir” will yield the following output:

(do_gc_mark) at line 19 in file gc_sample.pir
End
(do_gc_mark) at line 4 in file gc_sample.pir
End main

Refactored EventDispatcher.
Previously, my implementation of EventDispatcher only allows registered handlers for specific events, such as ‘Instrument::Event::GC::administration::do_gc_mark’. With the implementation of InstrumentGC, this design was rather inadequate, as I found out when I wanted to register a handler for all GC events, or a certain subset of GC events.

So refactoring and changing the internals of EventDispatcher came as a natural consequence of that. For what I wanted to do, I found that splitting the event type into 3 parts, namely Category, Group and Specific would do nicely for all cases that I could think of. To give an example, ‘GC::allocate::allocate_pmc_header’. If I’m only interested in the specific event, I can just register a handler for ‘GC::allocate::allocate_pmc_header’. If I’m only interested in events in the ‘allocate’ group, I can register a handler for ‘GC::allocate’. Similarly, for all GC events, I can register a handler for ‘GC’.

Thinking ahead for what I want to do in week 7, this dovetails nicely with Classes. As an example, ‘Class::ResizablePMCArray::push’, ‘Vtable::ResizablePMCArray::push_pmc’. On hindsight, maybe refactoring it again to handle more levels would be better, as looking above for the Vtable example, adding 1 more level would allow catching of push_* vtable entries as a group, ie. ‘Vtable::ResizablePMCArray::push::push_pmc’.
Added tests for EventDispatcher.
Rather self-explanatory.
Added tests for InstrumentGC.
Rather self-explanatory too.

Initial cut of InstrumentVtable.
As of now, all vtable entries have a working stub that can be attached and removed at will. What is missing is getting the information about the arguments to the vtable entry. As an example, VTABLE_push_pmc(INTERP, obj, pmc), currently, only the information ‘push_pmc’ can be obtained, with obj and pmc on the way. As an example:

vtable_push_pmc.pir

.sub '' :anon :load :init
    load_bytecode 'Instrument/InstrumentLib.pbc'
.end

.sub main :main
    .param pmc args
    $S0 = shift args

    $P0 = new ['Instrument']
    $P1 = new ['InstrumentVtable'], $P0
    $P2 = $P0['eventdispatcher']

    # Register a handler
    $P3 = get_global 'class_handler'
    $P2.'register'('Class', $P3)

    # Instrument push_pmc of class ResizablePMCArray
    $P1.'attach_to_class'('ResizablePMCArray')
    $P1.'insert_vtable_hook'('push_pmc')

    $S0 = args[0]
    $P0.'run'($S0, args)
.end

.sub class_handler
    .param pmc data

    $I0 = data['line']

    print 'Line: '
    say $I0
.end

vtable_test.pir

.sub main :main

    $I0 = 0

    LOOP:
      if $I0 > 5 goto DONE

      $P0 = new ['ResizablePMCArray']
      $P1 = box $I0
      push $P0, $P1
      inc $I0

      goto LOOP
    DONE:

    say 'Done'
.end

Running the command: “./parrot vtable_push_pmc.pir vtable_test.pir” yields the following output:

Line: -1
Line: 8
Line: 8
Line: 8
Line: 8
Line: 10
Line: 10
Line: 10
Line: 10
Line: 10
Line: 10
Done

Since I have not implemented the appropriate Instrument::Event class for Vtable events, the code in ‘vtable_push_pmc.pir’ is rather lower-level than I would have preferred. But it will do for now.

With that, this week I would like to:

Finish up InstrumentVtable
- Implement the Instrument::Event class
- Add a way to get at the vtable arguments
- Documentation + Tests
Do InstrumentClass, InstrumentObject
- Build on InstrumentVtable to add support for methods.
- Add ability to instrument a single object.
Start on user documentation.
There is currently no tutorial or anything on how to use the framework. Write something to show the simple stuff, instrumenting the ops.

parrot.org: Seven days of PIRATE and Tree::Pattern hacking

Sun, 04 Jul 2010 09:52:04 +0000

Last week, I generalized PAST::Pattern to Tree::Pattern. This week, I put that to use.

chromatic: Perl 6 Design Minutes for 30 June 2010

Sat, 03 Jul 2010 08:13:30 +0000

The Perl 6 design team met by phone on 30 June 2010. Allison, Patrick, and chromatic attended.

Allison:

working on Parrot packages for Debian experimental
seems like a good idea to do that before the 2.6 supported release
there was also a request for Rakudo packages
not sure if I'm the best person to do it

Patrick:

I'm sure we should package Rakudo Star

Allison:

Debian had a packager for those, but I haven't looked at the packages
this'd be an early run of what we'll do with Rakudo Star

Patrick:

we're not quite ready for packaging that yet
maybe a couple of weeks
finished the List and Iterator types for the #30 release
adjusted Rakudo's Associative and Positional roles
much cleaner implementation now
that'll require a few small spec changes
redid Rakudo's container types
more robust
preparing for autovivification of hashes and arrays
expect to finish those in the next couple of days
there was no container model previously; the code was consequently crufty
lots of cleanup of incorrect assumptions
Rakudo lists are now properly lazy
comment syntax fixed
ROADMAP updated
fixed the meaning of Nil; it's defined, not undefined
added the sink prefix (?)
fixed setting of $!
started fixing bugs and closing tickets on Monday, did 15 or 20
mostly already fixed in the previous couple of weeks
looking at the implementation of the series operator
spec is self-contradictory or ambiguous or both
waiting for Larry's clarification
fixed a bug in $*ARGFILES
had a nice contribution of that implementation last week
that behavior works on any set of files, not just those on the command line
working on autoviv
have some regex backtracking bugs to fix
will work on closures after that
put together three new YAPC presentations
the Rakudo Star presentation will become a video cast or a blog post or both

worked on a slew of Parrot optimizations for Rakudo
have a few more to go
might have to create a Rakudo branch temporarily
will try to help merge the new GC
working on a metamodel for Parrot objects, informed by Perl 6 and Moose

parrot.org: Green Threads and Sleeping

Thu, 01 Jul 2010 21:24:16 +0000

I've spent my Parrot time for the last week working on something I call Alarms. Alarms are like Timers, except instead of specifying a delay the user specifies when they want the Alarm to fire. This is a pretty simple concept. If the Scheduler did what I wanted, I'd be done long ago.

Jonathan Leto: Rakudo Perl 6 in Your PostgreSQL Database!

Thu, 01 Jul 2010 01:27:34 +0000

It has been a very exciting few weeks in the Perl 6 world with regard to database access. mberends++ just wrote a nice blog post about how Perl 6 support for DBI is ramping up with work on MiniDBI (formerly FakeDBI). Multiple developers recently made great progress on PostgreSQL drivers and have been improving the Parrot interface to libpq, Pg.pir .

I have also been dutifully hacking on PL/Perl6, which embeds Rakudo Perl 6 into your PostgreSQL datbase via PL/Parrot, and just recently merged the plperl6 branch, which adds the first basic support for PL/Perl 6. Here is how it currently works:

You must first install a recent version of Parrot and Rakudo Perl 6. I develop with Parrot trunk and Rakudo master because I often need very recent changes/bugfixes
You will need a moderately recent version of PostgreSQL. PL/Parrot has been known to work with versions as old as 8.3.8, but I mostly develop on the master branch, so 9.x will probably work best
When you type "make install" for Rakudo Perl 6, you will see that it installs a file called "perl6.pbc" into the lib/$version/languages/perl6 directory of the Parrot installation that Rakudo was configured with.

To tell PL/Parrot that you would also like PL/Perl6, you set the environment variable PERL6PBC to the full path of perl6.pbc like so:

export PERL6PBC=/Users/leto/git/rakudo/parrot_install/lib/2.5.0-devel/languages/perl6/perl6.pbc

A PBC file is Parrot bytecode, and the perl6.pbc file basically bundles all of Rakudo Perl 6 into a single file. You can play around with the Rakudo Perl 6 REPL by running:

$ parrot $PERL6PBC

which is pretty much the exact same thing as running the perl6 binary in your $PATH.

When you compile PL/Parrot with the $PERL6PBC environment variable set, it automatically creates a separate Parrot interpreter with the perl6.pbc bytecode loaded, so that Perl 6 code can be executed.

To run PL/Parrot tests, you type "make test". You can also do this the PostgreSQL way with "make installcheck", which will run the tests and verify that the output matches a certain, known output. "make tests" just generates the TAP output.

Currently PL/Perl 6 tests are not run by default, they have their own Makefile target: test_plperl6 This is what the output of "make test_plperl6" looks like currently, on commit 44a985f123 of the perl6_args branch.

We run a total of 9 tests, 6 of which correctly run Perl 6 code and pass. You will notice that all the failing tests are relating to passing arguments into Rakudo from PostgreSQL, which is what I am currently trying to get to work. Currently I am passing a Parrot ResizablePMCArray of the arguments to a Rakudo function and executing it, but the function can't seem to see it. My guess is that Rakudo wants native datatypes and not Parrot datatypes. If you know how to create Rakudo datatypes from PIR, please let me know :) I promise you will -Ofun.

What does PL/Perl 6 look like? Here is a nice example of a PL/Perl6 function which calculates the sum of all Fibonacci numbers <= 100:

CREATE FUNCTION test_fibonacci_plperl6(integer) RETURNS int LANGUAGE plperl6 AS $$
[+] (1, 1, *+* ... 100)
$$;

You will notice three spiffy new Perl 6 operators in there, the summation operator [+], the new range operator ... (which used to be .. in Perl 5), and the *+* operator. What exactly is the *+* operator? pmichaud++ jokingly referred to it as the "cheerleading plus", but it is actually just a plain old infix "+" operator, sandwiched by two "*" (a.k.a Whatever) operands. It basically takes the previous two elements in a list, sums them together and returns the sum, which is exactly how the Fibonacci sequence is defined.

If you are interested in hacking on PL/Perl 6, PL/Parrot or anything related, come join us in #plparrot on freenode, join the mailing list and fork us on github !

Khairul: Week 5: Unforseen troubles.

Tue, 29 Jun 2010 20:24:08 +0000

Due to unforseen personal circumstances, I did not manage to do much this week. However, I did manage to finish instrumenting the GC subsystem. So, here’s a short recap:

Updated InstrumentOp.pmc
After discussion with cotto, I changed the interface slightly, combining 4 methods into 1 as getting the context information again and again is not very efficient.
Added tests for Instrument::Probe
This is self-explanatory. Tests instrumenting the core ops, dynops, op family and ensuring that the callbacks are really called.
Instrumented GC
Creating the stub functions for each GC_Subsystem entry was rather tedious, as there was quite a few of them. As of right now, all the GC_Subsystem entries except for is_blocked_mark, is_blocked_sweep and get_gc_info has their own corresponding stubs.

Each stub does the following,
1. Call the respective GC function.
2. Gather the data to send as part of the event to be raised.
3. Raise the event.
Furthermore, in addition to inspecting each individual GC_Subsystem entry, I have also grouped the entries into the following categories: allocate, reallocate, free, administration. This grouping is based on what each function is supposed to do, using the Mark and Sweep GC as the basis. The leftover functions are grouped under administration.

What is not done yet however is the interface for these events, which I’m currently in the midst of. My current problem is getting the EventDispatcher to recognise and dispatch to catchall event handlers, such as Instrument::Event::GC::*. This is needed as for example, wanting to only instrument the PMC related entries of the GC_Subsystem. What I have currently doesn’t allow this. So a rework is in order.

This week I’ll be continuing on the following tasks:

Finish up the event interface for the GC.
Instrument the PMC vtables.

parrot.org: A report from the (NFG) front lines.

Tue, 29 Jun 2010 18:56:40 +0000

After taking some time off GSoC work to pay the parrot Upgrade Tax that plumage had accumulated in the months since 2.3 shipped, and watching the World Cup this hasn't been an awfully productive week, code-wise. However, there's been quite a bit of work and planing done anyway.

chromatic: Modern Perl: The (Draft) Book

Mon, 28 Jun 2010 23:43:33 +0000

This took longer than I expected, but the draft of the Modern Perl book is available for review. I'm especially interested in hearing from people who don't consider themselves expert Perl 5 programmers. The goal of the book is to explain how Perl 5 works (and how to write Perl 5 effectively) to help novices become adepts.

parrot.org: Generalization: Tree::Pattern, Tree::Transformer, and Tree::Walker

Sun, 27 Jun 2010 08:48:14 +0000

Earlier this summer, when I was considering what sort of interface would be best for the PAST pattern matching and transformation I've been building, Bacek suggested an XPath-like syntax. XPath is, nominally, a language for pattern matching on XML, but it is applicable to any tree-like data structure where each node may have both attributes and children. It's a general-purpose tree-matching language.

chromatic: Perl 6 Design Minutes for 16 June 2010

Sat, 26 Jun 2010 17:07:30 +0000

The Perl 6 design team met by phone on 16 June 2010. Larry, Allison, Patrick, Will, and chromatic attended.

Larry:

documented TOP (again), and explained how parsing is initiated and how it actually works
series operator (...) now picks a monotonic function when using single characters as endpoints
STD can now catch duplicates involving protos as well as onlys
STD no longer advises removal of parens on spaceless sub() declaration
mostly advised sorear and pmichaud
Stefan is finishing the boostrap of the STD parser
also working on adding a parallel NFA and DFA engine
no, he doesn't want to generate all the states in advance
it works faster lazily

Allison:

working on chroot environments with something more secure than chroot
relevant to building Parrot packages
looking at some bugs for Will

Patrick:

Rakudo developers decided not to make extra special effort to make a June release of Rakudo Star
the calendar works against us
the new date for the release is July 29
we're I comfortable with hitting that target
we won't be happy with the results of moving heaven and earth to release in June
there are lots of advantages
one disadvantage is not having Rakudo Star at YAPC::NA
one big advantage is using the supported Parrot 2.6 release as the basis
I'll write a post outlining the plan in the next couple of days
otherwise working on lists and interators in Perl 6 and Rakudo
after deciding to make iterators immutable, Larry and I realized that solves many problems
everything works out as plain as day after that
very happy with that design
the incorrect assumptions of the old model were pervasive
replacing the old pieces is taking a while, which is no surprise
this approach feels right though
the new branch does things no previous version could do
slices work much better, for example
metaoperators work properly
map is lazy
slurpy arguments in lists are lazy by default
no weird binding or action at a distance problems
plenty of changes to Associative and Positional roles
those are now super clean and may be lazy
more features work
~30 failing tests (not test files, just tests) now, ~500 last night
most of the current failures are minor
will try to merge the branch before the release
replacing lots of ugly code with fewer lines of elegant code
Jonathan and others have worked on lots of other pieces
adding plenty of new features
looking forward to tomorrow's release

editing the Rakudo book
moving the Rakudo release date may let us have a printed book available about the same time
depends on how much there is left to write

chromatic: Perl 6 Design Minutes for 09 June 2010

Thu, 24 Jun 2010 12:24:33 +0000

The Perl 6 design team met by phone on 09 June 2010. Larry, Allison, Patrick, and chromatic attended.

Larry:

not much spec change this week
figured out a syntax for a regex block to return more than one cursor
based on gather/take
in STD hacking, continued to assist Stefan O'Rear in getting STD bootstrapped via viv
now that it's bootstrapped, we're refactoring things that make sense now
we're now starting to move bits of Cursor code from Perl 5 into Perl 6
refactoring the grammar for sanity of design
started upgrading STD to normal Perl 6 syntax where it previously catered to gimme5's limitations
for example, switched STD's old .<_from> and .<_pos> hash lookups to using .from and .pos accessors
started the prep work for moving EXPR out of STD to make it generally available to any grammar wanting operator precedence
in STD parsing, made Perl 5 $< detection have a longer token to avoid confusion with match variables
STD no longer attempts two-terms detection on infix_circumfix_meta_operator
STD now parses >>R~<< correctly, or at least dwimmily
STD doesn't complain about P5isms in printf formats like "%{$count}s"
STD was parsing /m and /s with the opposite semantics
termish now localizes $*MULTINESS in its scope so that inner declarations aren't accidentally multified
STD now carps about package Foo; as a Perl 5 construct

Allison:

talked to Chris Shiflett, a PHP developer, on someone from the PHP community to sit on the Parrot board
will be in the US for a few weeks

Patrick:

working on list simplification
had a couple of breakthrough ideas on Monday
working on the implementation now
worked out inversion lists for character class matching in regexes
will make them faster, especially with long ranges of character classes
fixed a half-dozen tickets in RT
fixed Rakudo hash constructors
fixed an intermittent bug with colon-pair signatures
two possible parses exist in STD, but we removed an unneeded one in Rakudo
fixed a bug with Parrot's exit opcode
NQP and PAST needed an update not to cheat with PASM constants
I fixed that too
Vasily added multisub and multimethod support to NQP, that was a big plus
fixed the ** quantifier in regexes to understand surrounding whitespace
regex engine tried to match beyond the end of a string, so I added guards for that
will work on lists furiously before the next release
I don't think it'll take long
closures are next, hope to have those in place by the weekend

released a new version of Pod::PseudoPod::LaTeX to support the various books in progress

Andrew Whitworth: Parrot-Linear-Algebra: Change of Course

noreply@blogger.com (Whiteknight) — Thu, 24 Jun 2010 01:10:10 +0000

I've added a lot of functionality to PLA in the last few days, including a large boost to the test suite this evening. Since Kakapo doesn't have a release that targets Parrot 2.3.0, I decided to refocus my efforts on Parrot 2.6.0. However, with all the changes in Parrot since 2.3.0 came out, the upgrade path was much harder than I anticipated. Luckily I have some superstars like darbelo and NotFound on my side, otherwise I would still be fighting with things.

After talking to darbelo I've decided PLA is not going to attempt to track supported releases, at least not for normal development. It's too much of a pain in the ass, and nobody else is doing it. Sure it's best practice, but it's the kind of thing the whole ecosystem really needs to do together. If the HLL compiler is tracking Parrot HEAD because of a desperate need of new features and improved performance, then it doesn't make any sense for libraries not to be keeping up pace.

Darbelo made one particularly good point:

If we track HEAD the we'll run on 2.6 when it's released. We branch on the release day and do whatever 'release engineering' we need to do on the branch and tag a release when we're done.

That's it, in a nutshell: If we track trunk closely enough, we will always be ready with a release following Parrot's release. Spend a few days to prepare the necessary accouterments, then cut a tag and continue on with normal development. Our releases can be tied to Parrot's supported releases, but our ongoing development will target trunk as closely as possible. I don't anticipate that we will want to cut a PLA release every time Parrot does, but if we follow a 4-month schedule and follow supported releases that should be just fine. At least, fine for now.

PLA is going to track Parrot trunk and hopefully be prepared to cut a release following Parrot 2.6.0. That gives us about 4 weeks to get all our ducks in a row (and our Kakapos too!).

parrot.org: Cleaning up the Scheduler

Tue, 22 Jun 2010 21:00:38 +0000

In my work on Timers, I've concluded that the scheduler really is the right place for scheduling things (who would have guessed?). How the scheduler *should* work is described in pdd25. How the scheduler actually works is a bit different, and how I want the scheduler to work to implement green threads is a bit different from that...

Andrew Whitworth: BLAS, LAPACK, and Parrot-Linear-Algebra

noreply@blogger.com (Whiteknight) — Tue, 22 Jun 2010 21:00:00 +0000

Last night I got back to work on Parrot-Linear-Algebra, my matrix package for Parrot. I'm really starting to get happy with the functionality it provides and am looking to start moving the project to the next level of functionality and usability.

I hadn't touched it in a while, for a variety of reasons. Austing Hastings has been very busy with other projects and hasn't cut an "official" release of Kakapo that works with the last supported Parrot release, 2.3. I've applied some hotfixes that make Kakapo build and pass tests on 2.3, but that's not quite good enough. When I make a PLA release I don't want "clone the Kakapo repository from Gitorious" in the instructions, because then the instructions get out of date when Kakapo is updated to be incompatible. What I want is to say "Download Kakapo version X from this URL", which will be invariant.

Last night I added some prototype new features to the NumMatrix2D PMC, the basic numeric matrix type. I'm going to mirror the majority of those changes to the two other general-purpose types; ComplexMatrix2D and PMCMatrix2D. I added methods to do basic elementary row operations (swap two rows, add a multiple of one row to another, and multiply the contents of a row by a non-zero constant), and used those operations to put together an example program to calculate both the Row Echelon and Reduced Row Echelon forms of a matrix. Those methods, combined with block manipulation methods I added previously and the new GEMM method I added last night as well, create a basis for us to create a comprehensive library of linear algebra routines.

But that brings me to my next concern: How to implement the remainder of the fundamental algorithms a linear algebra pack is going to want to provide? Obviously I would love to wrap the LAPACK library up and call the routines it provides directly. LAPACK provides a huge number of routines for doing things like singular value decomposition, QR, LU, and other types of decomposition, solving the eigen probem, solving systems of equations in two variables, etc. In fact, LAPACK provides almost all of the functionality I would want PLA to have in the next few months.

The problem, however, is that LAPACK is not nearly as accessible from C code as I would like. In fact, there is no "standard" for C-bindings to the library, and several lame attempts are available that tend to be incompatible with each other. The reference version, and only reliably-available version, of LAPACK is written in FORTRAN. The standard CLAPACK library is just a machine-lead translation of the FORTRAN sourcecode, with a few points in the code needing to be manually tweaked after conversion. It has a few problems, including the fact that every single function parameter (even basic ints and floats) must be passed by reference. The ATLAS library, which PLA currently prefers to provide BLAS bindings, provides some LAPACK C bindings of it's own, but only a very very small subset of all LAPACK functions are provided, and the ones it does have hardly support all the operations PLA is going to want to provide.

CLAPACK, being more or less the standard could be made to work, except that it doesn't come as any sort of user-friendly package. There are no CLAPACK packages for Debian (Ubuntu) or RedHat that I have seen, and that raises the barrier to entry significantly, something that I want to avoid.

I could use the LAPACK library directly, and twist my code to match the FORTRAN calling conventions and data alignments. That's not unthinkable, though it would require some more work on the part of PLA developers than any other solution.

I could skip LAPACK entirely, and instead rely on something like GSL with proper C bindings built-in. GSL does provide several important matrix operations and decompositions, though I would need to do more research into the capabilities of that library.. What I don't want is to lose focus and have this project grow to try and become a general wrapper for all of GSL. I want PLA to stay focused on Linear Algebra only. We could maybe create sister projects to encapsulate more of the GSL functionality and use PLA under the hood to implement the basic data structures, of course.

Maybe we will get lucky, and the new NCI system will have support for calling FORTRAN routines from shared libraries. I don't think we can just anticipate this kind of functionality, at least not during the GSoC program.

A "nuclear" option, perhaps, would be to not rely on any external library for these things and instead brew up all the basics myself. I'm not against such work per se, but it would be a huge investment in time and the case cannot be made that it's the best use of my limited development time. I did put together a Gauss-Jordan elimination routine last night, it wouldn't be too too much effort to put together a generalized QR decomposition algorithm and a singular-value decomposition algorithm, followed by routines to calculate eigenvalues, eigenvectors, and matrix inverses from those things. If PLA had a larger team of active developers who wanted to participate in this kind of work it would be an easier decision to make, but if it's primarily me and a handful of occasional-contributors, this really isn't a doable task.

My plan for PLA in the near future is this: After the 2.6 release I want to push for a stable release of Kakapo, and then using that I want to cut a release of PLA. From that point forward, PLA will target the 2.6 version of Parrot at least until 2.9 and maybe later. The first release of PLA is going to provide three basic matrix types: A 2D matrix of floats, a 2D matrix of complex numbers, and a 2D matrix of PMCs. These three matrix types will have a large base of common functionality and each type will have some additional functionality too, as required by that type. Everything provided will be well-tested (including error cases, which I haven't exercised nearly enough so far) and some example programs will be provided in both PIR and NQP.

There are a few projects I am envisioning to start in the future that will rely on PLA, so I really am hoping to create a nice, stable, functional release within the next few months. I'll post more information about any other projects as they arise.

chromatic: Perl 6 Design Minutes for 02 June 2010

Tue, 22 Jun 2010 01:12:29 +0000

The Perl 6 design team met by phone on 02 June 2010. Larry, Allison, Patrick, Will, and chromatic attended.

Larry:

mostly, I supported sorear in bootstrapping STD to use viv instead of gimme5
his stage 2 and stage 3 now output identical Perl 5 versions of STD
produces a huge amount of warnings
appears to require Perl 5.12 at the moment
working on both of those
S03 refines hyper dwimminess to be more like APL, with modular semantics
S02 refines Blobs to simply be immutable Bufs, with similar generic characteristics
S02 now describes native blob types
implemented post-declaration checks for BEGIN and use, since those can't wait for end of file
STD no longer loses existing bindings when we go to a sublanguage
STD now uses $*GOAL variable only as informative, never as a "stopper"
instead, we create a <stopper> rule for $*GOAL if necessary
can check for that only, instead of that or $*GOAL
answering lots of questions on how STD and viv work besides that

Allison:

did a lot of research on graph color algorithms for register usage algorithms
will finish my finals on Monday

Will:

trying to herd the discussion of dynop libraries
a recent branch to close an old ticket broke a lot of assumptions
some bugs have become more visible because of these changes
hope to get that cleaned up this week

Allison:

I liked your suggestion of bringing back the getstderr and related opcodes

Will:

trying to resurrect Partcl
stuck on a TT #389 closing issue
not sure how to fix that, the way things are now

Patrick:

working on the iterator and list design
brainstorming the implementation
will implement somethine one way or another this week
people keep implementing workarounds for the current system
they'll bite us eventually
Moritz and I worked on making the regex engine returning real Perl 6 objects
that mostly works
exposes some places where lists don't work exactly right
the workarounds there made me replan the list and iterator implementation
answered some questions online
Jonathan added a better backtrace algorithm for Rakudo
reports Perl 6 source lines instead of PIR lines
I'll review his code
think I can borrow it for NQP for all HLLs
Jonathan reports that it was a lot easier in NQP than PIR

trying to answer a few Parrot design questions
looking at the continuation of design from Perl 1 - 4 to Perl 5 and Perl 6
hope to have coding time soon

Khairul: Week 4 Synopsis

Mon, 21 Jun 2010 16:27:17 +0000

Progress this week was not bad. To recap, here’s what I got done this week:

Handle exit/unhandled exceptions.Whenever the exit opcode is found, Parrot will throw a CONTROL_EXIT exception which will propogate up the exception handler stack to the first exception handler which can handle it. In most cases, this will be the C exception handler created before entering the runloop in the function runops (see src/call/ops.c), and trigger the interpreter cleanup routines. Exiting in this manner will not allow the instruments to be finalized. So by inserting another C exception handler after this point, such exceptions can be caught and the runloop can exit normally, allowing the instruments to be finalized. As a plus point, since C handlers act as a catchall for exceptions, any unhandled exceptions will also be caught, again, allowing the instruments to be finalized.To this end, I also modified the Parrot_runloop struct (see include/parrot/call.h) and the appropriate routines to have a reference to the thrown exception, allowing the C exception handler to know what was thrown.
Cleanup the hook tables on destroyThis was listed as todo in the source, so I did it by first creating a new function to delete a list and then calling that function for each hook list.
Update the op callback interface.Previously, on calling the callback, 3 parameters were passed. The parameters were, the current relative PC (Program Counter), a ResizablePMCArray containing the op number and its arguments, and the Instrument object itself. This wasn’t very convenient, as to get the information about the op, one has to get the OpLib instance and then obtain the OpCode instance for the op in question.So I reworked it instead, creating an InstrumentOp dynpmc which conveniently allows looking up most of the information required from it, along with the file, line and namespace the op is in (accuracy of this is subject to the core’s ability to obtain the info). Getting the file, line and namespace wasn’t particularly hard. With the initial guidance from cotto, I managed to trace it all the way to Parrot_Context_get_info (see src/sub.c), which conveniently is already marked PARROT_EXPORT and provides the information I wanted in the form of the Parrot_Context_info struct.
Additionally, in the process of fleshing out the InstrumentOp dynpmc, I also removed a todo item with regards to the special ops which have variable arguments (set_args_pc, get_results_pc, get_params_pc, set_returns_pc), which up until then, were simply ignored.
Hooks on dynops.This was listed as todo in runtime/parrot/library/Instrument/Probe.nqp previously. In the process of designing tests for the class Instrument::Probe, I got sidetracked into revisiting this todo. Now, upon detection of any dynop libraries, all probes that have pending op hooks are disabled and reenabled, and in the process attaching hooks to the relevant dynop if found. The tests for Probe.nqp and EventLibrary.nqp are still pending, as I’m looking at whether I can improve on the Instrument::Probe and Instrument::Event interface more, namely adding methods to get the current state of the objects.

With reference to the previous post, I did not manage to complete the following two tasks:

Adding tests for Probe.nqp and EventLibrary.nqp
New events (sub call, class events, exception events)

I hope to be able to complete these two tasks by Thursday (June 24th).

For this week, I would like to complete the following:

Add events for GC.This is mostly replacing the entries of interp->gc_sys with appropriate stub methods that will raise an event. Hopefully it shouldn’t take that long (Prays for no crashes, since raising the event will invoke the GC itself, methinks).
Dynamically remap dynops (Internally, nothing to do with the dynop_mapping branch).To date, I have not been able to successfully run tracer.nqp on perl6.pbc, at least as far as getting to the REPL prompt. This is mostly due to dynops (I think), since dynops used the perl6.pbc are dependent on the order that the dynop libraries are loaded during the compilation of that PBC. I have somewhat of an idea on how to approach this, but have not probed enough to know more details of it (namely in what order were the dynop libraries loaded, and how to remap the dynops to the op table). Hopefully it will be successful.
Prepare to instrument the PMC Vtables.
After discussing with cotto, it would be better if I do this first while waiting for the dynop_mapping branch to merge.

chromatic: Perl 6 Design Minutes for 26 May 2010

Sun, 20 Jun 2010 19:40:02 +0000

The Perl 6 design team met by phone on 26 May 2010. Larry, Allison, Patrick, Will, and chromatic attended.

Larry:

:() syntax is now always signature
we now use foofix:[...] as the general op form instead of foofix:(...)
refactored the sematics of :nth and :x
:nth() now only ever takes a monotonically increasing list
S03 now explains how "not-raising" works on != and ne
it now basically matches the intuitions of an English speaker via HOP definition of negate metaop
STD sometimes didn't require semi between statements
statement modifiers are expression terminators but not valid statement terminators
an unexpected statement modifier word like if could terminate one statement and start another
fixed up backslashes in character classes to allow \s etc and reject \u etc
STD was accidentally using the same lexpad for different multis
Cursor now treats :() on name extension as a signature always, never as a categorical
we shouldn't introduce the stopper for circumfix until we're in the circumfix, or we can't use the same char on both ends
placeholder messages error messages are now much more informative and correct
we now disallow use of placeholder after same variable has been used as a non-placeholder, even for an outer reference
renamed add_macro (which it doesn't) to add_categorical (which it does)
participating frequently in discussions on semantics both on irc and p6l
working closely with sorear++ as he brings viv closer to bootstrapping, yay!
soon can bootstrap past gimme5

Allison:

worked on Pynie this week in my limited spare time
one goal is to generate the parser directly from the Python grammar
wrote a small, lightweight PEG parser which generates a match tree from the Python 3 grammar
can generate a lexer directly
right now it creates a parse tree
looks similar to the match nodes of NQP-rx
dumps out a tree to the PIR parser
working on PaFo elections for next year, but trying to delegate those
will have more time after June 7

Will:

working on Perl 6 advent tests
many more people are doing more work than me
liasing with Rakudo folks for any important Parrot bugs before the Rakudo Star release
my current direction there is "don't break anything"

Patrick:

sorear added hash flattening to NQP
lots of work on closures in PAST and NQP
they properly clone
fixes some lexical problems
need to get that to work in Rakudo
that's tougher; Rakudo has to wrap Parrot subs
wrapper object needs cloning as well, along with its attributes
we'll add a new PAST node type to help
that node understands contexts
essentially a way to add void context optimizations to your AST
that solves many problems in Rakudo beyond closures
added a setting into NQP along with its test suite
not automatically loaded, but available
contains standard hash and array methods
Parrot's ops2c project uses those
other people can update and enhance that setting as necessary
NQP also has the ability to parse type names
NQP doesn't do anything with them yet
eventually they'll allow the use of multis
cleaning up some NQP bugs regarding lexicals and package storage of subs
Bruce Keeler enabled variable interpolations in regexes
working on some refactorings to simplify that approach
works in NQP and Rakudo now
that's a feature we've never had before
Rakudo's REPL now works better, thanks to sorear
HLLCompiler now written more in NQP as part of that
NQP now can do eval
NQP remembers lexicals in interactive mode now
adding that to Rakudo is more complex
working on that
pleased with the progress on #perl6

reviewing long term plans for GC and Lorito
should have more time free soon

Patrick Michaud: Rakudo Star (a "usable Perl 6") to be released by July 29

Sat, 19 Jun 2010 17:04:21 +0000

As many of you know, last summer we announced that we would be releasing a "usable release of Rakudo Perl 6" to be called "Rakudo Star" in the second quarter of 2010. We later refined our target release date to be April 2010.

Until March of this year we were well on track to meet the April 2010 release date, but then I had an family medical emergency that took me away from Perl 6 development. As a result of my situation, the Rakudo and Perl 6 team met online in early March and decided that an April release date would be unrealistic, and we instead focused our efforts on trying to make a June release for Rakudo Star, to keep with our original "second quarter 2010" goal.

Ultimately it ended up being twelve weeks before I was able to return to active Perl 6 development (i.e., late May). During my absence the others on the Rakudo and Perl 6 team made incredible progress on Rakudo Perl 6; I think their progress shows that a truly capable (and growing) team of developers has coalesced around Rakudo Perl. Thanks to their efforts, as of late May the compiler had nearly everything we identified as critical for Rakudo Star in the ROADMAP, with only a few key features blocking on my personal participation. We therefore felt we still had a good likelihood of meeting the June 2010 target, and continued to work with that goal in mind.

As part of planning this week's Parrot and Rakudo releases, we all met online to solidify our plans for the Rakudo Star release. After much discussion, we decided that although we could likely make some sort of Rakudo Star release in June, there was too much risk that releasing in June would fall well short of our vision of what we want Rakudo Star to be.

Therefore, we've decided to to let the release date slip one more month and release Rakudo Star not later than July 29, 2010. We are firmly committed to the July 29 date; whatever we have available then, that's what we release. I know that another delay will be frustrating to many (it is very frustrating to me), and that some will undoubtedly cite this delay as yet more "evidence" that there will never be a release of Perl 6. But given the circumstances, I think we feel that we promote Perl 6 better by moving the release date by a month than we would by releasing something less than our vision.

For those who might claim that we should "release early", we are still continuing to make regular monthly compiler releases. The most recent release (#30, "Kiev") comes with a lot of improvements over previous releases, and I truly expect the next release (#31, "Atlanta") to continue the trend. As always, we continue to invite people to try out the compiler releases and to visit the Perl 6 website to see what Perl 6 is already doing today.

Finally, on a personal note, my wife and I sincerely appreciate the ongoing support, prayers, and understanding we have received from the Perl community (and especially the Rakudo and Perl 6 teams) during these difficult times. While my wife is still not "out of the woods" yet, things are far better now than they were in the Spring, and we continue to work towards and pray for her full recovery.

More details about the Rakudo Star release will be forthcoming over the next couple of weeks.

Andrew Whitworth: Parrot's Deprecation Policy

noreply@blogger.com (Whiteknight) — Sat, 19 Jun 2010 13:24:00 +0000

Parrot user kthakore sent a very interesting email to the Parrot developers list this week to criticize the current deprecation policy. After taking some time off from development, he returned to find that his extension no longer built on Parrot HEAD and he couldn't quite figure out why. There was a deprecation notice for the feature that changed, but the notice was so vaguely worded and so short on explanation that it offered very little help to the beleaguered extension developer.

When we're talking about code, there are only a handful of things that we really need to worry about: utility, performance, security, stability and maintainability. Well-written code, for the most part, can satisfy all these requirements. Sometimes trade-offs are made, such as trading a certain amount of maintainability to optimize for performance, but in those cases some well-placed comments can help alleviate or minimize any regressions. Code, while technical and deep, is often pretty easy: we follow rules and policies, make changes, measure results, wash, rinse, repeat.

Not so easy are the softer sides of open source software: the people. People come in varieties: core developers, extension developers, testers, documenters, end-users and well-wishers of varying levels of technical competency. Keeping all these groups of people working together nicely and happily can be quite a difficult challenge, and there is likely no way for all groups to be 100% happy 100% of the time. The tradeoffs here are harder to understand, harder to manage, and a misstep can have huge negative consequences for the project.

I've long been a detractor of Parrot's current deprecation policy: It's too rigid, too narrow, and doesn't really do anything to help the people who need helping. It also doesn't really take into account Parrot's current stage in the development life cycle. Parrot, as I've mentioned on occasion, has plenty of warts and has suffered many growing pains. It is folly to think that we should be making blanket guarantees about what will or will not be present in various releases, or how quickly we can or cannot make changes.

When there's a problem or a bug or a misfeature, people want those things fixed quickly. A good example of this were the PCC refactors a few months ago. Even though those refactors created some backwards-incompatible behavior our users (who the deprecation policy was at least nominally designed to protect) were trying to rush them through. Rakudo developers specifically were blocking on the PCC improvements and having to wait for months and months would have been bad for them. The PCC refactors were high priority and high need.

The deprecation of several core ops and their conversion to dynops recently is an example from the other end of the spectrum (and the source of kthatkore's frustration). While we followed the letter of the deprecation policy, these things didn't need to be removed with haste and created a bit of hassle for users who weren't expecting it. I'm not saying they shouldn't have been removed (I'm always a proponent of removing cruft), but it does expose some short-comings of our deprecation policy and process.

What we have is a series of conflicting motivations, even for individual groups. Consider:

Core Developers: Core developers want to remove bad features, want not to maintain bad features, want to add new good features and want to create the best software for the users. Developers need to work on fun new features, but also need their work to be used and appreciated. Take away either of those pieces, and many volunteer developers will simply walk away. Moving too quickly alienates the users and creates a huge disconnect between what the developers are developing and what the users are actually using. Moving too slowly is boring and developers start to leave for greener pastures.
Extension Developers: Want to add new extensions to the Parrot ecosystem for the benefit of users, but have to deal with binary compatibility at the C level which changes much more frequently than the "normal" user-visible interfaces like PIR. Parrot has a lousy extension API right now, so necessary improvements there require extension developers to stay up-to-date with current core development. At the same time, all sorts of other changes break things in new versions, even when necessary features are fixed.
Users: For stability, it's good for users not to upgrade. To fix bugs and get new features, it's good t upgrade. Upgrading brings rewards but also hassles: Core features disappear, new bugs are added, and extensions are broken by binary incompatibilities. Upgrading Parrot means needing to upgrade (or, at least, rebuild) extensions too.

It's difficult to tell an extension developer to stick to the supported releases because the extending API is so lousy and incomplete. Having to wait 3 months or more for a necessary change is hard for these small projects, and their developers can quickly lose interest and motivation. Until we reach some basic level of usability, we have to expect that extension developers are going to be tracking Parrot HEAD more or less closely. I think it's a little disingenuous to simultaneously expect fully that developers will be tracking the repository HEAD but also write in our deprecation policy that they should only track the stable releases, and you really need to ask who exactly that policy is designed to protect in this case.

We really need to account for different levels of user and different levels of feature. End-users shouldn't be using Parrot directly. Joe Schmoe at home is not and definitely should not be writing his tools in PIR. If he's writing his code in a higher-level language like Rakudo Perl 6, NQP, or Winxed, he's buffered from disruptive changes made to the Parrot core VM. It's the developers of HLL compilers and extensions that need to worry about these kinds of changes.

Likewise, we need to differentiate between issues of multiple severities.When a big issue is blocking development in extensions and HLL compilers, it behooves Parrot to ignore the mandatory wait period and to make those fixes post haste. Alternatively, a change which is not necessary and would cause a block or a slow-down for extension developers should be put off for longer and made with more care.

What we need, in a nutshell, is a policy that actually does what we claim the current deprecation policy does now: Protect our users from disruptive changes, but also enable forward progress to be made without being forever tied to every bad decision ever made. I suggest these changes to policy and process:

Deprecation should be added before a disruptive change is made
The deadline for the deprecation shouldn't be blindly tied to the next stable release, but intelligently selected with input from the affected parties. We do have weekly planning meetings where these kinds of things can be decided. If we need to regularly schedule additional meetings with other parties (HLL compiler devs, extension devs, etc) we should do that as well.
Deprecations should be well-documented and publicized. Information about the deprecation should include what exactly is changing, how users of those features can work around the changes, and who to contact when/if problems arise.
Information about the deprecation should be sent to the users, not just dumped in DEPRECATED.pod where we expect people to be looking regularly. An email list for this purpose was suggested and I like that idea (other ideas were also suggested that I also like). Any way to send the information directly to the user is a good thing.
Where possible, both old and new versions should be provided simultaneously for a period of time while users transition. This is most important in the C-level API where function wrappers can easily be provided to translate old calls into new ones.

I'm still a big proponent of the idea that the deprecation policy should be opt-in, in the sense that only features that we've put a stamp of approval onto should be covered under the deprecation policy and anything else should not be relied upon. You can use a feature that we haven't approved, but then you're responsible for paying the price when that feature changes or disappears completely. You would also be responsible for sending the core developers feedback about which features you would like to see be added and approved.

Having a deprecation policy is a good thing and a necessary part of a mature software project. However, the policy we have currently fails on a number of counts and requires some serious re-thinking if we want to make it better. I sincerely hope, and I know several other people also hope, that we do make it much better in the future.

Andrew Whitworth: ParrotTheory: Locks and Synchonization

noreply@blogger.com (Whiteknight) — Fri, 18 Jun 2010 13:54:28 +0000

I've talked a good amount in the past and recently about threading, so today I'm going to talk about some of the small bits that make threading and concurrency work. I'm thinking that we're also going to need to implement some of these primitives in Parrot eventually, so consider this blog post an extremely verbose way of planning for that.

Let's start the discussion by looking at two small code snippets; a structure definition and a routine that uses that structure:

typedef struct _array {
  int length;
  int *values;
} array;

void push( array *ary, int x) { 
  array->length++; 
  realloc(array->values, newlen * sizeof(int));
  array->values[array->length - 1] = x;
}

This isn't nearly as contrived an example as it may seem initially, though I purposefully made the code a little bit naive. It's worth noting here that the idea of a structure which contains both an array and the integer length of that array is very common. It's how Parrot implements it's STRING type and several of its array PMC types as well. In fact, the push_int VTABLE method for the ResizableIntegerArray PMC probably looks extremely similar to this example code.

Astute observers, and veterans of threading trench warfare will see a problem with this code: There are no locks and no concurrency safeguards, so this code isn't thread safe. Let's take a short walkthrough of this code where we have a preemptive thread switch in the middle to another thread also attempting to access this same method on the same object:

We have an array object with length 5 and items {1, 2, 3, 4, 5}
Thread A attempts to push the number 6 to the array, Thread B attempts to push the number 7.
Thread A enters the function. We set length to 6 and realloc to gain a sixth slot. The array now contains values {1, 2, 3, 4, 5, ?}, because the last value isn't initialized by realloc.
At this point, there is a preemptive thread switch and Thread B takes over control.
Thread B enters the function and sets length to 7 and reallocs again. The array now contains values {1, 2, 3, 4, 5, ?, ?}.
Thread B sets the sixth element to 7. The array is now {1, 2, 3, 4, 5, ?, 7}
Thread B gets preempted, Thread A takes over.
In thread A, the value of length is still 7, so [ary->length - 1] is still 6. When we add x to the array, we now have {1, 2, 3, 4, 5, ?, 6}

There are some things we could try here, such as saving the values we need from the structure to local variables, so that even if we preempt in the middle of the functions the length field will be an accurate index. Or, we can try to rearrange the order of operations so some errors appear less frequently or less obviously. The real problem here is that we have three operations that need to stay coupled: Increasing the known length of the array, reallocating array storage, and assigning an item to that storage. Any break between a set of coupled operations causes a problem. We call these areas of code where operations are sensitive to cross-thread interference critical sections.

As another example of a very small critical section, consider this one line of code:

foo = i++;

This seems pretty simple, but in fact it is not. This line of code requires several assembly language instructions, especially when we are talking about a non-optimized build:

Fetch the value of i from memory into a processor register
Copy the value of i to the variable foo
Increment the register containing the value for i
Copy the value of the register back to memory where i is located.

A preemptive thread switch can happen in between any of these steps. Consider the case where we break between steps 2 and 3 to another thread performing the same operation: If i is 5, Thread 1 foo is 5, Thread 2 Foo is 5, and at the end of the code snippet i is 7! If this code snippet is, for instance, in an SQL database generating unique integer keys for rows in a particular table, we've just generated non-unique keys and created a world of hurt for ourselves.

To get around these kinds of problems, one solution is to use a synchronization primitive. A synchronization primitive is any of a class of algorithms and objects that are designed to synchronize and limit access to shared resources. In this sense, a resource is anything that multiple threads might want to access: An IO stream, a global variable, a shared pointer, even a sensitive sequence of instructions, etc. Any time we have a critical section of code that is sensitive to sharing we want to find a way to limit access to a finite number of simultaneous threads (usually one). There are several ways to do this.

A mutex, short for "mutual exclusion", object is a type of lock that helps to prevent access to a critical section. To pick a pertinent example, think of a mutex like a basketball. Only the one person in the game with the basketball can do things: shoot, pass and dribble. Other players on the team can do other stuff like running, covering, or posting, but they cannot do ball stuff without the ball. You cannot shoot the ball if you do not have the ball, you cannot pass the ball if you do not have the ball. This is a convention of the sport. If we were playing Calvinball instead, maybe we could do these things without looking preposterous. By convention also, if we as programmers declare that a certain shared resource can only be accessed by a thread (player) with the mutex (ball), the those are the rules for our system (game) and things can move along smoothly. Here's an example of that convention in action:

Mutex *m;
AQUIRE_MUTEX(m);
// critical section code
RELEASE_MUTEX(m);

The power in this code is that the AQUIRE_MUTEX() function will attempt to gain ownership of the mutex, and will wait indefinitely for the mutex to become available if some other thread already owns it. ACQUIRE_MUTEX is like waving your arms in the air, shouting "I'm open" until the current ball carrier passes the ball to you. Until you get the ball, you just have to stand with your arms in the air until you get it. Because of that behavior, no two threads can enter the same critical section, assuming of course that the programmer (you) has properly protected that critical section with the mutex. Keep in mind that there is no intrinsic property of the critical section itself that prevents multiple threads from running it simultaneously and corrupting data. The exclusion comes from the proper and pervasive use of locks like our mutex to keep the critical section safe. Here's another example:

Mutex m;

int pop(array* a) {
  ACQUIRE_MUTEX(m);
  int item = a->values[a->length - 1];
  a->length--;
  RELEASE_MUTEX(m);
  return item;
}

void push(array* a, int item) {
  a->values[a-length] = item; 
  a->length++;
}

In this example we can see that we aren't properly using the mutex everywhere, so we can't guarantee that we won't get corrupt data. Multiple threads could just as easily enter the push function simultaneously as could attempt to enter the pop function. If you don't use mutexes everywhere, it's almost as good as not using them anywhere. This is a convention that the coder must decide upon beforehand and follow diligently.

There are multiple ways to implement locks and mutexes. One idea is a spinlock, which attempts to access a flag and enters an endless while-loop until it can. An empty while-loop can be very inefficient on a processor, but if we call a sleep command inside the loop to allow other threads to run while we wait it isn't such a big problem. Spinlocks implemented by the OS inside the kernel event loop can be very efficient indeed. In fact, as a general rule, if the OS implements locking primitives they tend to be much better to use than anything you can write in userspace.

Another type of lock primitive is a semaphore, though it is subtly different. A semaphore allows a finite number of threads to access a finite number of shared resources at a time. Where a normal mutex, like a spinlock, allows only one thread to enter at a time the semaphore may allow one or more. Consider a case where we have five worker threads in a web server, and 100 incoming connections. A semaphore uses a first-come-first-served method to assign incoming connections to available threads. Each incoming connection attempts to access the semaphore. As requests are completed, threads signal their availability and the semaphore assigns the next connection in the list to that thread. A semaphore with only one shared object acts like a normal mutex or spinlock.

The overhead of a lock is the amount of effort it takes to acquire and manage the lock. In a uniprocessor system the lock may be very simple to obtain: First disable interrupts so we cannot be preempted by another thread, check the status of the lock, obtain the lock if it's available, and re-enable interrupts. In a multiprocessor system, especially one with shared memory, the overhead and error-checking involved can be much higher. In these systems the performance gain from using threads can be much higher too, so it's a trade-off.

Granularity is the amount of stuff in your critical section protected by a lock. Course Granularity means that we have lots of code inside our critical section. This is good because we need fewer locks and therefore experience lower overhead. Plus, it's easier as a programmer to make sure we acquire fewer locks over large swaths of our program. The downside is that the larger our protected critical section is, the more likely other threads are going to be blocked waiting to enter it. This, in turn, can create problems like high latency. Fine Granularity is the opposite, where we lock as little code as possible. The upside is that we don't have to worry about multiple threads blocking for long on small bits of code. The downside is that acquiring more locks means more lock overhead, and more programmer effort to implement all the locks consistently and safely. Fine granularity can also lead to deadlock, where multiple threads are stuck waiting for locks that other threads own.

The Python interpreter, as an example, implements a single Global Interpreter Lock, which is a lock to govern the entire Python interpreter. Only one operating system thread can be running the interpreter at once, to prevent corruption of global data. I think new versions of Ruby do this too.

There are other methods of synchronizing access to shared resources. One method is to make all data immutable; If you can't modify data, you can't corrupt it. Since Parrot's strings are immutable, you shouldn't ever need a lock when working with them. You may still need to worry about playing with a mutable container PMC which holds strings, or the mutable registers which point to strings, however.

Parrot is definitely going to want to make use of OS-supplied locks in some fashion. Maybe we want to make a PMC wrapper around system lock primitives, or we want to create some kind of lock manager that uses a single system mutex to distribute a series of immutable tokens to worker threads. The exact details of locking are certainly up for debate, but the fact that we don't want to brew our own should be obvious.

Since locks need to be used consistently for them to be of use at all strongly hints at the fact that Parrot should probably do the locking internally. We probably don't want to apply locks to every single operation, since the common case programs are single-threaded applications and we don't want to apply the performance penalty of lock overhead to programs which don't need it. If Parrot can identify only those PMCs which are shared, it can apply locks selectively to those PMCs only, limiting overhead to only the places where it is necessary. For instance, if we add a synchronize op:

$P0 = syncronize $P1

We can create some kind of wrapper PMC type whose vtables enter a lock, call the vtable of the synchronized PMC, and then release the lock. In this example, if everybody used $P0 when they wanted to modify $P1, all operations would be safe. The onus would be on the programmer to explicitly mark the PMC as synchronized, of course, and many programmers will probably forget to do that.

Maybe instead of passing PMC references between threads directly we create and pass clones and modify them separately on different threads. Then, when we want our changes from one thread appear in another thread, we would call some kind of propagate op:

propagate thread, obj

This would pause the specified thread, update the object contents, and then unpause the thread. This would be very similar to the message passing that languages like Erlang use (not exactly the same, because Erlang wouldn't pause the recipient for this, but you get the idea).

Maybe we have a system where we only share read-only copies. So thread A would own the PMC, but thread B could get a read-only copy of it. This would completely obviate the need to lock the PMC since only one thread could write to it, but then we need some kind of mechanism where thread B could make modifications back if necessary, or maybe B could gain the writable copy and make A's copy read-only. This system could get very complicated very quickly, however.

We could also avoid most locks if we used a transactional memory system to avoid memory corruption, but that could still add overhead to the single-threaded common case and then we would still want a system of locks for other operations that don't require locking a PMC.

These are only a handful of the many potential options that Parrot has ahead of it, and I can go into greater detail about any of them that people are interested in thinking about. I think Parrot is going to want at least some kind of locking mechanism, so we could start prototyping those things immediately if we wanted. How these mechanisms get implemented and applied within Parrot is obviously the bigger issue that we can ignore for now.

parrot.org: A first optimization with PAST::Pattern, in detail

Fri, 18 Jun 2010 07:10:40 +0000

In my previous post, I described in tutorial form how to implement a very simple constant folding optimization for Integer addition. In this very brief post, I describe in greater detail how that optimization works.

chromatic: Perl 6 Design Minutes for 19 May 2010

Wed, 16 Jun 2010 21:38:09 +0000

The Perl 6 design team met by phone on 19 May 2010. Larry, Will, and chromatic attended. Patrick added his notes later.

Larry:

S03 makes more explicit that doctrine that ~~ topicalizes, and removes smartmatch table fossils that automatically fall out from that
S05 renames 'accent' to 'mark' for better Unicode conformance
:a and :aa changed to :m and :mm
S05 disrequires retroactive semantics on :samecase and :samemark
the method form must now explicitly add case or mark modifiers to the pattern
regularized mm// to ms// to avoid confusion with new :m ignoremark option
STD now does a bit better at diagnosing bogus ??!! constructs of various sorts
STD now correctly adds operators to symbol tables as subs
CORE.setting now has protos of all the operators so they can be recognized as subs too
Cursor now canonicalize operator names in the symbol table
btw, not quite like specced
STD now reads user's mind on 'Str $toto' to intuit missing declarator
STD now properly diagnoses a typename between routine declarator and sub name

Will:

working on code for Carl Masak, trying to get his poker code example running on Rakudo
both fun and frustrating
some stuff doesn't quite work yet
going through the Advent examples
adding them to spectests
make sure we won't regress on such public examples
other people are helping with that now

will get back to editing the Rakudo book soon
hope to have it in print by YAPC, but no guarantee

Patrick:

fixed closures in NQP, as a precursor for fixing them in Rakudo
worked with sorear on REPL in Rakudo and PCT in general
ported the NQP "standard library" done by japhb++, bacek++, and many others into the nqp-rx repository and made it part of the standard build sequence for nqp and Parrot
decided we need a new "context sensitive" node type in PAST, will be used to create proper closures and to handle sink context
worked with bacek on adding better multimethod support to PAST and nqp-rx
discovered a problem with lexical subs in NQP being automatically entered into the package namespace (and some existing code relying on this behavior)
did some initial fixes to at least get things entered properly, but a complete fix may require a deprecation cycle
plan to review others' patches this week
plan to fix REPL, closures, and sink context in Rakudo (since those are currently large pain points)
plan to work on loops and iterators after that

chromatic: Perl 6 Design Minutes for 12 May 2010

Wed, 16 Jun 2010 01:47:02 +0000

The Perl 6 design team met by phone on 12 May 2010. Larry, Allison, Patrick, and Will attended.

Larry:

clarified usage of brackets around infixes
added various 128-bit types to the spec; we might make them arbitrarily extensible via role
at least LLVM could support this, even to non-powers-of-two sizes
modernized the paleolithic grammatical category description in S02
STD now uses double-quote rules for interpolating @foo[] into regex
STD now gives better message on 1__3
added the specced 128-bit types to CORE.setting
added minmax function to CORE.setting
implemented circumfix:«X Y» as grammar derivation
currently only allows a >> inside
now also recognizes foofix:("\x[face]") and foofix:("\c[YOUR CHARACTER HERE]") without actually evaluating
playing with factoring yaml out of gimme5, since viv is not likely to go that route.
mostly just answered a lot of questions on irc
egged people on about concurrency issues

Patrick:

thought on handling closures properly
have a solution, just need some time to implement
discussion on changes to CodeString
work on compiler toolkit to avoid CodeString, using StringBuilder instead where possible, in PCT, NQP, and rakudo. Pretty easy, no downstream projects block on a deprecation issue
after that, lists
also been answering questions on interactive mode (REPL) for rakudo et al. (the issue with losing lexicals)

Allison:

resolved the git conversation pretty well (for Parrot's repo migration)
worked on a pure PEG parser (following the paper), straight PIR, single day; now self-parsing. Interesting project, is lightweight. currently has memoization, but that might not be right for us because of backtracking. With some more effort, could probably handle EBNF form (useful for python)
could be setup for developer status for Debian which will improve our packaging status for Debian and Ubuntu

Will:

Parrot CodeString performance improvements
we're definitely faster in branch, but some feedback from pmichaud should help us clean up the API a bit as well, look for those to hit trunk in the next few days
Parrot makefile deps cleanup

parrot.org: A New Design for Timers

Tue, 15 Jun 2010 21:30:39 +0000

The next step in green threads is to make them preemptive: after one thread has run for a while, it needs to be stopped so a waiting thread can have a turn. Parrot has a mechanism for doing things after a set time called Timers which would be perfect for this. Unfortunately, they don't really work.

parrot.org: Progress, refactorings and tests.

Tue, 15 Jun 2010 17:11:28 +0000

Since my last post I haven't spent as much time as I'd like coding, since getting a win32 development environment setup was a bigger time sink than I expected, but I've still managed to stay pretty close to the schedule. The main feature for the next week, once I'm done with the charset-level stuff will be Iterators, which will finally enable NFG 'literals' in PIR source, for which I have added, failing, tests the past week.

parrot.org: GSOC NCI Updates

Tue, 15 Jun 2010 15:15:13 +0000

The NCI updates using the libffi are coming along. I ran into a bit of an annoyance when I found out that there are places internally where signatures are not specified for some functions, but luckily it was easy to remedy once I had figured out the problem. All of those cases used the same signature ("vJP"), it was only hard to find the right place to add that.

Jonathan Worthington: Perl Mova + YAPC::Russia

Tue, 15 Jun 2010 15:11:09 +0000

On Friday I flew from rainy Sweden to scorching hot Kiev to attend a combined Perl Mova + YAPC::Russia. The passport queue wasn't overly long, and I'd happily managed to be hand luggage only, so I wasn't too long in the airport. I planned to take the "official" bus, but before it appeared I heard a lady shouting about a mashrutka going to the center, so I took that instead. Understanding basic Russian - well, it coulda been Ukrainian too - sure comes in helpful. I was dropped by a metro station, leaving the 15 euro-cent (yes, really) metro ride a few stops to where I was staying - a hotel right on the main Independence Square. I was a little bemused as I checked in to be spoken to in a mixture of Russian and Ukrainian, by the same person in the same conversation.

My plane had been slightly late, so I was also slightly late joining the pre-workshop dinner. After I'd filled up on a tasty dish of borsch, we wandered on to a pub that offered no less than 22 different beers. I was...happy. :-) Some beer later, it was time for a little stroll, which terminated in a Japanese restaurant, where I had my first encounter with wasabi vodka. All in all, a very fun start to my visit here.

I was worried I'd sleep badly in my room since it was a little hot, but I actually slept really quite well and was nicely refreshed for the hack meet. There were a lot of people there hacking on all kinds of different projects. Some folks were interested in contributing to Rakudo, and so we gathered around on a table and I helped each of them find and get started on a task. It went extremely well - everyone in attendance contributed Rakudo patches or code that allowed us to close an RT ticket right there on the day, or that will after I apply patches. Of note, people who were new to Rakudo hacking:

Located and unfudged tests for \e that was recently fixed, allowing the ticket about that to be closed.
Implemented .all, .any, .one and .none method versions of junction constructors and added tests for them, allowing that ticket to be closed.
Debugged and then wrote a patch that fixed a bug in range iteration, plus added tests, allowing an RT ticket to be closed.
Implemented .cando and did some related refactors in the area that I suggested and also located some tests to unfudge. Due to network issues that one didn't get applied on the day, though I have it and have just applied it locally, so it'll be in soon.
Tracked down what was wrong with forming colon pairs from variables involving twigils, patched it and made sure we had some test coverage of that; again, we closed a ticket.
Wrote a patch to fix a bug in the series operator that had been reported in RT, along with some test cases for it.

I think it goes without saying that this is really quite impressive, especially given they had to share one Rakudo core hacker between them for guidance. In fact, I even had time to cook up a few patches myself amongst guiding others! It was great fun to hack alongside such pepole. I handed three spectest repo commit bits out that day, and I think they were all used. It's experiences like the one at this hack meet that make me really glad that we're writing most of Rakudo in Perl 6 or a restricted dialect of it - all of the above tasks required (apart from a one line change in one of them) no PIR or C knowledge at all. Some of those at the hackmeet said they may find time for a bit more Rakudo hacking in the future, which would be really great. :-) After the hack meet, there was more nice food - including a Chicken Kiev - and some lovely beer at prices I've become rather unacustomed to in Sweden.

Day two was talks day. I had one 40 minute talk on Perl 6 signatures which went well. I received some great questions, and I hope I answered them all to people's satisfaction. Later on, I gave a lightning talk about Rakudo * and what it would contain, and showed off a simple little example site that used Rakudo Perl 6, including the modules JSON::Tiny by moritz++ and FakeDBI by mberends++ along with Blizkost (sorear++ for that) to get at the Perl 5 CGI module (though I'll refactor it to use Web.pm in the near future - I just wanted an excuse to show off Blizkost). The evening was, of course, more food and beer, and a lot of fun.

The final day of the workshop was a bit different - a river cruise! It was very relaxing, and gave me yet another way to enjoy this beautiful city. Certainly good for unwinding after a workshop. Most people left either after that or were flying home today; I on the other hand used "no direct flight on a Tuesday" as an excuse to get another day to potter around Kiev, and today I've enjoyed doing exactly that, gently strolling between my favorite sights and stopping in the odd open air cafe in a park to keep myself refreshed in the somewhat stuffy weather. Soon it'll be time to take my last dinner here, an evening stroll and maybe find some outdoor place to sit and nurse a final beer before it's time to get some sleep before my flight home tomorrow.

Beautiful Kiev. It gets harder to leave each time I come. :-)

parrot.org: Parrot 2.5.0 "Cheops" Released!

Tue, 15 Jun 2010 13:12:21 +0000

On behalf of the Parrot team, I'm proud to announce Parrot 2.5.0
"Cheops". Parrot
is a virtual machine aimed at running all dynamic languages.

Parrot 2.5.0 is available on Parrot's FTP
site, or follow the download
instructions. For those who would like to develop on Parrot, or help
develop Parrot itself, we recommend using
Subversion on our
source code repository to get the latest and best Parrot code.

Parrot 2.5.0 News:

Core

Added ByteBuffer PMC to allow direct byte manipulation
Modified some PMC vtable functions to reduce complexity, simplifying coverage.
Modified PAST to generate symbolic PASM constants in PIR output.
General STRING API cleanups
Increased test coverage of core PMCs
Fixed up 'exit' opcode, added CONTROL_EXIT exception type.
Experimental 'unroll' opcode renamed to 'finalize'

NQP-rx
- Added proper support for multisubs and multimethods
- Fixed sigspace handling ** quantifier in regexes
- Added \e strings
- Added use of inversion lists for charclass lists in regexes
Platforms

EPEL (Extra Packages for Enterprise Linux) packages for RHEL6.beta are available

Begin moving towards Lorito, the ops refactor to enable pervasive self-hosting and JIT compilation.
- All ops are now built with the self-hosted opsc compiler.
- For more Information about Lorito see:

Khairul: Week 2 + Week 3 Synopsis: Dynlib Digressions

Mon, 14 Jun 2010 18:43:13 +0000

The process of loading a dynop library is seemingly straightforward. Simply using the “.loadlib” directive or the “loadlib” opcode in PIR will result in the library being loaded and integrated into the interpreter environment. That is, until you have more than 1 interpreter in play, or, the bytecode that you load has different opcode numberings that what is currently reflected in the interpreter.

Parrot’s solution to the former is to simply disallow loading dynop libraries when there is more than one interpreter in play. There exists a certain assumption on how interpreters are created, in that, there is only one main interpreter, and additional interpreters are spawned as ParrotThreads (at least that is how I understand the code). Creating interpreters in the model above will yield the expected behaviour, in that everything works normally when the main interpreter will preload everything that is needed by the threads, thus obviating the need to load of dynop libraries in the threads. However, if one creates an additional interpreter outside this model, it breaks the assumption and one can load dynops in the second interpreter. The main reason for this is that only the first interpreter is registered in the “interpreter_array”, while the second is not. Since the code to detect if there is more than 1 interpreter depends on the “interpreter_array” and the associated “n_interpreters” count, the second interpreter is non-existent in the point of view of the core.

With regards to dynops, in the process of updating the core op tables, there is a high possibility that pointers to the various op tables change to point to new locations in memory. With 1 interpreter, everything is fine, as the op table references of that interpreter is updated accordingly. Adding a second interpreter to the mix will result in a segmentation fault when the second interpreter tries to access the outdated references to the op tables. There has to be a way to detect or notify any changes to the op tables to all interpreters currently existing.

Trying to change the code in the core to achieve this is an exercise of frustration. After trying out a few things, namely forcibly registering all interpreters by modifying Parrot_cx_init_scheduler and broadcasting messages in dynop_register by using Parrot_cx_broadcast_message and trying to make sense of the failures I’m getting (mostly assertion failures), I discovered a few things. First, I discovered that there are no GC runs after a thread is created (if I understand correctly, see src/pmc/threadinterpreter.pmc). Apparently its not really stop the world, more like stop the GC and I don’t see anywhere where the GC is reenabled. Second, there is no simple way to halt a thread and know that it is halted. I can broadcast messages to all interpreters for all I want, but unless I know for sure that all interpreters have stop, it is dangerous to go ahead with updating the op tables. One way I wanted to try out was to use pt_thread_wait and pt_thread_signal, but that got blocked as I can’t get those symbols exported (putting them in thread.h doesn’t help, marking them as PARROT_EXPORT also doesn’t help). Since that didn’t work, I’m stopping this experiment as it has taken way too much time and its not part of my objectives.

All that effort is not wasted though. Now that I understand more of how the internals work, I could do my own DIY dynlib detection. In the same vein, I found the reason for a segmentation fault that I encountered when I try to run “tracer.nqp” against “examples/pir/io.pir”. I worked around that by simply making the child interpreter’s vtables point to the parent’s instead. Digging into the internals these past week, I think I figured out why. So apparently, NQP loads the OS dynpmc. “examples/pir/io.pir” also loads in the OS dynpmc. So at this point, there are two different entries in two different vtables with different base_type numbers. For normal dynpmcs, this is fine as each interpreter will only look at their own vtable entries. However, the OS dynpmc is a singleton. Thus, when NQP loads it, an instance is created and stored. When io.pir loads it again, the stored instance is then saved in the MRO (method resolution order) list of io.pir’s copy of the OS vtable entry. So, io.pir’s OS vtable entry’s index is around 85, but the one in its MRO list is about 101, and this difference leads to a segfault when trying to invoke a method of OS.

Back on track, I do my dynlib detection in the Instrument dynpmc itself, in the form of the function detect_loadlib. Since I discovered that each interpreter stores a hash of the dynlibs it has loaded in the iglobals hash, detecting newly loaded dynlibs is simply comparing an old list of loaded dynlibs with the current list. So, now the Instruments dynpmc can detect all dynlibs loaded through “.loadlib” directives and “loadlib” opcodes, and creates the task/event “Instrument::Event::Internal::loadlib” which can then be handled in PIR. With that in place, I can now proceed to go on and implement more events to raise.

Currently, I have three events, which are Instrument::Event::Internal::loadlib, Instrument::Event::Class::instantiate, Instrument::Event::Class::callmethod. An example script to use these events is shown below (in NQP):

Q:PIR {
	load_bytecode 'Instrument/InstrumentLib.pbc'
};

sub loadlib_cb ($task) {
	my $data := Q:PIR {
		find_lex $P0, '$task'
		%r = getattribute $P0, 'data'
	};

	say('Library loaded: ' ~ $data[0]);
};

sub class_cb ($task) {
	my $data := Q:PIR {
		find_lex $P0, '$task'
		%r = getattribute $P0, 'data'
	};

	say('Class instantiated: ' ~ $data[0]);
};

my $args := pir::getinterp__p()[2];
$args.shift();

my $loadlib_evt := Instrument::Event::Internal::loadlib.new();
$loadlib_evt.set_callback(loadlib_cb);

my $class_evt := Instrument::Event::Class::instantiate.new();
$class_evt.set_callback(class_cb);

my $instr := Q:PIR { %r = new ['Instrument'] };
$instr.attach($loadlib_evt);
$instr.attach($class_evt);

$instr.run($args[0], $args);

Running this against “./examples/pir/io.pir” yields:

Library loaded: io_ops
test4
test5
test1
test2
test3
Library loaded: os
Class instantiated: OS

Well, it’s not much but its a start. Not too mention I’ve fixed most of my crashes already.

4th Weeks Tasks:

Rethink the callback interface: Passing so many things to the callback is not very fun.
Class events: Add ability to inspect per class.
Internal events: Exceptions, per sub calls.
Tests for Probe, EventLibrary: Now that the interface is mostly settled, start writing those tests.
File, line information: Figure out how to get file and line number information. Look at the profiling runcore/unhandled exceptions/etc. I think those print out that info.

parrot.org: Adding Optimizations to HLL Compilers with PAST::Pattern

Mon, 14 Jun 2010 04:48:32 +0000

Parrot Abstract Syntax Trees(PASTs) are one of the forms code being compiled in a PCT-based compiler takes before being evaluated or compiled to bytecode. With PAST::Pattern, you can find all the sub-trees of a PAST that match a certain pattern and transform them. This can be used to perform optimizations at the PAST level.

parrot.org: It's Finally Time to Write Some Optimizations (Almost)

Sat, 12 Jun 2010 08:43:49 +0000

Wherein I dash your hopes of spending the weekend hacking on making your favorite HLL compiler generate better code only to give you hope at the end that it will not be long at all before you can:

parrot.org: The Last Week and a Half in PAST Optimization

Mon, 07 Jun 2010 20:56:42 +0000

I noticed yesterday that I forgot to post a blog post last week. I'll try to make up for that with double the blog post goodness this week(expect a second post Thursday or Friday).

Khairul: Square one + a bit.

Fri, 04 Jun 2010 12:09:26 +0000

During my meeting with cotto on Tuesday (my time), he suggested looking into replacing entries within the op_func_table on the instrumented interpreter (CHILD). So the past few days, I’ve been working on it. The initial plan was to create a copy of that table, replacing ops that have hooks attached with a stub function that will fire the hooks before executing the op itself. The operation itself went swimmingly, until the monster known as “dynops” popped up to say “OHAI!”.

Before I go into the problems I face with dynops my initial approach, I have to say that cotto was right in that this approach will have better performance characteristics that what I was doing earlier, which was to store the hooks in a ResizablePMCArray which holds the hooks in a Hash. That was a bit of a cop-out on my part, as that was the path of least resistance in terms of adding new and removing hooks. Doing it in that manner simplifies the code for adding and removing hooks, at a cost of going through more levels of VTABLEs. With that, I’ve changed the data structure to hold the probes. It is now an array of linked lists, with a 1-1 mapping between each linked list and op. With this major change, running the example code [0] against examples/pir/mandel.pir yields approximately 5% performance increase (user 16.077s vs user 17.001s previously).

Back on topic, so the plan is to create a copy of the op_func_table, instrument it and change CHILD’s to refer to it instead of the core op_func_table. This was rather easy. Then, dynops dropped by.

With dynops in the equation, it did not turn out easy. Problem is, if the instrumented table is not switched back to the core table, the core table will end up pointing to the instrumented table instead at the end of dynops_register (see src/runcore/main.c). Also, how is the supervising interpreter (PARENT) supposed to know when new dynops are loaded in the CHILD? The PARENT has to know when the core op_func_table is changed, so that it can update its own references accordingly. Currently, dynops_register gets around this problem by prohibiting loading dynops when there is more than 1 interpreter in play.

So, hold on. “more than 1 interpreter in play”. But there are two interpreters, PARENT and CHILD! Apparently, creating interpreters through Parrot_new will only register the first interpreter in the list of interpreters held by core. Thus,

Parrot_Interp PARENT = Parrot_new(NULL);
Parrot_Interp CHILD = Parrot_new(PARENT);

PARENT will be registered, as traced through Parrot_new -> Parrot_cx_init_scheduler -> pt_add_to_interpreters (I might have missed some steps in between).

However, CHILD is not, as in Parrot_cx_init_scheduler (see src/scheduler.c), since CHILD has a parent interpreter, pt_add_to_interpreters (see src/thread.c) is not called. So, as far as core is concerned, there is only 1 interpreter in play, and that interpreter is PARENT. Thus that is why the check “n_interpreters > 1″ in dynop_register fails, and how I inadvertently was able to run 2 interpreters and have dynops loading, although it segfaults later on when PARENT tries to access its outdated tables. If CHILD is manually registered by calling pt_add_to_intepreters, then that is a no go, as then both PARENT and CHILD cannot load any dynop libraries. Given that CHILD has to be able to load dynop libraries, I’m at an impasse.

To recap, this is the problem at this juncture, if CHILD loads a dynop library, the instrumented table must be swapped out and replaced with the core table. After loading is complete, the instrumented table is then updated and swapped back in. After all that, PARENT must be notified of this loading so that it itself can update its own table references.

In my previous post, I mentioned making use of the events system to detect when CHILD is going to execute a “loadlib” op and to raise an event that PARENT can handle and do the necessary stuff. But that won’t work for “.loadlib” directives (see [1]), as that is not an op. So scratch that idea for now (although, I think it is a good idea for normal “loadlib” ops). Digging into the events system, I noticed that it already does some of what I was doing, in that, to check for events, a copy of the core op_func_table is made (see include/parrot/interpreter.h , interp->evc_func_table), and all entries in that table points to the “check_events__” op. This table is also helpfully taken care of in dynop_register, in that it helpfully extends it whenever new dynops are added.

Then I realised something, I’m going to create a copy of the op_func_table, and replace each entry with a stub function. This table will then be used for op lookup and execution by the runcore. And I control CHILD’s runcore. Why don’t I move the call to the stub function in the runcore itself. So that’s where I’m at now, square one + a bit.

To get around the problem brought about by dynops temporarily, I simply add a check in the runcore to update PARENT’s op tables whenever it does not match CHILD’s op tables. I did some thinking on this, maybe make dynop loading a STOP THE WORLD event, just like GC. This I think can be done in dynops_register, broadcasting to all to halt and when all is halted, proceed to load the dynop library before broadcasting again to do the necessary updates before resuming itself. This I will revisit again later, seeing that I’m running out of time for this week’s tasks, which is, to recap, bug hunting and squashing of tracer.nqp, implementing the event notifications library and tests.

So bug hunting. Apparently I did something and now I can obtain and print out the STRING KEY constants. But the segfault with example/pir/io.pir remains when I try to trace it using tracer.nqp. Running it under a debugger shows that it segfaults when trying to access INTERP->vtables[SELF->vtable->base_type]->_namespace (in src/pmc/default.pmc line 549). Before this line, control is in find_method_direct_1 (in src/oo.c line 1051), where the variable _class is being queried for its namespace. Only problem is, the base_type of _class is 101, which is not a valid number given that at that point in time, there is only about 87 pmc types.

Running simple-tracer.pir [0] on it has no problems, of which I can only suspect that a GC run happened and the object got cleaned up, so bad data was being read. This would make sense given tracer.nqp is written in NQP and it does lots of string concatenations, so that should cause the GC to run more frequently. Now, if there’s a tool that I can use to confirm this… Oh wait, I’m supposed to be making those tools.. GAH!

[0] Simple-tracer.pir (Edited 13/06/2010)

.sub '' :load :init :anon
    load_bytecode 'Instrument/Instrument.pbc'
    load_bytecode 'Instrument/Probe.pbc'
.end

.sub 'main' :main
    .param pmc args

    $P0 = shift args

    .local pmc pr, in, probe_class
    probe_class = get_hll_global ['Instrument'], 'Probe'
    pr = new probe_class
    pr.'make_catchall'()
    pr.'set_callback'('cb')

    in = new ['Instrument']
    in.'attach'(pr)

    $S0 = args[0]
    in.'run'($S0, args)
.end

.sub 'cb'
    .param int pc
    .param pmc op
    .param pmc instr

    print 'Op: '
    $I0 = op[0]
    say $I0
.end

[1] http://irclog.perlgeek.de/parrot/2010-06-01#i_2391934

Andrew Whitworth: ParrotTheory: Green Threads

noreply@blogger.com (Whiteknight) — Wed, 02 Jun 2010 13:30:17 +0000

This summer I'm mentoring GSoC student Chandon in his project to bring Hybrid Threads to Parrot. Today I'm going to talk a little bit about that, and maybe give some information about what Parrot needs to do to properly support a full and robust threading system in the long term. I've written about threads previously on this blog, so I won't cover all the nitty-gritty details again.

At the risk of over simplicity there are basically two types of threads to consider: OS Threads and Green Threads. OS Threads or "native threads" are what most people are probably familiar with: These are the threads managed by your operating system kernel. Native threads represent a flow of execution of native machine code on a single processor core, and multiple native threads can run simultaneously on multiple processor cores. On a single core, they run sequentially but in time slices so short it appears to the human observer that things are happening at the same time. This appearance of simultaneity is extremely important for computer users, especially for graphical interfaces, because people don't like waiting and don't like it when the computer "freezes" or "locks up" during periods of heavy computation.

A native thread represents a system context: A separate call stack and a snapshot of the processor state at any given time. By saving the processor state and call stack away, we can switch to a different task and later resume our original task as if nothing has happened.

I don't know where they were originated, although Green Threads were probably most famously implemented in early versions of the JVM. Instead of being managed by the OS, they are managed by the VM. Green threads are scheduled onto OS threads in the same way that OS threads are scheduled onto processor hardware. The VM saves it's current state into a VM context object, moves to a different task, and then reinvokes that context object to resume the original task.

Green threads are very interesting tools, when they are available. They tend to be extremely inexpensive compared to native OS threads. They are managed by the VM, and if they all execute sequentially on a single OS thread the risk of lock contention and data corruption drops almost to zero. The down-side to green threads is that they are all multiplexed onto a single OS thread, so they each get a smaller share of the processor's time.

To help enforce that green threads in the VM may only execute on a single OS thread at a time, the VM may implement a global interpreter lock (GIL). Famous examples of this occur in Python and Ruby. A GIL helps with integration to C libraries which are not thread-safe, and also helps to prevent things like callback functions from executing in a different OS thread and causing unprotected data corruption. The downside to the GIL approach is that extra overhead must be spent to perform locking/unlocking and checking of the GIL.

Chandon's Hybrid Threads idea tries to combine the two models of OS and Green threads. Parrot will maintain a pool of OS "worker" threads, based on user settings and the number of available processor cores. Parrot will then be able to schedule a queue of green threads onto the available OS threads. Yesterday on his blog he actually posted an example of using Parrot's existing continuations to form a system of cooperative green threads. Instead of happening at any time, green thread switches in his example are manually triggered by calling the functon th_resched(). If we move the call to th_resched into the interpreter's scheduler object so it can be called when the scheduler runs, we gain a proper preemptive green threads system without too much effort.

There are some difficulties with this model, especially since Parrot currently has no mechanism to manage data contention across OS threads, and also considering that each OS thread ill have it's own interpreter object, which will need to be created at some runtime expense. Parrot is also going to need to add new mechanisms and policies for dealing with data which is considered "global": namespaces and class definition metaobjects, etc.

We're also going to need to start answering some very difficult questions: If a continuation created in interpreter A is passed to interpreter B and invoked, what do we do? If an exception is thrown by interpreter A and is caught by a handler defined in interpreter B, what do we do? If an object's vtable override is preempted by a green thread switch and the object is in an inconsistent state when it is attempted to be accessed in a different green thread, what do we do? How do we even detect when any of these things happen? What mechanism are we going to need to write to allow sharing PMCs between OS threads? Do we use STM again? If so, do we write STM ourselves or try to use a pre-existing and proven library? Considering that we now have immutable strings (which don't need to be locked because they can't be written), do we maybe want to consider a system where shared PMCs are either read-only in other threads or maybe are COW to prevent corruption? How do we deal with long-running operations, such as blocking IO and calls to long-running library functions?

For Chandon's project to be a success I think we need just two things: a robust green threads system, and a prototype OS threads system with a scheduler that can intelligently assign green threads to OS threads. Once we have that system in place, along with some rules or algorithms against sharing writable data until we sort out the details for how that becomes possible, we can add new mechanisms piece-wise to enable new features and functionality. If he has time sure he can do more, but if we can get the basics in place we will be in a much better state than we were before he started.

parrot.org: Threads are continuations

Tue, 01 Jun 2010 20:46:06 +0000

Parrot has Continuations. In fact, Parrot is *based* on Continuations: rather than having a call stack and stack frames, each sub call has a return continuation - a pointer back to the call frame and bytecode position to return to after the sub completes.

As the Scheme and Ruby users keep telling us, continuations are pretty neat stuff. In fact they're so neat that they give us light weight cooperative threads "for free".

parrot.org: NFG is (somewhere in the vincinity of...) here

Tue, 01 Jun 2010 17:49:42 +0000

"N̈" (Or "n̈" if you don't like caps) is a grapheme from several minor extended Latin alphabets. It occurs in the orthographies of the Jacaltec Mayan dialect, Cape Verdean Creole, and in the rockumentary "This is Spın̈al Tap". Today I want to talk about the injustices this symbol has faced in the past and how, starting today, parrot can right them.

Khairul: Week 1 Synopsis

Sun, 30 May 2010 17:14:55 +0000

Last week marked the start of the GSoC coding period. Prototyping during the bonding period helped me in creating an initial version of my framework and currently, I’m focusing on creating an interface to inspect each opcode that is being executed. This interface is the class Instrument::Probe [0]. Initially, this class was written in PIR. It quickly became rather untenable, with labels and jumps all over the place. My mentor, cotto, suggested implementing it in NQP instead, and that suggestion has proven to be a godsend. The code is cleaner, and it’s easier to follow, not to mention adding new features to it.

As of now, the class Instrument::Probe allows the creation of probes that inspect the opcodes being executed. Each probe has its associated callback and a list of opcodes that it is inspecting. Whenever the runcore encounters any of these opcodes, it will call the associated callback. This system is rather flexible in that probes can be enabled or disabled dynamically, and multiple callbacks can be associated with a single opcode. With that in mind, each probe also has an associated finalize callback which is called only at the end of execution and only if the probe is enabled at that point.

With that in mind, I’ve implemented a simple tracer example (examples/library/tracer.nqp [1]) that tries to mimic the output of the tracing runcore. It mostly works alright and is rather slow, which is a given seeing how much work it is doing per op, both at the PIR (or rather NQP) and the runcore levels. Not to mention its tendency to segfault every now and then, which I’m still in the process of tracking down. One current segfault happens when running it against examples/pir/io.pir, and it doesn’t happen when a simpler tracer that only prints the op number is used. Further testing shows that it seems to happen on the opcodes “get_results” and “set_results”, although I’m not quite certain why, probably because I’m accessing something the wrong way. Bug squashing is not a fun activity.

For this week, I’m working on to solve a few problems in addition to tracking down the segfaults above. First of which are dynops. This past week has seen major changes to the op libraries within parrot, due to “ops_massacre”. These changes made me think about dynops and how it will affect my project. Currently, the runtime library does op lookups using the OpLib and OpCode pmcs, and this lookup is done before the code is loaded/compiled. Due to this, the opcodes contained in the dynop libraries will be non-existent from the point of view of the library . One way i’m thinking of to solve this is to defer registering hooks for dynops, trying whenever new dynlibs are loaded. This can be done with a simple probe that looks for the opcode “loadlib” and raises an event when that opcode is encountered (I’m assuming “.loadlib” and “loadlib” works the same way). Which brings me to the question of is there an event system I can use? Simple tests and some digging (t/pmc/scheduler.t and PDD-24) showed me how to use Parrot’s event system and I think I can utilise this to implement the above and to serve as the basis for the other events such as PMC creation and GC mark cycle, barring any unforeseen issues.

To recap, I spent most of this week learning just enough NQP to reimplement the runtime library in NQP and chasing down bugs uncovered by the simple tracer example. This coming week will be spent on squashing the segfaults the tracer produces and make it faster and implementing events and a runtime library to abstract it away. And I should start on the tests too.

[0] https://svn.parrot.org/parrot/branches/gsoc_instrument/runtime/parrot/library/Instrument/Probe.nqp
[1] https://svn.parrot.org/parrot/branches/gsoc_instrument/examples/library/tracer.nqp

parrot.org: call for Parrot foundation members applications

Sat, 29 May 2010 13:30:41 +0000

We're coming up on the annual Parrot Foundation member's meeting. Which includes the nomination and election of the next Parrot Foundation board of directors. And being so, the foundation is inviting everyone eligible for membership to apply. To be eligible for becoming a member of the Parrot Foundation you must:

be nominated by two current members
have made at least two contributions to the project (or a language implementation targeting the project)
sign a contributor agreement

Andrew Whitworth: The AOT Advantage

noreply@blogger.com (Whiteknight) — Sat, 29 May 2010 11:50:48 +0000

In my previous blog post I briefly mentioned PHP, HipHop, and AOT. For readers coming late to the party, HipHop is a native code compilation system for PHP, created by Facebook. The idea is that compiling down a relatively large subset of PHP (basically everything besides eval) to C++ code and passing it to an optimizing compiler can produce a much more efficient program than interpretation alone or even compiling to bytecode in advance and executing that.

Let's look at this same idea, but in relation to Parrot's intended LLVM backend system.

We can start off with a basic PHP compiler. The compiler doesn't need to be particularly fast since compilation happens once in advance. Sure, we don't want compilation to take forever because that slows down development, but it doesn't have the same kind of speed mandate we would have in an interpreted environment where we compile the script every single time we run it.

Our basic, feature-light compiler compiles the PHP code down to PAST. From there we kick in a series of optimization stages. Again, if we know we're doing ahead-of-time compilation, we can add more optimization stages up front and take a little more time to compile. For a release build of a high-throughput application, maybe we really want to swing for the fences with this one and enable every optimization in the book. Let's hope it's a long, well-written book.

We compile the PAST down to Parrot bytecode, after all the optimizations have run their course. At this point, we hand off the bytecode to the LLVM backend. LLVM converts the Parrot bytecode into LLVM IR, and passes that through it's own impressive suite of optimizations. At the end, LLVM spits out a native executable which is screaming fast.

The converse side is a case where we want to shorten the turnaround time, such as when we are doing direct interpretation, or even making executables for debugging purposes and don't want our developers sitting around for a long time waiting for the thing to compile. We don't want to completely avoid optimizations, because after compilation the developers have huge and expensive test suites to run which can take much much longer.

What we do in this case is compile the code to PAST, maybe run a handful of "good bang for the buck" (GBFTB) optimizations like constant folding, and compile the PAST down to bytecode. We'll avoid the last step and just execute the bytecode directly, since this is a debug build. When we run it, the JIT engine will kick in to do basic runtime compilation of detected hot spots, giving us nice speedups for programs with tight loops. The JIT compiler will likewise use a few of the GTFTB optimizations, but won't go crazy; we're doing this at runtime now and we have a user waiting.

There are some people in the world of scripting languages who view the complete lack of native executable support as a virtue. After all, they say, the nature of these languages is fast turnaround; We can make changes and run them directly without needing to invoke a separate compiler to produce a separate executable. Plus, the immediate availability of human-readable source code furthers the common goals of open source software and knowledge sharing.

Fast turnaround, from script to execution, certainly is a virtue--in development. Once we're ready to cut that release however, things change pretty dramatically. At my job, I write some applications in C#. During development, I keep things in "Debug" mode, and do lots of tests very quickly. When it's time to cut that release I change the VisualStudio configuration to "Release" and then click "Build All". Where I was doing incremental debug builds, now I'm doing a full release build with optimizations turned on and debugging symbols stripped out. I also build a separate installer project, and to test everything I need to install the software first and run it from it's installed location. This is a lot of extra cost, but I don't do it frequently. I go from about a 10 second compile time turnaround to almost 10 minutes worth of compilation and installation work. The net result is a faster, better executable for my end users.

In the case of Facebook, using HipHop to compile PHP down to native machine code improves throughput performance by 30%, which translates into lower server load, lower wait times for users and, at the end of the day, real money. Say what you want to about obfuscation of the source code, 30% is nothing to ignore.

I like to think that the Parrot infrastructure will support any workflow that our end users want. Having speedy interpretation around for people who want that is a very good thing, and we do it already. But, having the ability to employ tiered libraries of optimizations and create native machine binaries for fast execution is a very very good thing as well. This is why I keep pushing for our eventual JIT system to also support AOT as well. LLVM will already handle all the nitty-gritty, so the amount of scaffolding we'll have to provide in Parrot is minimal in comparison.

Andrew Whitworth: Compilers Targetting Parrot

noreply@blogger.com (Whiteknight) — Fri, 28 May 2010 21:56:29 +0000

I got to thinking today about Parrot and came to an interesting conclusion: Writing our own parsers for various existing languages is nice, but if we could get the language communities involved it would be even better. There's a symbiotic relationship that could form from such an arrangement and I would like to talk about that here for a little bit.

Let's take PHP as an interesting example. The PHP community has been talking about some kind of rewrite of their core engine to support Unicode from the ground-up. I'm not really involved in those discussions, so I don't know what the current state of the discussion is. I couldn't even tell you if work has started on such a rewrite. However, I do know one thing: If the next version of the PHP compiler targeted Parrot as the backend, it would have built-in Unicode support for free.

There are some reasons why PHP people might not like this option, considering PHP's niche in the web server world and Parrot's current issues with performance and memory consumption. But, as I'll describe here, those are relatively small issues.

Pulling an extremely wrong number out of a hat, let's say that it takes 10,000 man-hours to construct a modern, though bare-bones, dynamic language interpreter from the ground-up. You would have some features like GC, but wouldn't have others like JIT. Unicode support but a small or even non-existant library of built-in optimizations. You build a compiler, you build the interpreter runtime, and then you build a language library runtime so the programs you write don't have to reinvent every wheel from scratch. It's a daunting task, but it's very doable and has been done many times before.

Now, let's say you take that same developer pool of 10,000 man-hours and instead build a compiler on top of Parrot instead of writing a new VM from the ground up. You do get a whole bunch of the really tricky features immediately: GC, Exceptions, IO, JIT (to come), Threads (to come), Unicode, Continuations, Closures, a library of Optimizations (to come), and all sorts of other goodies. You also get a nicer environment to write your parser in than bare lex/yacc (though, I think, you could still use lex/yacc if you really wanted). You get some libraries already, though you would likely need to write wrappers at least to get things into the proper semantics of your language. You really do get a hell of a lot and you could easily have a working compiler, at least the basics of one, up and running within a few weeks.

Let's even be conservative and say that your team can't build the compiler front-end in less than 1,000 man-hours on Parrot. Even then, you still have 9,000 man-hours left. You could spend some of that making a gigantic runtime library, but I like to think you could devote some of that effort back to Parrot too. If you spent even 5,000 hours making Parrot faster, or adding new features, or whatever you need, you'll still be spending less effort overall and getting a comparable result, if not a better one.

Keep in mind, on Parrot, you're going to get all the features that other people add, in addition to the ones you added yourself. Parrot really is a symbiosis: You hack Parrot to add what you need, and in turn Parrot gives you all sorts of other things for free.

Parrot has plenty of stuff around to get you started, even if it's not perfect right now. Remember that Parrot is extremely actively developed. You can do your development now, build your compiler and get your language ready and grow your regression test suite. When you decide that you need things improved, you can make targetted improvements incrementally. If you want a better GC, for instance, you can go in and just add that part. But, you have the benefit of knowing that Parrot already has one and your test suite team doesn't have to wait for your GC team to finish writing their first GC. That's good, because those damned GCs can take a while.

My point, in a nutshell, is this: Parrot has some really cool features, but the coolest is that it provides a standard base level, a standard toolbox, that you and your compiler team can take and use immediately if you want it. You can get started right now with the kinds of high-powered features and tools that it would take you years to get right by yourselves. Yes there are improvements needed and some features yet to be desired, but adding them to Parrot represents smaller, more incremental improvements than trying to write an entire interpreter yourself.

Parrot is adding some really cool features this year too. I fully expect by the 3.0 release we will these new things:

A new, fully-featured robust threading system
A pluggable optimization library, with optimizations able to be written (likely with some bootstrapping) in the high-level language of your choice
A new GC with radically improved performance characteristics
An extremely powerful instrumentation and analysis framework (again, with tools able to be written in the language of your choice, possibly with some bootstrapping)
A flexible native function call interface for interacting with existing libraries

If you can wait a little later, I fully expect by 3.6 or 3.9 that we will be rolling out a powerful LLVM-based native code compilation core which will support JIT and (I hope!) AOT. You don't have to look much further than Facebook's HipHop project to see that compilation of dynamic languages down to native executables will be a big benefit, especially if we can aggressively optimize all the way down. I'll talk about that particular issue in a later post.

Parrot does have some problems now, like performance. I can certainly understand the criticism on that point. However, for a fraction of the effort it will take you to write a new interpreter or virtual machine from the ground up you can target Parrot instead, get a bunch of free features, and maybe spend some of your free time helping us add the optimizations and improvements that you need. I really and truly believe that it's a win-win proposition for everybody involved.

parrot.org: PAST::Transformer and the Foundation for PAST Optimization

Thu, 27 May 2010 01:14:08 +0000

Coding for GSoC officially began this Monday, but I decided to get a head start and started working on PAST::Walker Thursday. PAST::Walker is the foundation for the rest of what I will be doing regarding PAST Optimization, and the foundation is largely laid. There have been a few complications, and a few decisions remain to be made(and implemented) before I can say that PAST::Walker and PAST::Transformer are finished, but implementing PAST::Walker has gone smoothly and quickly for the most part.

parrot.org: Dynamic code points, the grapheme tables and not getting your services denied.

Tue, 25 May 2010 01:15:44 +0000

One of the features of NFG I've mentioned before is that it solves our problems with variable width characters without taking additional storage space over UCS-4, on top of which it's defined. The artifact that allows us to pull off that trick is called the 'grapheme table' and today, I'll try to explain how it works.

parrot.org: How do threads fit into parrot?

Tue, 25 May 2010 00:54:53 +0000

(This post is to show my thinking on my GSoC project. Sorry in advance if it's not especially coherent or interesting out of context.)

Over the past week I've been examining the Parrot code to figure out how hybrid threads are going to fit in. First, if I'm going to make Parrot do stuff concurrently, I have to figure out what Parrot is actually doing at all.

Andrew Whitworth: The Fitness of Parrot as a Target Platform

noreply@blogger.com (Whiteknight) — Fri, 21 May 2010 14:05:37 +0000

In response to the blog post I wrote the other day, I received a very poignant question regarding Parrot's stability as a target for HLL compilers. Since the comment was anonymous and I can't ask permission, I'm going to repost a portion of it here:

However to be honest, its scary that over the last 10 years so much was done and discarded and is being rewritten. I agree with the "throw away the first" reasoning but that is something I would expect in a 2-3 year project not a 10 year old project.

Could you comment on the risk associated with for a HLL project that starts to build on Parrot with an ETA of say 1 year? Can it be said with a reasonably level of certainity that whatever sub-projects (JIT, treads etc.) are added to Parrot now, they will take an evolutionary approach rather than being scrapped and rewritten?

My intent is not to put down Parrot as I really want to use it but I want to voice my fears and hope to understand the parrot-dev perspective on making Parrot a stable (for some definition of stable) platform to build on.

This really is two questions rolled into one:

Parrot has been in development for about 10 years, why are these big changes being made so late in the project?
Are these big changes going to affect the stability of the external API, and is Parrot a stable, reliable platform for HLL designers?

I'll try to tackle these two separately.

The 0.0.1 release of Parrot was cut in September 2001, according to documentation in our repository. I don't know much about the early history of the project; I've only been a Parrot contributor for about two and a half years now, about one fifth of the total life of the project. Looking through the NEWS entries starting back at 0.0.2, it does look like things were moving at a rapid pace back in 2001 and so maybe Parrot should be a lot further along than it is now.

There are a few responses to this. The obvious ones: Volunteer developers are a limited resource, and Parrot is an enabling technology so many developers have been focusing their efforts outside the core (HLL compilers, libraries, programs written using both of those, etc). Core development effort and HLL compiler development effort are certainly not directly transferable, but in the early years there was a lot of cross-work among the central developers, and this did play a factor. There are less obvious reasons, such as attempts to support various platforms (Parrot has supported or attempted to support several architecture and compiler combinations, some of which were exotic even 10 years ago), attempts to improve performance throughout (sometimes prematurely), and the fact that the NEWS file is misleading because the early "action-packed" releases that seem to show a high development velocity were not time-based releases and often represent several months of work. Hell, there were only 6 releases in all of 2003 and 2004 combined, but the NEWS entries for those releases don't look any more impressive than the news release for 2.4. The pace of development is certainly not constant (though I would say it is increasing) and while 10 years looks like a lot, it's 10 years of inconsistent development time from contributors with other things going on as well.

The biggest factor to keep in mind is that Parrot development from 2001 until 2010 didn't happen in a straight line. Parrot started out as simply the internals engine for the Perl6 compiler and grew to become larger than that. Along the way a lot of designs and plans were hammered out, then thrown away entirely. As I mentioned last time however, there isn't a whole hell of a lot of prior art in the area of dynamic language VMs; guesses were made and some of them ended up being wrong.

We could have declared Parrot more or less "complete" some time ago and shipped a program that was known to have some serious flaws. Instead what we've been doing, and what I think is the more responsible course of action: fix it until it is correct. So to answer the first question: yes it's been 10 years, but Parrot isn't perfect yet and we are going to continue working on it until it is, however long that takes.

The second question can itself be decomposed into several parts: Will things that I rely on change? If so, how quickly will they change? Will the work I do now need to be thrown out when the next big feature set lands? Is Parrot mature enough for me to build on top of it a software project that I want to lead into maturity? These are all good questions, and very concerning for HLL developers. The answer isn't clear-cut, but I will try to explain it as well as I can.

The most important point I can raise when tackling this question is to mention our support policy. Our support policy, which I have complained about as being too strong, goes something like this:

We have 4 supported releases per year, every 3 months. We support these releases, including bug fixes, until the next supported release comes out. HLL compilers and other external projects are highly encouraged to target these supported releases (many do not, but they do so at their own risk).
We have a defined external API which cannot be changed without notice. A deprecation notice for any change to any externally-visible feature or behavior must be included in a supported release before that interface can be changed or removed. Assuming you target supported releases as suggested in #1 above, this means you have 3 months of prior warning to prepare your project before disruptive changes to Parrot are made.
New features that are added will be tagged "experimental" so projects can examine them, provide feedback, and get an idea of where things will be going in the future. If people start relying on experimental features and new problems aren't created, they tend to stay around.
Not quite part of the policy, but still relevant: When we cut releases, or change features, we try to do extensive testing in HLLs and external projects. Where problems are found, we typically try to either offer fixes or workarounds.

It's interesting to note that many of our most disruptive changes in recent months were actually driven by the HLLs and external projects themselves, as fixes to bugs or longstanding problems. It's also worth mentioning that when the question is raised most HLL developers seem to want Parrot to make fixes and improvements more rapidly than it has been doing.

Another part of the question is whether the new features and things that we will be adding/changing in coming months will be gradual and be able to be inserted non-disruptively into existing software projects. Let's give a quick rundown of some systems that could be seeing major changes in the next couple months:

GC: GC is a very internal thing, when it works properly, you don't even need to know it exists. Any changes here, so long as they don't introduce bugs, will be invisible to the HLL developer, and will only serve to improve performance (or, improve it under certain workloads, depending on algorithm).
JIT: In the past the JIT system was separate, and you needed a separate command-line switch to activate it. In the future, I'm hoping we get a trace-based system that kicks in automatically when a need is detected. JIT, like GC, shouldn't change execution behavior, only performance, so changes here should be invisible to the HLL developer.
Threads: We don't really have a good, working, reliable threads implementation now and HLLs are generally not using them. Anything we add/change/fix will be as good as a newly-implemented feature and can be added post-facto by HLL developers with no stress.
NCI: This mostly affects users of external libraries, and writers of interfaces to those projects. Improved NCI gives us improved access to more libraries. The interface here may change in a disruptive way but, I hope, this won't be a huge issue to most projects
Packfiles: Packfiles aren't really portable now, so people haven't been using them to their potential. In any case, Packfile structure and handling are mostly transparent to the HLL developer. In the future what we will see are better usage patterns, improved performance and decreased bug volume, at no expense to the HLL developer.
PCC: We have a pretty extensive library of tests for PCC behavior and a defined standard inferface that won't be changing any time soon. Sure, the internals may change (hopefully become faster) and new features will be added, but exising code will not notice.
PASM: We're looking to add a suite of optimizations to PASM. These optimizations will likely be opt-in initially, so absolutely nothing changes for HLL developers who don't want them. If you do want them, your programs will probably only get faster at runtime at the cost of some additional processing in the compiler. This is a standard and non-exciting trade-off that most good compilers offer to their users.

In summary, Parrot is a good, stable platform for HLL developers to use. Yes there are big changes planned to the internals of the VM, but most of them are going to bring big improvements to the compilers and library projects that run on top of Parrot, without affecting the externally-visible interface much and without bringing new problems. That really is one of the driving benefits of a VM system like Parrot: Write the improvement once and get the benefits everywhere. The interface is in flux, but changes happen slowly and with plenty of prior warning (and support, where support is needed). As the big systems in Parrot get fixed, things will get even more stable and the benefits will become even more apparent.

I hope that helps answer some concerns.

Andrew Whitworth: Parrot 2.4.0 "Sulfur Crest" Released!

noreply@blogger.com (Whiteknight) — Fri, 21 May 2010 13:56:01 +0000

"So there me was beating boulder into powder because me couldn't eat it, and magic ball land in lap. Naturally me think, "All right, free egg." because me stupid and me caveman. So me spent about three days humping and bust open with thigh bone so me could eat it good. Then magic ball shoot Oog with beam, and next thing me know me go out and invent wheel out of dinosaur brain. Magic dino wheel rolls for three short distance until me eat it. The point is, me get smarter. Soon me walk upright, me feather back dirty, matted hair into wings for style, and me stop to use bathroom as opposed to me just doing it as me walk. " -- Oog, Aqua Teen Hunger Force

On behalf of the Parrot team, I'm proud to announce Parrot 2.4.0 "Sulfur Crest." Parrot is a virtual machine aimed at running all dynamic languages.

Parrot 2.4.0 is available on Parrot's FTP site, or follow the download instructions. For those who would like to develop on Parrot, or help develop Parrot itself, we recommend using Subversion on our source code repository to get the latest and best Parrot code.

Parrot 2.4.0 News:

- Core
  + Various long-standing bugs in IMCC were fixed
  + STRINGs are now immutable.
  + use STRINGNULL instead of NULL when working with strings
  + Fixed storage of methods in the NameSpace PMC
  + Added :nsentry flag to force method to be stored in the NameSpace
  + Added StringBuilder and PackfileDebug PMCs
  + Added experimental opcodes find_codepoint and unroll
- Compilers
  + Fixed reporting of line numbers in IMCC
  + Removed deprecated NQP compiler, replaced with new NQP-RX
  + Removed NCIGen compiler
- Deprecations
  + Tools to distribute on CPAN were removed
  + Deprecated dynpmcs have been removed to external repositories
  + Removed RetContinuation PMC
  + Removed CGoto, CGP, and Switch runcores
- Tests
  + Many tests for the extend/embed interface were added
  + done_testing() is now implemented in Test::More
- Tools
  + The fakexecutable tapir is renamed parrot-prove
  + Performance fixes to the pbc_to_exe tool
  + Fix data_json to work outside of trunk
  + The dynpmc GzipHandle (zlib wrapper) was added
  + The library Archive/Tar.pir was added.
  + The library Archive/Zip.pir was added.
  + The libraries LWP.pir, HTTP/Message.pir & URI.pir were added.
- Miscellaneous
  + Six Parrot-related projects accepted to GSoC
  + Improve use of const and other compiler hints

Thanks to all our contributors for making this possible, and our sponsors for supporting this project. Our next release is 15 June 2010.

Enjoy!

parrot.org: GSoC Project: NCI and Stack Frame Improvements

Thu, 20 May 2010 18:55:15 +0000

My Google Summer of Code (2010) objectives are to integrate libffi into the NCI framework and to build a new Stack Frame builder that takes advantage of the llvm.

Andrew Whitworth: Bright Blue Yonder

noreply@blogger.com (Whiteknight) — Thu, 20 May 2010 01:04:09 +0000

Yesterday I released 2.4, "Sulfur Crest". It's a cooler, more succinct name than "Sulfur-Crested Cockatoo". Every time I do a Parrot release I find myself searching through Wikipedia to look for interesting Parrot names (and colorful pictures too!).

I picked an interesting quote for the release announcement. Amid the broken grammar and colorful imagery was a key point that I wanted to express: Parrot is getting smarter.

If you had told me two years ago, or even a year ago, that Parrot would have the features it has now, or that it would be on the verge of adding the features that we are currently planning, I might have been incredulous. Parrot has matured a lot recently, and I really think all that effort is going to start paying off.

Parrot had a lot of features when I first joined the project. Many of these were first drafts and prototypes which have since been ripped out or reimplemented. This is a normal, natural, healthy part of the software development process, and nobody should be surprised or upset about it. Pop quiz: Name for me one other dynamic language virtual machine, open-source or otherwise, that aims to support the same number of programming languages and runtime environments that Parrot aims to support? Smalltalk comes to mind as a virtual machine for a dynamic language, but that VM backend was never intended to support anything besides Smalltalk. At least, not well. .NET supports a few languages, but is certainly not dynamic.

There's a great rule in programming that we always throw the first one away. The first draft is never the final manuscript, the first stab in the dark never hits the target. The original Parrot designers and developers didn't have a whole hell of a lot of prior art to copy from. They made some guesses, but there was no way they could have gotten everything right the first time, no way they could have forseen all the problems that the project would run into over time. But that lack of information and that uncertainty can't hold people back. You grit your teeth, you write a prototype, and you resolve to yourself that you always throw the first one away.

Let's take JIT for an example. In a basic JIT system, you have a snippet of code with a runtime cost of E. A JIT attempts to compile that code into a piece of machine code with a runtime:

OE < E

Where O is some sort of optimization factor. Of course, there is some kind of overhead H, in compiling the code at runtime:

H + OE < E

H involves constructing the machine code and performing optimizations. The more optimizations we perform, the larger H gets, but the smaller O can get. If E is large enough and if we tune H properly, eventually we cross a threshold and the left side of the inequality becomes smaller than the right side of the equation. At this point, JIT is a net performance win for the application. Simple, right?

Anyway, I'll skip the rest of the theory lesson. The point is this: Parrot's original JIT didn't really have anything a modern compiler system would recognize as "optimization passes", so it wasn't really tunable. It also didn't readily support code generation on any platform besides x86. These things weren't going to be easy (or, even possible) to add either. When the burden of maintenance became too high, and when it was clear that the old prototype system was doing more harm than good, we ripped it out.

There were a number of such systems in early Parrot, prototype code that got Parrot up to the first plateau but needed to be redesigned and reimplemented for Parrot to move up to the next one. There are several projects either going on right now, or about to get started to work on these systems. When you look at the list, you realize that while it's ambitious, with the current team and the current state of the VM it is entirely plausible that they will all succeed. A quick overview:

Allison, Bacek and chromatic are talking about several ideas for implementing new, more efficient GC algorithms to replace our first GC prototype.

Plobsing has been hard at work redoing our freeze/thaw serialization system and, along with NotFound and others is working to fix several long-standing bugs in the PBC system.

Nat Tuck is preparing a GSoC project to implement a new threading system to replace the first prototype version of that system.

Tyler Curtis is preparing a GSoC project to implement a first prototype of a PAST optimization system.

Daniel Arbelo is preparing his GSoC project to implement a first prototype of the NFG string normalization form.

Muhd Khairul Syamil Hashim Is putting together a GSoC project to implement an instrumentation framework for Parrot so we can get some high-quality analysis and debugging tools for Parrot.

John Harrison is preparing his GSoC project to develop a new NCI frame builder with LLVM, which will replace the old prototype frame builder and (hopefully) lay the groundwork to start replacing our prototype JIT system.

...and there are even other projects that I can't think of off the top of my head right now too.

If you had told me a year or two ago that all these projects would be on the table, or that we would have such high chances of them all succeeding, I would have called you crazy. But after all the work we've put in between then and now, and considering the high caliber of our development team, I'm feeling pretty confident that we will be successful and Parrot is going to gain some of these awesome new features. Cross your fingers!

parrot.org: UCS-4, NFG and how the grapheme table makes it awesome.

Tue, 18 May 2010 21:06:26 +0000

After laying down the foundations of what NFG does in previous blog posts I've started implementing, as part of my work in this Summer of Code, a new Unicode encoding for parrot, UCS-4. In this post I'll try to explain what it is and how it makes NFG easier to achieve.

Khairul: Ops

Mon, 17 May 2010 15:08:49 +0000

In Parrot, opcodes are currently implemented as a standalone piece of code that is invoked whenever its corresponding opcode number is encountered in the bytecode. Each opcode is described internally through the use of an op_info_t struct, which, taking from <parrot/op.h> is the following:

typedef struct op_info_t {
    const char    *name;
    const char    *full_name;
    const char    *func_name;
    unsigned short jump;
    short          op_count;
    arg_type_t     types[PARROT_MAX_ARGS];
    arg_dir_t      dirs[PARROT_MAX_ARGS];
    char           labels[PARROT_MAX_ARGS];
} op_info_t;

These descriptions are stored as an array, with the index of each entry being the opcode’s own opcode number. This array is pointed to by a field within the interpreter struct, the field being “op_info_table”. This is just for the descriptions of the opcodes. In Parrot, currently each opcode is defined in a C-ish language that gets parsed into C source code, with each opcode constituting a function that satisfies the following function prototype:

typedef opcode_t *(*op_func_t)(opcode_t *, PARROT_INTERP);

Similar to the opcode descriptions, these opcode function pointers are stored as an array, with the opcode’s number being the index to the position of the opcode’s function pointer. This table is pointed to by the field “op_func_table” within the interpreter’s struct.

So what happens during execution? A pointer, which is the program counter (PC), is passed to the runcore’s runops function. This runops function is the one that looks up the function pointer for the current op pointed to by the PC and calls that function for execution.

The code for an opcode will do a number of things. First, it will grab the current context of the interpreter. The current context consists of a number of items, but chief and most important is that the current context holds the current values of the registers (Parrot being a register-based VM). With this context, it will then grab the required values from certain registers and after performing the required logic, will write values to a certain register. After that, it will advance the PC by a certain number, which generally is (PC + 1 + NO_OF_PARAMS)).

So how would you know which registers the parameters are in? The PC actually points to somewhere within the bytecode. Generally, the PC will point to the start of an opcode. If an opcode has no parameters, advancing the PC by 1 will get us to the next opcode to execute. However, if the opcode has parameters, advancing the PC by 1 will get us the register or INTVAL constant for the first parameter. Similarly, advancing by 2 will yield the register or INTVAL constant for the second parameter.

This is all well and good except for the fact that Parrot allows the loading of dynamic op libs, which extends the capabilities of the VM by providing additional ops, similar to how MMX/SSE/SSE2/etc extends the capabilities of an x86 processor. Parrot also shares the two tables “op_info_table” and “op_func_table” between interpreters, which does make sense, as having duplicates of these tables can be expensive memorywise, given that Parrot already has more than 1000 core opcodes.

Handling these two issues won’t be trivial. With regards to dynamic op libs, there needs to be a way to detect when the library being loaded is a dynop library. One way is to trap loadlib opcodes and then peek to check if the op tables change. It would also be good to investigate if we can duplicate any shared tables such that the interpreter running the instruments has its own private tables that will be untouched by any changes to the tables of other interpreters.

Andrew Whitworth: Immutable Strings branch performance

noreply@blogger.com (Whiteknight) — Thu, 13 May 2010 12:35:11 +0000

Parrot now uses immutable strings internally for it's string operations. In a lot of ways this was a real improvement in terms of better code and better performance for many benchmarks. However, many HLL compilers, specifically NQP and Rakudo suffered significant performance decreases with immutable strings. Why would this be?

It turns out that Immutable strings are great for many operations. Copy operations are cheap, maybe creating a new STRING header to point to an existing buffer. There's no sense to actually copy the buffer because nobody can change it. Substrings are likewise very cheap, consisting of only a new STRING header pointing into the middle of an existing immutable buffer.

Some operations however are more expensive. A great example of that is string appends. When we append two strings, we need to allocate a new buffer, copy both previous buffers into the new buffer (possibly translating charset and encoding to match) and creating a new header to point to the new buffer. With the older system of COW strings, appends were far less expensive, and many pieces of code--especially NQP and PCT code generators--used them in large numbers.After the switch to immutable strings, any code that was optimized to use lots of cheap appends began to take huge amounts of time and waste lots and lots of memory.

The solution to these problems is not to use many bare append operations on native strings, but instead to create a new StringBuilder PMC type. StringBuilder stores multiple chunks of strings together in an array or tree structure, and only coalesces them together into a single string buffer when requested. This allows StringBuilder to calculate the size of the allocated string buffer only once, only perform a single set of copies, not create lots of unnecessary temporary STRING headers, etc.

Several contributors have worked on a branch, "codestring", for this purpose, and some results I saw this morning are particularly telling about the performance improvements it brings. Here's numbers from the benchmark of building the Rakudo compiler:

JimmyZ: trunk with old nqp-rx real:8m5.546s user:7m37.561s sys:0m10.281s
JimmyZ: trunk with new nqp-rx real:7m48.292s user:7m11.795s sys:0m10.585s
JimmyZ: codestring with new nqp-rx real:6m58.873s user:6m22.732s sys:0m6.356s

The "new nqp-rx" he's talking about there is nqp-rx modified to make better use of the new CodeString and StringBuilder PMCs in the branch. The numbers are quite telling, build time is down about 12%. I think the finishing touches are being put on, and the branch will be merged back to trunk soon.

chromatic: Perl 6 Design Minutes for 05 May 2010

Mon, 10 May 2010 06:07:34 +0000

The Perl 6 design team met by phone on 05 May 2010. Larry, Allison, Patrick, Will, and chromatic attended.

Larry:

various spec updates, some major
removed p5=> description because it's not supported in core
deleted self:sort construct because self isn't a real syntactic category
explained Perl patterns in terms of PEGs, and spec'ed tiebreaking rules explicitly
last but not least, finally purveyed the long-threatened revamp of proto to keep routine and method semantics similar
they all now work much more like the multiple dispatch semantics currently used by STD, where we always call the proto first
the proto is then always in charge of the actual multiple dispatch; it can of course delegate that
and the default for a null body corresponds closely to current semantics
in hacking news, the lexer generator mislaid any alternative that was a bare . pattern, so cursor_fate never called its alternative, oops
took me a long time to run that one down, because it resulted in a horrendous backtrack causing mysterious misplaced errors
revamped character class parsing to be more helpful and correct
STD now check a normal regex bracket's innards for old-school character class, and warns if found
added a .looks_like_cclass method to Cursor to detect most accidental uses of P5 ranges
some valid P6 brackets will complain, but the workarounds are easy
just put whitespace on both ends is one way
removed a few of these old-school-ish character classes from STD
changed :tr language to :cc language since character classes share it
(translation pays more attention to ordering, but the language is the same)
turned out parsing character classes discovered issues in STD; various character classes needed to backslash # that would otherwise be a comment
to that end, we now allow \# in character classes instead of misparsing as unspace
if we find an invalid - in a regex, we now presume we're in an old-school character class and fail with a sorry instead of a panic to give the character class code a shot at it
STD now uses ~ syntax for regex brackets to set $*GOAL correctly
cleaned up recursive panic detection; it was possible to get both false positives and negatives before
STD shouldn't use 'note' to emit a panic inside a suppose because that leaks the message that should be trapped
STD now suppresses duplicate sorry messages more correctly
sorry no longer uses panic in a supposition, but dies directly to throw the exception to the suppose's try block
STD now allows subscripts on regex variables so $x[0] isn't taken as a character class; still needs speccing

Patrick:

can we make them consistent?

Larry:

historically S05 has allowed bare arrays to mean interpolation

Patrick:

we've never had a working implementation of that

Larry:

a bare @ would be illegal

Patrick:

it's currently illegal

Larry:

you'd have to backslash it to match part of an email address
it's not like the @ alternations are a big deal one way or another
that'd be a little more consistent
I forced it to think of the sigil as $ than what it really is

Patrick:

after seeing how Jonathan et all did interpolation for quoted strings, I thought we should do the same thing in regexes

Larry:

STD now has a partial fix to prevent leakage of ::T from role signatures
unfortunately, the current fix will lose signatures of file-scoped generic roles
this probably has to do with not knowing whether we're really going to want a new pad; unfortunately we'd have to look ahead to know that currently
various other minor tweaks and bug fixes in STD and Cursor

Patrick:

mostly responding to messages and reports
should be able to get back to coding full-time and online for the next week
plan to resolve the list and closure issues with NQP and Rakudo
will answer other questions and try to keep other people productive
planning for the Rakudo Star release on June

Allison:

busy with the last week of classes
spent most of it writing a little language with PCT
it was easy to use and easy to swap the stages of PCT
I remembered what Patrick did with LOLCODE
also had a discussion of source code control systems
next week should be more productive
need to work more closely with Debian packagers to get packages into Debian

Will:

cleaning out as many deprecations in Parrot as possible
trying to improve the speed of CodeString after the immutable STRINGs merge
bundling lots of little concats helps
hope to merge in an optimization branch for that by the weekend
want to make that faster or less memory intensive
may require the use of a new StringBuilder for Parrot
hopefully will result in a faster Rakudo build

Patrick:

I've never seen CodeString take a long time
unless you run into memory problems
* discussion of the StringBuilder PMC *

still working on optimizations, particularly CodeString
looking at more PBC and PBC-building optimizations
PBC size went down dramatically and startup improved for Rakudo
should have that much faster for the 2.4 release
will poke at GC tasks starting next week

Khairul: Instrumenting Parrot

Sun, 09 May 2010 16:13:18 +0000

Having an instrumentation framework opens the doors to having many different tools that can help to diagnose problems within a piece of code. One main example of this is Valgrind. Valgrind provides an interface for making many different tools that help to diagnose and identify certain specific problems, ranging from memory leaks to multithreaded data races between threads. Furthermore, the framework is also used to provide profiling tools, such as Callgrind and Cachegrind, to determine useful information such as call graphs and execution times of functions. As such, given a good framework to work with, a whole new world of performance and error analysis tools is opened up.

In the case of Parrot, which aims to be the virtual machine for many different kinds of languages, having such a framework can assist in creating debugging tools that cater to each languages’ idiosyncrasies or be general enough to apply to all. With proper hooks in place, such a framework can also provide the ability to inspect on the inner workings of the virtual machine itself, introspection that is more in depth that what is currently offered in the various PMCs offered by Parrot.

Thus, over the summer I will be working on creating such a framework for Parrot. In general terms, I would be looking to create an interface for various user-generated tools to hook into a Parrot program. Such tools can be written in PIR, and are run in a separate interpreter from the code to instrument against. There will be three layers that I hope to be able to provide an interface for instrumenting, which are the opcode layer, PMC layer and the GC layer. Explanations for these three layers are given below:

Opcode layer
This layer allows the tools to profile and inspect the code on an opcode level. Abstractions can be made such that the tools can inspect on subroutine, class or file levels.
PMC layer
This layer allows the tools to observe the creation, deletion and accesses of the various PMCs.
GC layer
This layer allows the tools to observe the behavior of the GC, seeing when it is invoked, what gets freed and etcetera.

I intend to implement the interfaces in the order shown above and as each layer gets implemented, create tools that serve specific purposes such as call graphing, I/O monitoring and more. Tentatively, the API for the tools should look like the following:

$P0 = new ['Instrument']
$P0.’attach’(‘ops’, ‘catchall’, ’sub_callback)
$P0.’finalize’(‘finalize_sub’)
$P0.’run’(args)

In the example shown above, the ‘:main’ subroutine of the tool will perform the required initialization, creating an Instrument instance that it then registers callbacks into before executing the code to instrument against.

During this Community Bonding period, I will be prototyping the instrumenting framework as a PMC and will be looking at ways on how to insert hooks into the various PMCs and the GC subsystem in a safe manner. I would be posting more information on the project based on my prototyping efforts here and at the following blog, http://parrot.mangkok.com, and would welcome any feedback regarding this project on #parrot or through email at khairulsyamil@gmail.com.

Andrew Whitworth: The Merits of a Distributed Workflow

noreply@blogger.com (Whiteknight) — Thu, 06 May 2010 12:51:01 +0000

Long-time Parrot contributor kid51 posted a nice comment on my previous post about Git and SVN. He had some issues with the proposed distributed workflow I suggested. As my reply started to grow far too long for the stupid little blogger comment edit box, I realized I should turn it into a full blog post.

I've been pretty conflicted myself over the idea of a release manager. On one hand releases are boring which means anybody can do them and they don't require a huge time commitment. On the other hand, there is hardly a lot of "management" that goes on in a release: The release manager has some authority to declare a feature freeze and has some ability to select which revision becomes the final release; but that's it. Sure release managers also try to motivate people to perform the regular tasks like updating the NEWS and PLATFORM files, but my experience is that the release manager ends up doing the majority of those tasks herself. I really feel like we need more direction and vision than this most months, and the release manager is a good person (though not the only possible person) to do it.

Our releases, especially supported releases, have had certain taglines stemming back to PDS in 2008. The 1.0 release was famously labeled "Stable API for developers" and the 2.0 release was labeled "ready for production use", when even a cursory review shows that these two releases hardly met their respective goals. A release manager with more authority to shape that release, and more authority to shape previous releases as well might have done more to change that. A development focus, be it for weekly development between #ps meetings or for a monthly release, only matters if somebody is focusing on it and motivating the rest of the community to focus as well. That person either needs to be the architect or some other single authority (though playing cheerleader and chief motivator constantly would be quite a drain on that person) or the release manager. The benefit to using the release manager to motivate the team and shape the release is--even though it's more of a time commitment for the release manager--that we share the burden and no one person gets burnt out month after month.

A tiered development system has a number of benefits. Bleeding edge development can occur unfettered (as it happens now in branches). From there we can pull features into integration branches where we assure all the assorted changes and new additions work well together. Development releases can be cherry-picked to represent the stable baseline features that we want people to play with and rely on, and long-term supported releases would represent only those features which are tested, documented, and worthy of being included under our deprecation policy. I don't think end-users of our supported releases should ever be exposed to features marked "experimental", for instance, or any feature at all that we don't want covered by our long-term deprecation policy. If any feature included in a supported release must be deprecated and cannot be removed or fundamentally altered for at least three more months, we should be particularly careful about including new things in a supported release, and there should be some level of gatekeeper who is able to say "this thing isn't ready for prime time yet". That person, I think, should be the release manager.

Compare our system for dealing with new experimental features (Goes into trunk, maybe with mention in #ps but with very little fanfare, and is then automatically included in the next release unless somebody removes it), to a system where features are added to a development branch, vetted, tested, documented, and then pulled into a release candidate branch only when it's known by the community to pass muster. I think that's a much better system and would lead us to releases with higher overall quality and stability.

All this sort of ignores your point, which is an extremely valid one, that switching to Git represents more than just a small change in the commands that a developer types in to the console. Yes, it's a small difference to say "git commit -a" instead of saying "svn commit". Yes, it's a small personal change for me to commit locally and then push. The far bigger issues are the community workflow and even the communtity culture changes that will occur because of Git. These things don't need to change, we could lock Git down and use it exactly the way we use SVN now, but I don't think any of the Git proponents in the debate want it that way.

I would much rather plan for the changes in work flow and even build enthusiasm for them primarily than sell the small changes in tooling and then be swept up in larger culture changes that nobody is prepared for. I think we should want these changes and embrace them, in which case Git is just the necessary vehicle, and not he end in itself.

Andrew Whitworth: SVN or Git

noreply@blogger.com (Whiteknight) — Wed, 05 May 2010 22:21:46 +0000

As I mentioned in my last post, the discussion about whether we move to Git from SVN has been raging pretty hard in the past few days, and while the general opinion seems to be one in favor of a move, there certainly isn't concensus on the point just yet.

I'm in favor of Git, although I certainly wouldn't list my bent as a "passion" as some opinions in the community have been described. In Parrot world I use SVN exclusively. I could use git-svn, or I could try to use dukeleto's Git mirror on Github, but I don't. As far as I am concerned those add an additional layer of complexity, an additional abstraction that layer that I can ignore most of the time until something goes wrong. It simply isn't worth it to me to try and force a Git interface onto Parrot's SVN repository.

Let me tell a short story.

When I was a graduate student I was given a "computer" for my "desk" in my "office" that I could use to do my normal school work and also to develop my thesis. This computer was a complete beast of a performance machine. At least, it had been several years earlier when it was first purchased. By the time I got it, circa 2006, it's 12Gb hard drive and screaming 128Mb RAM weren't quite as impressive comparatively.

I was paranoid about data integrity, considering I was working on the most important academic project of my whole life on the least qualified machine on campus. So, I took some steps to help shore up my defenses. First thing, I scrounged together several other small harddrives from other dusty computer carcasses. I removed the CDROM drive and jammed four HDDs into my case. Two of them were just hanging inside because there wasn't enough space to physically mount them all. I also had a shared network drive supplied by the university, a flash drive, and a personal laptop.

I set up SVN on the computer, and backed up my work as follows: On the second drive was where my work was (the first drive was barely large enough to hold Windows XP and the other software I needed). I had a script that I used to simultaneously svn co my work to the third drive and xcopy my working directory to the fourth. I had an hourly task that would then backup my svn repo from the third drive and an xcopy of the directory on the fourth drive to the network shared drive. At the end of each day, I would xcopy my working directory to my flash drive, go home, and store a copy on my personal laptop. About once a week I would email myself a copy of my thesis paper, in hopes that if all else failed, at least Google could safeguard the all-important final deliverable.

This may all sound excessive, but by the end of the year I had lost two hard drives (one of which actually caught on fire), my flash drive got crushed, and the network drive has experienced several outages.

Story aside, the idea that there is only one copy of the entire history of the Parrot repository (or even two copies, assuming the server has a tape backup plan) is hardly reassuring to me. This is why I really like the Git idea that everybody has a complete copy of the entire history of the repository.

Let's look at things from a different angle: Git is a distributed version control system. While it certainly supports it, there is no true need for a single, central, "master" repository to work from. We could, as a community, radically change our workflow if we had Git. If we were going to use it exactly the same way as we currently use SVN, there really isn't enough of a reason to switch.

What we have now is a single master trunk, which is where most of the action takes place. People make branches, do work, merge them to trunk. Then, on the second Tuesday of every month, somebody copies trunk to a tag, calls it a release, and we continue on with our normal development.

So let's imagine a different workflow. I'll posit one example, but this certainly isn't the only choice. With Git, everybody has their own local branch where they do work and make commits. Hell, "everybody" here can be a much larger group than the committers we have now: Anybody can make a fork and do work on their own local branches. We could have something like a master integration branch where trusted committers could pull changes from the entire ecosystem. The integration branch would be more like a testbed and less like a master reference copy.

From the integration branch, a monthly release manager would cherry pick the good stable features of the integration branch into a release master branch. At the end of the cycle, the release manager's master branch would get tagged as a release, and would become the new baseline integration branch. Remaining changes from the previous integration branch could be pulled in if they were worthy or if they gained more maturity. At that point, the old integration branch could disappear, and the cycle starts again.

This idea is more complex than our current system, but it does have a few nice features. First, we can strive for higher-quality releases by keeping closer track of what ends up in the release branch. Second, we can try to get more people involved by allowing everybody to create a development branch to do "real" work, not just our designated committers. Third, the release manager has more control over releases, including being able to shift the focus on the release and be able to drive it in a way that is not possible now.

I like Git, I like the idea of it and I like the things that it makes possible. I don't dislike SVN, but it doesn't have any compelling features that make me want to stay with it. SVN is good but not great, and there's one usage pattern for SVN that works but is limiting. SVN isn't hurting Parrot, but it's not lifting us up to the next level either.

Andrew Whitworth: Been away for a while

noreply@blogger.com (Whiteknight) — Tue, 04 May 2010 23:28:25 +0000

I've been quite absent from all things Parrot in recent weeks. Work and other engagements have been keeping me occupied, and the advent of spring has motivated my family and myself to spend more time outdoors. My son, who has started teething, has the same threshold for deciding to put things into his mouth as the Perl 6 language designers have for putting features into their language: "Doesn't matter what it is, jam it in there!". But I jest.

A lot has happened in Parrot-land since I fell off the edge of the world. Parrot 2.3 was released to much fanfare, as always. Allison fixed TT #389, though her strategy was different from what chromatic and I thought was necessary. So long as it works and there are no more bugs, I'm happy to see it closed by any means. TT #389 is one of our older bugs, stemming all the way back to the first Parrot Developer Summit in 2008. It has certainly been a large pain to Rakudo developers in particular.

Bacek and chromatic also put together an implementation of immutable strings, and following the 2.3 release merged it into trunk. Immutable strings have some performance advantages, but also as was quickly found, some performance drawbacks. Specifically, string append operations can be a little bit more expensive because a new buffer needs to be allocated instead of resizing one of the argument buffers in place. A strategy to get around that, which many of our developers have been pursuing is to reduce the number of append operations.

The issue of version control software has been bubbling up to the surface again, and a long thread appeared on the mailing list yesterday after some lengthy chats on IRC. If I can continue clawing my way out of my little isolation hole and make another post this week, I would like to discuss that topic a little bit more.

GSoC students also got selected this week. I'll have plenty to post about that in the coming days and week.

I'm the release manager for 2.4 coming out on May 18th. Hopefully it's going to be a good one!

chromatic: Perl 6 Design Minutes for 28 April 2010

Sun, 02 May 2010 00:27:42 +0000

The Perl 6 design team met by phone on 28 April 2010. Larry, Allison, Jerry, Will, and chromatic attended.

Larry:

caught up on a week's worth of backlog
made a few spec tweaks
discussed them with other people
trying to make error messages more awesome in STD
working on the ability to parse the insides of character classes
STD doesn't like parsing itself recursively there
need to iron out a few things
enum names can now be variables

Allison:

Debian packages ready to ship to Debian sponsors
putting together a list of GC tasks
cleaned out the existing page, have the big things listed
trying to decide which tasks to do first
doing a lot of reading and research
my little language project is due on Monday
HLLCompiler was enormously useful
will start working on the GC stuff next week
should also start a fresh pass through the ticket queue
added a workaround for the final remaining TT #389 bug
Jonathan had a test case

Will:

tried to focus on getting Rakudo blockers removed

spent some time getting Rakudo to work with trunk
will need a Rakudo guts hacker for the last part
worked on the compact_string revamp branch with Vasily
merged now
that makes trunk about 12% faster than the 2.3.0 release
will work on a few Rakudo profiles once it works with trunk again
expect at least a 5% performance improvement there
have some other ideas, but won't do them without profiling first
came up with a scheme to reduce PBC size by coalescing strings
Peter Lobsinger is exploring that

chromatic: Perl 6 Design Minutes for 21 April 2010

Wed, 28 Apr 2010 22:20:35 +0000

The Perl 6 design team met by phone on 21 April 2010. Larry, Allison, Patrick, Will, Jerry, and chromatic attended.

Larry:

been under the weather, so didn't get much done other than keeping up with questions
S05 now allows negative quantifier ranges on reversible patterns
S02 now defines the term now to return the current instant
like rand and self, it does not parse as a function, since it never takes arguments
we now specify what kinds of math are allowed on instants and durations
improved error message on attempt to use old-school backreferences in regexes
STD now implements the now term and several other time-related names
we now allow enum names to be "constant variables" so that a class enum can declare an accessor
thinking alot about a better unification of the semantics of protos
this may also solve the current ambiguity in the meaning of postfix parens
in any case, this is for post Rakudo *

Allison:

mainly worked on packaging for Debian and Ubuntu before the release
closed TT #389, no methods in namespaces
collecting thoughts on what we need next from the GC
we've done a lot of small cleanups
now we need to solve some persistent problems
might need to make some fundamental changes, like reducing copying
coming up on my final week of classes, so lots of work there coming up

Will:

updated a spectest
minor ticket wrangling in Rakudo's RT queue

Jerry:

GSoC will make its acceptance announcements soon
expect TPF will get 10 slots

Patrick:

reviewing Rakudo's current state
made a couple of minor NQP patches
reviewing patches, especially from Moritz and Bruce Keeler
should check them in, probably with some refactorings
hope to work on the List implementation, especially laziness and context

fixed as much of line numbering as I found broken
working on branch merges
still looking at optimizations
will focus most energy this month on the sweep-free GC
hope to encourage other people to work on identified optimizations
will review Solomon Foster's Mandlebrot example, especially with regard to performance

Patrick Michaud: More Perl 6 Anti-FUD

Fri, 23 Apr 2010 23:38:22 +0000

More Perl 6 Anti-FUD here:

http://www.perlmonks.org/?node_id=836533

http://www.perlmonks.org/?node_id=836564

and thoughts on the name "Perl 6"

http://www.perlmonks.org/?node_id=836626

Patrick Michaud: A wholly inadequate reply to an Anonymous Monk

Thu, 22 Apr 2010 21:47:27 +0000

[Reposted from http://www.perlmonks.org/?node_id=836349. Normally I wouldn't repost but I'd like it to appear in my normal Perl 6 journal and rss feeds as well as on PerlMonks. --Pm]

An Anonymous Monk on perlmonks.org recently started a discussion on "The current state of Perl 6" in which he (she? they?) offers his views on various aspects of Perl 6 development. Personally, I find nearly all of Anonymous Monk's comments in the thread to be so devoid of logic and so far removed from reality/facts that it's hard to imagine providing an adequate reply to them.

Beyond that, I have other priorities in my life right now. Uppermost is that my family is going through a very difficult period at present -- I need to take care of them first. Rakudo and Perl 6 come in at a somewhat distant second to that. Responding to garbage being spouted by anonymous person(s) probably ought to not even be anywhere on my radar.

But at least one post in the thread bugs me sufficiently that I'd like to respond, even if it's an inadequate response. And I don't want my response buried in a thread somewhere, so it gets its own post.

Anonymous Monk writes:

Oh I'll tell you how you do that [write a grammar engine capable of supporting Perl 6]. It's very simple. You get people skilled for this exact task ! Those skills are acquired in universities(preferably good ones) where professors teach courses like "Formal Languages and Automata" or "Compiler theory". If you have a bunch of people who are open-source contributors but don't have the knowledge or haven't studied the right things ... [t]hey don't know it in theory and they're trying to put it in practice! (emphasis in original)

Who does Anonymous Monk think is working on Perl 6? Personally, I have a Ph.D. in Computer Science and taught university language courses for nearly 14 years. Damian Conway certainly also has a fairly strong university background (Monash University) and understands the theory and practice of language and compiler design. Jonathan Worthington has a degree from the University of Cambridge, where he specialized in compiler construction and formal languages. Larry Wall has a degree in Natural and Artificial Languages with two years of graduate work in Linguistics at U.C. Berkeley. Many of our other key contributors also have degrees and backgrounds in language theory and practice. I'd be willing to compare the academic credentials of this group against any other dynamic language team Anonymous Monk might care to postulate.

If you can't get real compiler people to use Perl6 and help with it, the average open-source rookie won't be able to deal with this.

Somehow I have trouble classifying Larry, Damian, Allison, etc. as being "not real compiler people". And they along with many other Perl 6 contributors have a lot of experience in nurturing open source developers and projects. If you feel Larry et al. are indeed unqualified to be working in the field of programming language design and implementation, you probably don't want to be using Perl at all, much less Perl 6.

People like Anonymous Monk spew lots of opinions about how long they think Perl 6 development should take, and then speculate on the reasons why it has taken longer than they estimate it should be taking. The speculations that crop up most often are things like "the people involved are clueless about language design/implementation" (see above), "the design process itself is flawed", "they're building on an inadequate toolset like Parrot", etc. For such people it's easy to toss out random thoughts about "the problems with Perl 6" without bothering to look at the obvious evidence to the contrary, as Anonymous Monk does above. Indeed, I'm often amused when people suggest that we should be doing things that we're already doing .

Returning to the topic of developing a grammar engine, which Anonymous Monk claims above as "very simple" and just needing "people skilled for this exact task", it's interesting to contrast his opinions with the actual development of the Perl 6 standard grammar (STD.pm6). I think STD.pm6 is also indicative of the challenges confronting all Perl 6 implementors. Consider:

Larry is the primary implementor for the standard grammar.
The standard grammar doesn't need a frozen specification to be implemented, because its implementation is considered part of the spec.
The implementation is built entirely on top of Perl 5 -- there are no "unproven virtual machines" involved or holding things back.

I think one could consider this to be an almost perfect situation for developing a new language implementation -- an experienced language designer/implementor working without significant external restrictions on top of an advanced programming platform like Perl 5. Yet it's been three years since Larry started working on this implementation of the standard grammar and parser, and while STD.pm6 is very powerful, it's also certainly not "finished", and has been through a number of significant refactors in its development. This says to me that the amount of time and effort involved is more likely due to the sheer audacity and ambition of what Perl 6 seeks to accomplish.

(Some will undoubtedly look at the above and knee-jerk respond with something like "If it's taken three years just to create the parser, then finishing a compiler will take far longer still and therefore Perl 6 will never see the light of day." This argument ignores the fact that other pieces are being implemented in parallel with Larry's work, that software development is not a strictly sequential process, and the obvious fact that there are already "working" Perl 6 implementations available.)

Anyone who thinks that Perl 6 is fundamentally based on traditional compiler construction techniques taught in universities frankly has no clue as to what a fundamental paradigm shift Perl 6 represents to language design and implementation. It's this fundamental change that ultimately gives Perl 6 its power, but it's also why Perl 6 development is not a trivial exercise that can be done by a few dedicated undergraduates. As TimToady on #perl6 says, "We already have lots of those kinds of languages."

Personally, I'm impressed and honored to be associated with the people who are working on Rakudo and Perl 6. I understand that people are frustrated (and even feel burned by) the long wait for something as cool as Perl 6; I share the frustration and like to think that I'm doing something constructive about it. But I also find the vast majority of suggestions, comments, and conclusions coming from Perl 6's anonymous detractors to be (1) things we've already done or are doing, (2) ungrounded in reality, (3) in direct contradiction to reasonably observable facts, or (4) attempts to discredit a product they have little interest in ever using themselves. And far too many of the comments, like the ones I've highlighted in this post, are so easily refuted with just a little bit of fact digging and common sense, it's often hard to believe anyone can be seriously making them in the first place. Yet there they are.

Returning to my original theme, I think my response here is inadequate because it leaves so many other of Anonymous Monk's claims in the thread unrefuted. I could undoubtedly spend many more hours analyzing and responding to the many fallacies and untruths in the thread, but frankly I don't believe that's the best use of my time. People such as Moritz Lenz, chromatic, Michael Schwern, and others are also writing extremely well-reasoned posts refuting the garbage, for which I'm grateful, but it's far easier for Anonymous Monk and his like to spin garbage than it is for a small number of us to clean up after it. And it does need to be cleaned up, otherwise it festers and results in even more public garbage that someone has to clean up.

I hope that this post will at least encourage more people in the Perl community to critically examine the things they hear and read regarding Perl 6, especially when coming from sources with no apparent standing or reputation within the community. And maybe a few more people will even assist in publicly refuting the garbage ("many hands make light work"), so that the sloppy thinking, analysis, and dialogue that people like Anonymous Monk post doesn't spread to infect all of Perl.

P.S.: Some may reasonably conclude from what I've written above that Perl 6 is somehow "aiming too high", that our goals should be scaled back to make something ready "right now". I have two responses to this: (1) we are making things ready 'right now', just grab any of the available packages and start working and reporting bugs, and (2) there are already 'scaled back' versions of Perl 6 appearing and being used, such as NQP or even the components that execute the standard grammar. Some of these other projects, like NQP, are being used for "real" programs and applications today; they're not simply theoretical exercises or research projects.

Others often claim that all this effort on Perl 6 would be better spent on improving Perl 5. In my experience, those of us working on Perl 6 have absolutely no qualms with seeing Perl 5 improve and continue to grow -- we welcome it. Indeed, many of the features appearing in Perl 5 today come directly from ideas and things first attempted in Perl 6, and we're genuinely happy to see that. But just because Perl 6 developers also like Perl 5 doesn't mean that doing Perl 5 core development is interesting to us, or that (in my case at least) we'd even be qualified to do Perl 5 core changes. We aren't commodity programmers that are interested in simply being unplugged from one project and into some other project that others think is more worthwhile. Personally, I'd prefer to see people who are really into Perl 5 improvements continue to work to make them happen, and that the surrounding ecosystem continue to evolve to enable that to happen more easily for more people. Indeed, this is my wish for all open-source projects, even the ones I don't find interesting or otherwise disagree with.

chromatic: Perl 6 Design Minutes for 14 April 2010

Wed, 21 Apr 2010 04:09:26 +0000

The Perl 6 design team met by phone on 14 April 2010. Larry, Allison, Patrick, Will, and chromatic attended.

Larry:

on p6l, did a bit of bikeshed paint removal with regard to hyphens vs underscores
S02 now explicitly disallows both whitespace and unspace in top level of an interpolation
per spec change, when STD is parsing an interpolation inside quotes and looking for a possible postfix, we now presume that a backslash belongs to the quotes and is not an unspace
in the src/perl6 directory, renamed all Perl 6 .pm files to .pm6 to avoid confusion
this was necessary because the implementation of STD translates Perl 6 back to the corresponding Perl 5
the ambiguity was causing problems with tools such as NYTProf
Cursor.pmc now prefers .pm6 over .pm in any particular directory when searching for Perl 6 code
as usual lately, most of my hacking work was in improving the human interface of the parser
STD now distinguishes two final messages: "Parse failed" vs "Check failed"
STD now warns on attempts to smartmatch with True or False
STD now distinguishes continuable-but-fatal "sorry" messages from immediately fatal "panic" messages
sorry messages will eventually fail at check time
changed many of STD's semantic errors to use sorry messages when the parse state is not affected
modified moritz++'s conflict marker patch to be more like the Clang compiler's behavior
conflict markers now emit a "sorry" message and continues parsing one side of the conflict
also fixed a buglet that prevented it from processing the conflict marker if first thing in the file
while fixing the vws (vertical white space) rule for that, also changed it so that extra lines are now eaten with \V*\v for consistency
it had be using \N*\v
gimme5 now supports pointing to both ends of missing goal message
STD's "Couldn't find final ..." messages now use that capability to point to both ends of the error
standard quotes now also use the ~ compositor to set the goal and get that behavior
STD will now dwim <<op>> ("Texas hypers") better even if op contains angles
suppressed confusing backtracking on ~<< that produced a misleading quotewords error
some other patches
CORE.setting now recognizes the 'note' function
gimme5 now translates note to print STDERR
cleaned up some unneeded locmesses
Actions.pm now handles prefix metaops without spewing spurious yaml dumps

Allison:

worked on TT #389
the actual fix was about two lines
spent a lot of time fixing tests around it
didn't like the original two-line fix
fixed it in IMCC by passing along the :nsentry flag
NQP-rx still depends on that feature
I understood from Patrick that NQP-rx doesn't need that feature
don't want to launch that before the 2.3 release
worked on a lot of smaller issues
worked on the Parrot Developer Virtual Summit
will talk about some process changes more, as there are details to work out
will work on GC as the next priority
useful for Rakudo in general and Parrot concurrency

Patrick:

catching up on mail and tickets
should get back to coding in the next couple of days

worked on the immutable strings branch
need a couple of changes in the Rakudo binder
now it's time to convince everyone else it's a worthwhile design change
going to work on bugfixes
will try to land the constant string cache
otherwise, added some optimizations

Will:

worked on Partcl
fixed a Parrot bug that broke Rakudo
does Rakudo need TT #389 in 2.3?

Jonathan Worthington: Perl 6 talk in Malmö

Tue, 20 Apr 2010 09:55:36 +0000

Recently I moved to Sweden to join startup company Edument. They are, happily, very open to and supportive of my involvement in the Perl 6 project, and are organizing and sponsoring me to give a talk on Perl - both 5 and 6 for the Dataföreningen in Malmö on Tuesday 27th April. I believe this is only open to those who are members of the user group; if you are, feel free to come along though.

Jonathan Leto: PL/Parrot Flies

Tue, 20 Apr 2010 02:55:57 +0000

PL/Parrot recently started passing all tests and marshalling three basic data types of integers, floats and strings, between PostgreSQL and Parrot Virtual Machine.

One of the important next steps for PL/Parrot is to improve the security subsystem, which will require changes to Parrot core. Parrot has only a high-level Parrot Developer Doc (PDD) that describes how security should work, but still lacks an implementation. Thus, PL/Parrot finds itself in the situation of being able to drive the development of security features in Parrot. This is very exciting and I am researching various hardening solutions. Various ideas have been thrown around, but removing or replacing certain opcodes at the lowest level is the most secure solution. How to do this, properly at run-time, is the fun part.

I will be talking about PL/Parrot at PGCon 2010, which will be very exciting. I am hoping to meet many people in the PostgreSQL community and show some PostgreSQL internals hackers why hacking on PL/Parrot is so fun and so important.

PL/Parrot is so important because PL's are hard to write and maintain. My vision is that most of the hard work goes into PL/Parrot, and then languages built on top of Parrot (HLL's), will have a very simple time piggy-backing on top of that infrastructure. Currently, only PL/PIR exists. Stored procedures can be written in PIR. The next steps will be getting a HLL working on PL/Parrot, such as NotQuitePerl (NQP) or Rakudo Perl 6. If you want to get your favorite HLL working on PL/Parrot, I surely won't stop you.

There are also many benefits to writing stored procedures in PIR. PIR will have the closest possible speed to C than any HLL run on Parrot, so if speed is your main concern, you will want PL/PIR. Parrot has a garbage collector, so you don't have to do memory management in PL/PIR. This prevents crashing the server instance due to double-free errors and fun stuff like that. PIR is also platform-independent: no more need to figure out how to get your stored procedures written in C compiling on a new platform.

If you have tuits or ideas for implementing PL/NQP or PL/Rakudo or any other comments about PL/Parrot, I would love to hear them. Feel free to fork the github repo, join us in #plparrot on irc.freenode.net or on our mailing list. We also need a website and mascot. There are many ways to help, but patches are always welcome :)