Author: bruinbrown

Simplifying MBrace Deployment

If you’ve used MBrace then you’ll know that it’s a really powerful tool for quickly performing data science and data engineering related tasks. It’s always had a great integration with Azure, however, it’s typically relied on the use of some legacy Azure services, notably in the form of Cloud Services. In recent weeks at Compositional IT, we’ve been looking at whether we can modernise some of the Azure integration by moving the Cloud Services over to the newer App Service, which simplifies many of the challenges you’re likely to have when developing applications by providing simplified stories for deployment, monitoring and development. It’s also fully supported by the Azure Resource Manager tooling which allows us to configure our Azure services in a declarative manner.

After getting MBrace running as a webjob, we also managed to create an Azure Resource Manager template which allows us to create all of the resources required by an MBrace cluster and then deploy the worker onto the cluster. Currently, it relies on a few PowerShell commands to submit the deployment, but there’s plenty of other options including the Azure CLI and the Azure management libraries. However, there’s also the option of creating a Deploy to Azure button allowing us to deploy an MBrace cluster in as little as 3 clicks.

If you now browse to https://github.com/isaacabraham/mbrace-arm you’ll see a shiny Deploy to Azure button. Click on that and you’ll get directed to the Azure login page. Login to your Azure account (if you don’t have one yet, sign up and you’ll get a $200 free credit for your first month). Create a new resource group (it’ll make it a lot easier to kill the cluster when you’re done) and specify the machine size and count you want. Click next and next again and the deployment starts and then you’ll have an MBrace cluster available for use. During my testing it’s taken about 90 seconds on average to spin up a 10 node MBrace cluster which is really fast.

deploytoazuredeploy

Unfortunately, the Deploy to Azure website doesn’t list out the outputs of an Azure Resource Manager template, which means we have to go into the Azure portal to retrieve connection strings for Azure Storage and Azure Service Bus, fortunately clicking the Manage your resources link takes you directly to the page in the Azure Portal for our Azure Resource Group. Click there and then grab the connection strings for your Service Bus account and Storage account.

Now open up your editor of choice, here’s mine below, VS Code, write up a quick script to do some processing in MBrace, execute it in F# Interactive and it gets shipped up to your shiny new MBrace cluster hosted in the cloud.

mbraceazure

Advertisements

Reactive Applications with Akka.Net available on MEAP

The book that I’ve been working on for the past several months is now available on MEAP directly from the Manning website and is available at https://manning.com/brown2 If you’re interested in Akka.Net, the actor model or the reactive manifesto then it’s definitely worth checking out. Currently the first 3 chapters are available with the first chapter available for free, however, there is plenty more to come over the next few months in the run up to the print release.

The Internet of Things at Scale with F# and Actors

This post is in the FSAdvent series of blog posts. See more at https://sergeytihon.wordpress.com/2014/11/24/f-advent-calendar-in-english-2014/

It’s Christmas time which means only one thing – It’s socially acceptable to have enough lights flashing at all hours of the day to make the runway staff at Heathrow jealous. But how do I go about managing potentially thousands of chains of fairy lights all attached to my house?

(more…)

Running F# on a PlayStation Vita

At the progressive F# hackathon yesterday I set myself the task of getting something written in F# running on a PlayStation Vita. It’s something that I’ve wanted to do for a while now and thought there was no better time to try it. Here’s the results running on an actual device which is always a bit more impressive than just running it within an emulator.

(more…)

Creating an infinite type system with F#’s type providers

Good things always come to an end, right? Not necessarily. By now we’ve all seen just how awesome type providers are and you might have even made one or you’re wanting to make one. One of the cool things about them though is that you can start to make type systems which never end. For example in the Brainfuck type provider you can create programs of any length by accessing types which belong to types so you can create a program like so.

Brainfuck.TypeProvider.``+``.``+``.``.``.ConsoleOuput

Now in this case, everything up until the ConsoleOutput is a type and ConsoleOuput is just a property on the final type. We’ll look at how you’d go about implementing this small section of the type provider, so I’ll assume you already know the basics of type providers. If you don’t and want to learn check out this post by Michael Newton here and maybe even visit F# London in May here. If you want to follow along at home and implement the steps as we go through then you can download the skeleton project from here.

The key thing that we’re going to be looking at here is how to create the same series of types onto each other and how to get some state flowing through. We’ll be making a type provider which allows us to create words through a type system.

So you’ll have the shell of your type provider with the base type that it provides and now you want to provide a lot more. The first thing to do is to look at the ProvidedType.AddMemberDelayed() method. This method takes a function as a parameter which is of the form unit -> MemberInfo. A MemberInfo is what all of our Provided_ types inherit from so we can add pretty much anything. In our case we’ll be using a combination of ProvidedType and ProvidedProperty.

We’ll create a function now which matches the definition then, so how’s about something like the following.

let rec createTypes () =
  let ty = ProvidedTypeDefinition("A", None)
  ty.AddMemberDelayed(createTypes)
  ty

let rootType = ProvidedTypeDefinition(asm, ns, "StartHere", None)
do rootType.AddMemberDelayed(createTypes)

And then in our provided type we’ll call AddMemberDelayed passing in a reference to our function. If we test that now we’ll see that we have a type provider which won’t stop providing a child type with the same name. Let’s now switch to using the AddMembersDelayed, AddMemberDelayed is great if we only want to add 1 thing at a time, but we want to add the alphabet each time. It’s pretty much the same concept except instead of creating a function like unit -> MemberInfo we create a function like unit -> MemberInfo list.

let rec createTypes () =
  alphabet
  |> Seq.map (fun t -> let ty = ProvidedTypeDefinition((t |> string), None)
                       ty.AddMembersDelayed(createTypes)
                       ty)
  |> Seq.toList

do rootType.AddMembersDelayed(createTypes)

So now we have the alphabet available through our type provider. It would be nice if we could now pass some state through and then access our word through a property. We’ll modify our function definition to be of string -> MemberInfo list but now it doesn’t match the required function definition. We’ll store our word so far in a closure and pass that closure to the AddMembersDelayed method instead. It’ll look something like this.

let rec createTypes wordSoFar =
  alphabet
  |> Seq.map (fun t -> let newChar = t |> string
                       let newWord = wordSoFar + newChar
                       let ty = ProvidedTypeDefinition(newChar, None)
                       ty.AddMembersDelayed(fun () -> createTypes newWord)
                       ty)
  |> Seq.toList

do rootType.AddMembersDelayed(fun () -> createTypes "")

And we can then easily add in a property to access the word so far like this.

let rec createTypes wordSoFar =
  alphabet
  |> Seq.map (fun t -> let newChar = t |> string
                       let newWord = wordSoFar + newChar
                       let ty = ProvidedTypeDefinition(newChar, None)
                       ty.AddMembersDelayed(fun () -> createTypes newWord)
                       let wordProp = ProvidedProperty("Word",
                                                       typeof<string>,
                                                       IsStatic=true,
                                                       GetterCode = fun args -> <@@ newWord @@>)
                       ty.AddMember(wordProp)
                       ty)
  |> Seq.toList

And now you can access the words like so:

Word.TypeProvider.StartHere.T.E.S.T.Word // returns "TEST"

Type providers can look really complex at first but they’re actually really easy to work with and make something pretty cool pretty quickly. Also if you wanted to expand this further you could load in a word list from a static parameter, then parse that into a trie and only allow the type system to create words from that word list. (Here‘s a hint from Chris Smith if you wanted to try that)

A quick COM and F# tip

COM’s got a few quirky nuances. I spent this evening battling a problem in which a C# wrapper had been written around a COM API. This C# wrapper defined an interface called IMediaDet, it also declared a class called MediaDet which should inherit from the interface but doesn’t. Instead the interface and the class shared the same GUID. So I had a C# example which looked like this.

[Guid("65BD0710-24D2-4ff7-9324-ED2E5D3ABAFA"),...]
interface IMediaDet {}

[Guid("65BD0710-24D2-4ff7-9324-ED2E5D3ABAFA"),...]
class MediaDet {}

var mediaDet = new MediaDet() as IMediaDet;

At first you’d think that mediaDet should always be null but it wasn’t, it was actually being assigned due to the underlying COM implementation. Unfortunately in F# something like the above statement isn’t allowed and rightly so, it doesn’t make sense. Thankfully Ross McKinlay was available on Twitter to help and suggested the following

let mediaDet : IMediaDet = unbox(new MediaDet())

The point of the unbox is to avoid the type checker. For the most part though, you should always listen to the compiler and avoid doing things like this as it stops you trying to do something stupid like use COM.