Golang Interface for types Implementing the String() func

Earlier today my forgetful nature struck me again and I was trying to remember what the name is of the built-in Go interface for types that implement the String() method / func. It’s the Stringer interface in the fmt package. I figured I better make a quick post here so that I’ve got an easy place to check next time I forget. Truth is, now that I’ve written a post about it I probably won’t forget the next time. Year 12 chemistry exam all over again!

The source for the interface can be found at https://golang.org/src/fmt/print.go#L62.

SharePoint Event Receiver Tips and Gotcha’s

Over the last decade of being in SharePoint, I keep running into various “gotcha’s” with event receivers that aren’t always obvious. Even though SharePoint server side development is going the way of the dodo, I imagine I’m still going to be writing the odd event receiver for client’s for another few years so here’s a list of things I need to remember for next time.

  • When you’re doing a list item added event receiver, use the ListItem on the SPItemEventProperties object you’re passed as the ListItemUniqueId won’t resolve anything if you try to access it directly from the list.
  • If your event receiver isn’t being fired even though everything appears to be deployed correctly, check that you are actually referencing the correct event receiver in your Elements.xml file. Too many times I move a class from one assembly to another, change a class name etc. and forget to update this file. Often no error messages are logged in ULS so it looks everything is fine.
  • This is sort of a “best practice” which you should be doing anyway. Make sure you separate out the innards of your event receiver into a different class that can be easily mocked. Things may work in development and test, but when you move stuff into staging or prod, you shouldn’t have any development tools available to you. In these scenarios, being able to run diagnostics from a PowerShell command line is very helpful and if you have easily “testable” classes, you can instantiate these from PowerShell.

go-func (yourself): Making Golang a Little More Functional Part 1

TL;DR: I’ve written a command-line utility called go-func that is intended to be used with go-generate declarations to make it easier to use functional programming paradigms in Go. v0.1 only has the feature to generate map/filter/reduce functions for a slice. You’ll fine the source code at https://github.com/chris-tomich/go-func.

Functional Bandwagon

Like so many other programmers at the moment, I’ve jumped on the “functional” bandwagon and started re-learning Haskell (haven’t touched it since uni) and learning/playing with Clojure. It’s not because I have any intention of using these languages for my current project/start-up, Oystr (https://oystr.co), but more to improve my skills as a programmer in the imperative languages I use daily like Go and C#.

One of the things I’ve really begun to like in Haskell is it’s list comprehensions and the idea of the map/filter/reduce paradigm. Recently I found myself needing it to simplify the code for the frameworks I’m building for Oystr. As Go doesn’t have generics, I found myself copying and pasting a whole heap of code and decided this would be a perfect job for go-generate. I figured I’d look around first for something that did it but other than some tools to emulate generics (I’m not sure this needs to be done – even as a C# developer since .NET 1.0 who has been using generics since they introduced them in .NET 2.0) I didn’t find anything so thought I’d write it myself.

In this post I’ll introduce the map/filter/reduce paradigm, for those that may not be familiar with it. I’ll also give an example and explanations for using go-func with your project and what I plan to do next with it.

Business As Usual – Loops

Normally in an imperative language like Go or C#, to perform the same operation on a list of items we would iterate through them using a loop like a for, foreach, while etc. Through each iteration of the loop we would perform the necessary operations. Ultimately it’s up to us to define the control structures (ifs, switches, gotos, breaks, continues etc.) that will help us to get the job done. To demonstrate a common example of this process, I’ve created the following sample where we do the following tasks –

  1. Iterate through a list of words we’re given.
  2. Create two sentences with that list of words –
    • A sentence with spaces in between each word.
    • A sentence with spaces in between each word with a number of different words having been filtered out.
func StandardLoopExample(bagOfWords e.StringList) (string, string) {
    sentence1, sentence2 := "", ""

    for _, word := range bagOfWords {
        sentence1 = sentence1 + word + " "

        if word != Filter1 && word != Filter2 && word != Filter3 && word != Filter4 {
            sentence2 = sentence2 + word + " "
        }
    }

    return sentence1, sentence2
}

A Better Cleaner Way? – Map / Filter / Reduce

At first I wrote “better” but I crossed this out because I realised that my personal preference for map / filter / reduce over loops is that I personally find them much cleaner and thus easier to read. Before they become cleaner/easier to read for others, you first need to know what is a map, filter, and reduce.

Map (or Apply)

The theory behind a map operation, also sometimes known as “apply”, is that a function is “mapped”/”applied” to each element in a list. For instance, if I had the following list of items that I named “myList”;

[10, 20, 30, 40, 50]

and I applied the following function to the first element in “myList”;

func Inc(x int) int {
    return x + 1
}

it would mean that I would be increasing the 1st element (the value 10) by 1 meaning I would get the value 11.

If I applied this function to the entire list, then I would increment every element in the list by 1 and the resulting list would be;

[11, 21, 31, 41, 51]

I could then say I’ve “mapped/applied the function Inc to myList”.

Filter

The purpose of a filter operation is to remove elements according to a given predicate. In our previous example we had the following list that we called “myList”;

[11, 21, 31, 41, 51]

If we wanted to remove all values greater than 40 we may filter “myList” with the following predicate;

func RightSize(x int) bool {
    return x <= 40
}

If we used this predicate to filter “myList”, we would end up with the following list being returned to us;

[11, 21, 31]

Thus we could say we’ve “filtered myList using the function RightSize”.

Reduce

The reduction operation is a summary operation and will perform an operation that results in a single result being returned that is a combination of the elements in a list. This is best explained using an example so, with our mapped and filtered “myList” from our previous example, we’ll take all the elements, double them and then add them together using the following operation;

func Summarize(prev int, current int) int {
    return (2 * current) + prev
 }

The result of this operation will be the integer value 126. The way this works is like so;

  1. 1st time running the method, Summarize(0, 11) is called. This returns 22.
  2. 2nd time running the method, Summarize(22, 21) is called. This returns 64.
  3. 3rd (and final) time running the method, Summarize(64, 31) is called. This returns the final value which is 126.

Unlike a map and filter operation, a reduce operation does not necessarily produce another list but instead produces a result based upon all the elements in the list.

You will have noticed that in running through the reduction process, I introduced the “0” value in the 1st time running the Summarize method. This is called the initialisation value and is necessary for any reduction operation as the first element doesn’t have anything before it. This normally would be something inert like 0, an empty list, nil, etc. but doesn’t necessarily have to be and really depends on your reduction scenario.

“Map / Filter / Reduce”-based StringList Example

So with this understanding of what a map / filter / reduce operation is, if we re-visit our loop example we’d end up with something that looks like the following.

func MapFilterReduceExample(bagOfWords e.StringList) (string, string) {
    query1 := bagOfWords.Map(func (word string) string { return word + " " })

    sentence1 := query1.Reduce("", func(init string, word string) string { return init + word })

    query2 := query1.Filter(func (word string) bool { return word != Filter1 })
    query2 = query2.Filter(func (word string) bool { return word != Filter2 })
    query2 = query2.Filter(func (word string) bool { return word != Filter3 })
    query2 = query2.Filter(func (word string) bool { return word != Filter4 })

    sentence2 := query2.Reduce("", func(init string, word string) string { return init + word })

    return sentence1, sentence2
}

To me this a lot easier to read than what we had before. We can see that we create our first sentence straight after adding spaces to the end of every word. We then add filters for the words we want to remove and then generate our second sentence.

The only issue that remains is the fact that our type “e.StringList” needs to have Map(), Filter(), and Reduce() functions implemented for it. If we had to implement these every single time, this would be a pretty tiresome process and so I’ve developed go-func to do this for us.

Introducing ‘go-func’

You can get go-func at https://github.com/chris-tomich/go-func. Either clone the package or use go get and then build the command line utility with the following command (change the directory structure to fit your go-workspace).

go build -o $GOPATH/bin/go-func $GOPATH/src/github.com/chris-tomich/go-func/cmd/go_func.go

go-func is a simple tool that you run via “go generate”. If you’ve never used go generate before, it’s a tool that will search for comments beginning with “//go:generate” in your source code and then run the commands straight after that declaration. When it was introduced in Go 1.4, they included an example utility called “stringer” which generated boiler plate code to output a friendly string name for enumeration constants. go-func (at this early stage) does something similar and will generate map / filter / reduce operations for collections that you define. Below is an example of a custom collection type declaration and the associated go generate command to generate the code.

package string_list_example

//go:generate /home/chris/go-workspace/bin/go-func -singular=string -collection=StringList
type StringList []string

As you can see in the example above, go-func takes in 2 parameters at this stage (I’ll make it a little smarter in the future). It takes in –

  1. -singular = This is the type that the slice holds. In this example it’s just a “string”.
  2. -collection = This is the name of your type that is based upon a slice type. In this example this is “StringList”.

And that’s it! When you run go-generate in the directory of your source file, the following code will be generated (with your types in place of “string” and “StringList” of course). The name of the file will be in the format “<original file name>_<your type>_generated.go”.

package string_list_example
// The following code was generated by the go-func utility. Please run go generate on the file from which this
// generation originated from. It is not recommended to modify this file as future go generate calls are likely
// to overwrite this file.

type StringListMapFunc func (string) string
type StringListFilterFunc func (string) bool
type StringListReduceFunc func (string, string) string

func (l StringList) Map(fn StringListMapFunc) StringListQuery {
    query := NewStringListQuery(l)

    query = query.Map(fn)

    return query
}

func (l StringList) Filter(fn StringListFilterFunc) StringListQuery {
    query := NewStringListQuery(l)

    query = query.Filter(fn)

    return query
}

func (l StringList) Reduce(init string, fn StringListReduceFunc) string {
    query := NewStringListQuery(l)

    return query.Reduce(init, fn)
}

type StringListOperation struct {
    Map       StringListMapFunc
    Filter    StringListFilterFunc
    Reduce    StringListReduceFunc
    Operation string
}

type StringListQuery struct {
    list StringList
    operations []*StringListOperation
}

func NewStringListQuery(list StringList) StringListQuery {
    query := StringListQuery{
        list: list,
        operations: []*StringListOperation{},
    }

    return query
}

func (l StringListQuery) Map(fn StringListMapFunc) StringListQuery {
    op := &StringListOperation{}
    op.Map = fn
    op.Operation = "Map"

    l.operations = append(l.operations, op)

    return l
}

func (l StringListQuery) Filter(fn StringListFilterFunc) StringListQuery {
    op := &StringListOperation{}
    op.Filter = fn
    op.Operation = "Filter"

    l.operations = append(l.operations, op)

    return l
}

func (l StringListQuery) Reduce(init string, fn StringListReduceFunc) string {
    reduction := init

    for _, elem := range l.list {
        value := elem
        isIncluded := true

        for _, op := range l.operations {
            switch (op.Operation) {
            case "Map":
                value = op.Map(value)
            case "Filter":
                isIncluded = isIncluded && op.Filter(elem)

                if !isIncluded {
                    break
                }
            }
        }

        if isIncluded {
            reduction = fn(reduction, value)
        }
    }

    return reduction
}

func (l StringListQuery) ExecAll() StringList {
    newStringList := StringList{}

    for _, elem := range l.list {
        value := elem
        isIncluded := true

        for _, op := range l.operations {
            switch (op.Operation) {
            case "Map":
                value = op.Map(value)
            case "Filter":
                isIncluded = isIncluded && op.Filter(elem)

                if !isIncluded {
                    break
                }
            }
        }

        if isIncluded {
            newStringList = append(newStringList, value)
        }
    }

    return newStringList
}

At the moment it is a little limited (you can only do slices) but I’ll update this shortly so that you can use maps and arrays as well. In the future I intend to create a number of code generation operations within go-func to help developers make their Go code slightly more “functional” but for now I only intend to have the map / filter / reduce feature as it satisfies my immediate needs.

Why Use ‘go generate’?

My personal preference for using map / filter / reduce operations is that they result in what I feel is cleaner code. If I was using C# then development of this basic framework isn’t an issue (in fact it’s already been implemented through LINQ and the associated extension methods) as I have support for generics in C# which would make it easy to create either a simple generic collection with map / filter / reduce arguments on them or map / filter / reduce generic extension methods that are applied to any IEnumerable or IEnumerable<>.

Unfortunately (or fortunately depending on who you speak to) Go doesn’t have generics and so what’s possible with C# is not possible with Go. It’s a discussion for another day but the long story short, I appreciate the reasoning and don’t actually believe that introducing generics into Go will make it even 2 times better than it is now and so I decided to look for a more “Go-ish” way. One thing we are provided in Go (and what seems to be a little under utilised) is code generation with go generate. It’s not a fantastic tool (honestly it’s just a glorified command runner based upon your source) but if the core Go team are promoting it’s use, I’m happy to oblige.

If you boil down what generics do for us, they allow us to introduce types as variables and to retain strong typing rather than being forced to resort to ignoring types with interface{} or using reflection. I’ve seen a number of implementations online of functional-styled libraries for Go and quite a few of them use interface{}. Being a daily user of JavaScript, I’ve felt the pain of not having access to strong static typing. In fact, my choice to move Oystr from Node.JS to Go was (almost) entirely for this reason! When I see these libraries make such heavy (and what I feel is unnecessary use) of interface{}, it makes me want to cry tears of pain. As go-func generates statically typed methods for your collection type, we get to keep all the benefits of strong static typing, get to work in a functional manner, and we don’t have to manually repeat code! That’s not just a win-win, that’s a win-win-win!!!!

What’s Next? – Benchmarking

I have done some basic benchmarking to compare how my functional version compares with a loop and those compare with versions that are concurrent and it isn’t terrible but it isn’t fantastic. Below is a screen capture of the results.

go-func Initial Benchmarks

The times aren’t that different but the really bad part is the fact that the number of memory allocations are more than double! I have done some testing and you’ll be glad to know that is more related to the string operations being done.

In my next post in this series I’ll address the ways to keep the impacts to an absolute minimum. I’ll provide some tips on how I was able to get this down considerably and some ways that using this kind of functional paradigm can make things even faster as introducing concurrency into existing code is made easier.

Cryptic Error with Custom PowerShell Cmdlet

I was building a PowerShell cmdlet aimed at SharePoint 2013 earlier tonight and after deploying my WSP I was getting the following cryptic error.

Cannot process argument because the value of argument "implementingType" is null. Change the value of argument "implementingType" to a non-null value.

The error claimed to be a result of attempting to load the help XML file although this turned out to be a bit of a red-herring in my case.

Searching around online didn’t really show much so I thought I’d leave a quick note here about what eventually solved it for me. It turned out that in my registration XML (found under hive/CONFIG/POWERSHELL/Registration) I had declared the wrong assembly. By default my registration XML was using a $SharePoint.Project.AssemblyFullName$ token which would be fine in certain circumstances but as my assembly was being built in a different project with a different key, this was being filled in with the wrong details. After changing the <ps:Assembly Name=”$SharePoint.Project.AssemblyFullName$”> tag to declare the correct assembly, I stopped getting the above error.

Make Function Keys Default on Apple Keyboard in Ubuntu 16.04 Xenial Xerus

Even though I use a PC laptop, I have an Apple keyboard because I like how slim it is (means I don’t have to lift my palms too high off the table when typing). The problem I’ve had, since I started using Ubuntu heavily, is that Ubuntu will bind all the function keys to their related media functions. I’m sure that’s really intuitive if you’re not doing development – I am, and having to press fn+F5 all the time is a slight annoyance. After a little search I think I’ve found the best way to do it is detailed at the following link.

https://help.ubuntu.com/community/AppleKeyboard#Ubuntu_9.04_to_12.04_LTS_.28Precise_Pangolin.29

The method I’m using is the first method listed which involves adding a hid_apple.conf file to the /etc/modprobe.d directory with the contents

options hid_apple fnmode=2

Previously I was using a different method that involved adding the line “echo 2 | sudo tee /sys/module/hid_apple/parameters/fnmode” to rc.local but for some reason on my laptop this doesn’t always appear to work as my Apple keyboard hasn’t yet been detected/loaded when this particular line is executed.

Getting all folders in a SharePoint list

This blog post is no great revelation, it’s just a reminder to the future me about what SPList.Folders shows. I’ve forgotten the exact answer and subsequently tested it with PowerShell so many times but my memory for these little details is horrible. So the question I found myself asking, and what I tested in PowerShell, was whether SPList.Folders shows you ALL the folders, including sub-folders. The answer is yes.

Using Azure-Provided DNS

I came across this slight annoyance today. I have bunch of development servers on Azure with an AD deployment, SQL, etc. and I was wanting to modify the built-in Azure-provided DNS / Azure-managed DNS records. I’ve since found out at https://azure.microsoft.com/en-us/documentation/articles/virtual-networks-name-resolution-for-vms-and-role-instances/#azure-provided-name-resolution that you can’t modify the records for the built-in DNS and you need to setup your own DNS if you want this level of control. Managing my own, internal only, DNS just seems a bit unnecessary for a development environment so I’ll just modify the hosts file for now.

After a bit more investigation I’ve found out that I can’t seem to add to the hosts file as the servers appear to be ignoring it for Domain controller name resolution. I’m not sure if I’ve ever attempted using hosts file name resolution for joining to a domain controller so unsure if this is a result of a Windows Security Update or just a general check done by Windows anyway. Regardless, it means I’m going to need to set a custom DNS server in Azure which can be done under Virtual Networks -> (Name of Virtual Network) -> DNS servers -> Custom DNS -> Primary DNS Server.

Sharing an ExpressJS / Connect / PassportJS Session with Golang – Part 1

TL;DR – I’ve confirmed the process to extract and verify the session ID and accompanying signature from the authentication cookies set by ExpressJS and Connect. For a “no-sh*t-sherlock” version of this article (bare bones research) check out my Oystr map here.

ConnectSid

 

I’ve begun the process of writing some of my microservices for my start-up, Oystr, in Golang and I have encountered a slight issue with authentication. My current authentication is all handled by PassportJS in a Node.JS and ExpressJS stack meaning I’m going to have to somehow share the the session information between the two different platforms. I’m a one man team at the moment so re-writing everything in Golang is out of the question and I’m just battling too much with the lack of type checking in JavaScript to be able to continue completely on Node.JS. The solution for me is to be able to access the cookie information saved by ExpressJS and then pull the user information from Redis into Golang.

The Connect.SID Cookie

So this is what I know from previous experience with setting up PassportJS – I know that it uses the information stored in the “connect.sid” cookie value to find the corresponding user IDs in the Redis session store I’m using to persist sessions. Verifying this was the easy bit and was just a matter of opening up Chrome’s developer tools (F12) and then looking at the cookies. Sure enough I found it as shown below.

connect.sid Cookie

Looking at the value, it was clear to see that it was URL encoded so once I decoded this I got something of the form “s:sessionid.signature”. At this point you could say “now that I have the session ID I can go look for that ID in the Redis session store… my work here is done”, smack your hands and walk away… BUT if you do that you’re opening yourself up to a massive security hole. Someone could easily just try brute forcing the session ID in the hope that they hit the session ID matching a logged in user. To really make this secure you need to be able to verify the signature is correct for the given session ID.

Verifying the Signature

So after a bit of research (go look at my research map in Oystr, here, if you want to see my sources for yourself) I figured out the process for creating the signature. I decided that initially I’d prove the process works with the sample I took from my browser using commands on the Linux command line as I hate writing code for a process that doesn’t actually work. Writing code for something that isn’t even correct is a million times more annoying than having to continuously check your typing in JavaScript (and if you read my last post https://mymemorysucks.wordpress.com/2016/05/19/why-javascript-is-giving-me-depression/ you’ll realise how big of annoyance that is to me).

The basic process to create the signature from the session ID is –

  1. Create a SHA256 HMAC signature based upon the secret you declare in your ExpressJS middle-ware setup code. The code itself should look something like app.use(session({secret: ‘my secret’})).
  2. Creating the signature will generate a long hexadecimal value like “6F0AD0BFEE7D4B478AFED096E03CD80A”. You need to convert this to base64 and that value should match the signature stored in the “connect.sid” cookie.

It’s only two steps so it’s pretty basic. After some searching around I figured out you could do the conversion from the Linux command line using the “openssl”, “xxd”, and “base64” commands.

  1. You use “openssl” to first create the signature for the session ID.
  2. You use “xxd” to convert the signature, which is actually just a “string” of numbers and letters, into the actual bytes represented by the hexadecimal characters.
  3. You use “base64” to convert the bytes from “xxd” into a base64 format.

You can do all of that pretty easily with the following commands –

  1. echo -n <session id> | openssl sha256 -hmac <my secret>

    This dumps out a string like “6F0AD0BFEE7D4B478AFED096E03CD80A”. This is the <signature> to be used in the next set of commands.

  2. echo -n <signature> | xxd -r -p | base64

After that second command, assuming you did all that correctly and used the correct values, you should have a base64 signature that matches exactly the signature part of your example connect.sid cookie.

What next?

So now that I can calculate the correct signature, I can make sure that the session ID I’m being supplied by the user hasn’t been tampered with and that they are unlikely to be attempting a brute force hack. In my part 2 post I’ll give details and code as to how I made use of all of this in developing a Golang package to share the sessions and user profiles between my two platforms.

Why JavaScript is giving me depression…

Disclaimer: This is not a serious post. It’s intended to be a little tongue-in-cheek. Before you try and start a war in the comments section for either the “JS sucks” camp or the “JS is the second coming” camp, I won’t be approving any of those comments as THIS POST ISN’T SERIOUS and I’m not speaking for either side.

For the last year I’ve been building Oystr, a collaborative problem solving platform, upon Node.JS. Prior to moving entirely into JavaScript for the full stack, I had been developing on top of .NET using C# for about 15 years. Over the last year I’ve found working with JavaScript on the server to be liberating at times compared to the .NET/C# world. But this has come at a huge cost. My initial elation with JavaScript is now beginning to turn to depression. For a while I’ve been going backwards and forwards between elation and depression so much that I started to feel kind of bi-polar. But now, I think I’m ready to call it. JavaScript is making me depressed.

I think one of the biggest annoyances is the lack of types. I know people are probably thinking “I could’ve told you that” but let me explain. It’s not so much the lack of any strong typing, it’s more what strong typing provides you… auto-complete. Man I miss proper VS.NET auto-complete. The thing I love about JavaScript is it’s duck typing. But obviously this has come at the cost of my sanity.

But really auto-complete is just a convenience. That’s not the main reason I feel depressed with JavaScript. The main reason I feel depressed when I go back to working with JavaScript, is that re-factoring just feels so much harder. I don’t have any quantitative data to back this up, it’s just a feeling. But damn it’s stressful when I want to do something simple like rename a method. I’m always worried that WebStorm will accidentally rename similarly named methods or properties. And here is the worst part… I can’t do a compile to do a simple sanity check. Sure if something compiles it doesn’t mean it works for sure, but it sure goes a long way in giving you the peace of mind that “oh, it seems to be OK…” which is a hell of a lot better than feeling “have I missed or forgotten anything?!?!?!” and never actually feeling comfortable with saying it’s all fine until you’ve done a full systems test.

This compile-time related depression (I’m calling it CTSD for Compile-Time Stress Disorder) extends beyond the fact my compiler safety net no longer exists. It’s the fact that when I started programming 22 years ago, what got me so excited as a child when I moved from BASIC to C / C++ was that I could compile my code and when it compiled / linked properly, it felt like a milestone. Programming, for me, is more than just work. I love to code, it’s part of who I am. Taking away one of the things that makes me feel like I’m getting somewhere (whether that’s actually true or not) just sucks the fun out of programming for me.

I think for the sake of my sanity and to bring joy back into my nerd life, it may be time I put down JavaScript and start looking for something not CTSD prone.