2014-12-13

Hello OutSystems

Early this year I switched from IBM to OutSystems in a move some considered crazy, some considered bold and some considered fantastic. After 10 years working with infrastructure I am now working as a software engineer, the job I dreamed about when I was around 12 typing way on my Spectrum 48K. But I really don't get that kind of feeling like “I finally got there”. In fact I have grown an interest for many other aspects of computers and technology other than software so I am very glad to be a Cloud Software Engineer which means I get to build software with infrastructure and security in mind too.

OutSystems is a great place to work. The company makes a huge effort in order to keep people happy and motivated. There are rules though that really make perfect sense and keep everyone productive and motivated rather than annoyed and controlled. They focus on how you should act rather than things you can't do. They also target management behaviour rather than being limited to the base of the hierarchy. The rules help establish a company culture that everyone is proud off.

I am sure that as the company grows (and it is growing) there will be some challenges maintaining this culture. But I know management acknowledges that. That is the first step in the right direction. Management listens and management talks to you. There is a rule for that.

Then there is all this talent around you that push you forward all the time. I go to work everyday thinking I am on my way to meet a group of friends that happen to be great engineers and get together to have fun building awesome stuff. And you also get to talk to people from other teams and learn about their work too.

I am now in a company that has learned to base its productivity on employee satisfaction something I have been routing for. I just wish there were more companies in Portugal coming to this conclusion: that productivity can be a result of what employees "like to be" rather than "have to be".

2013-08-13

MyBot

I have been interested in tracking my every move automatically for some time. I cannot count the times I have tried to remember "when did I got to that place" or "when did I do that thing with my friends". This is probably a common problem but I think I probably forget more than most people and being a geek and a gadget lover, I became somewhat obsessed with having my gadgets track me so I could get back to that memories easily. There are lots of services that can be combined together to create something like this and I have tried some of them. IFTTT, Momento and Day One are very good examples. None of them is exactly what I need but they are an excellent help. There is also the problem of who to trust with my data and how long will these services keep running and doing this job for me before they shutdown for whatever reason and I have to move my data around once again.

So I decided to roll my own set of scripts that gather this data for me on my own hardware. This will take some work but since I also wanted to get back in touch with the software engineer in me, this could be a good opportunity to keep my skills in order.

I have to mention that I was also inspired by a Codebits talk by Alexandre Solleiro about Activity Streams. He talked about tracking your activities and building a platform for it. He also mentioned Nabaztag which is a cute rabbit bot (now called Karotz apparently) that interacts with you based on your activities.

Besides the data gathering, I am interested in that interactivity that can be built around this data powered bot.



MyBot design

First things first: how will I make this work.

I bought a raspberry Pi almost only with this project in mind and it's where the main action will take place. Maybe more than one in the future...

Since this is a hobby and I lack the free time, most of its architecture is being decided as I go. I just cannot afford a better solution at this point. I do have ideas and some of them are already in progress.

Python

This will be done in python. I am already learning python and its powerful habilities combined with its simplicity of use and portability make this the best solution. I am using a MacBook/Mas OS X and Eclipse for the development but this will run on Linux.


Modular architecture

I will split the bot into modules that will run in different processes. For example I will have a module for monitoring my twitter feed, another to receive commands from me, one for monitoring my email activity, etc.

The decision for different processes has nothing to do with performance and multicore (I am running this on a raspberry pi remember?). It is about interactivity (because modules run independently from each other without having to wait for full completion) and it is also about being fail safe. A module crash can not make other modules crash. Not even parent processes. I'm not sure yet if I want the child processes to die if the parent dies.

I also have a few ideas to have the different modules monitor each other but for now I will only have one controller process that will launch the other modules in different processes and keep an eye on them to relaunch them if necessary. This controller module will also be monitored by a another controller.

This will also allow to load new modules without disrupting other already running modules.

Why am I spending so much time around the fail safe features? Because it is fun... and because if a module fails when I am away from home just after introducing a few new buggy lines of code just before I left home, I want it to alert me and try to recover on it's own until I get the time to get back to coding.

Communication

During development I am using a command line console to communicate with the bot. But in the future, I want to remotely send commands or questions from anywhere. The easiest way for doing this is using email. It's probably the fastest app to launch on my phone and its extremely easy to set up a recipient for the email and at least a subject line. Of course these have some security concerns described below.
I am also thinking on communicating with it using twitter, IM or https service but these ideas are still a bit blurry at the moment with lots of questions in my head. Would also be cool to talk to it when I'm home :)

Security

See Concerns below...


Concerns

Security

I am a bit paranoid about security and since I am building something that will have access to all my life, security is a big concern.

This bot might have to know some passwords to collect data but I'm not sure yet how will I store them and use them. I have a few ideas but not mature enough. The first thing I need to do is reduce the need for the passwords and try to rely as much as possible on alternate authentication mechanisms and data that is already public. I'll then move to private data.

The way I communicate with the bot also concerns me. Emails will probably be the most used communication method but I have little control over authentication and no control over confidentiality of the emails so I will probably limit the kind of information and commands I can do with exchange using email. As IM is concerned, I do not know anything about it at the moment. HTTPS is a potential candidate for exchanging confidential information with the bot but I rather limit it to temporary URLs because I don't want to have an HTTPS service permanently available. SSH is probably the most secure option so far but would also be better not to have it permanently available. SSH is also cumbersome to use from a mobile device.

The code

Since I'm learning python, there are obvious concerns on what I'm doing and if I am using the best practices for coding in python. But I have a few issues I already know I will have to get back to:

1. Using multiprocessing module for launching the bot modules
The multiprocessing python module implements a very easy to use interface for forking processes but at the same time removes some control. Forking copies all the memory from the parent process to the child process. This means that all memory in my control module would be replicated to every other module that it spawns. I don't know if python's garbage collector can take care of this at some point but it won't be the most efficient solution anyway. I will probably move to fork()+exec() in the future or take a look at subprocesses module. In that case, I will also lose some great ways to communicate with the children processes present in multiprocessing module and will have to implement my own sockets communication module.

2. Import modules code
The code that imports the modules is hideous and does not seem very robust. Might have to recode it.

3. Will I ever finish this...
I have so little free time and so much projects I would like to do that some of my projects get left behind. This is probably the most laborious one so...
On the other hand, I really need this learning experience and I'm full of ideas and scenarios I would like to test and learn from (even if not the best solution for this kind of project). Even if I don't finish this project, I am already learning from it.
2013-05-16

Trying out Post app for iPad

Really. This is just a test.




Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6
Pre
Blockquote



Seriously. Just testing.
2011-06-17

Geek on wheels

After years of dreaming of ridding a motorcycle, I finally got my license. But that wasn't all I needed to accomplish my dream. I also needed a bike. Off course there were always a lot of biker friends around to borrow a bike from. My favourite was my sister-in-law's 2008 black and white Triumph Bonneville T100. But it's not the same thing. I needed MY OWN bike.


Two years too late, it finally came the time to get my bike. My search was a bit short. Too short maybe. But what did I search for?


  • not too expensive
  • classic look
  • easy to ride
  • practical


As you can see, I lack the mechanical expertise to look for mechanical characteristics. I did ask a few experts but their opinions were almost always concluded with "... but you should try before you buy and choose whatever makes you feel better...". So short story really short, I got myself a 2010 Triumph Bonneville T100 Black and I'm enjoying every minute. Ridding this bike seems to me a pretty good definition of feeling good.

I did look at the Harley-Davidson Sportster 883 Iron, sometimes compared to the Bonneville, but I was already in love with the Bonneville. I have to admit what I mostly like about this motorcycle is how it looks. It's absolutely beautiful. I love it's classic look and British accent. It is also a very easy bike to ride even for a newbie like me and very practical for day to day ride. So it certainly seems to fit me very well. It's not that good in the "not too expensive" department but it scores so well in my other requisites that it won hands down.

Now let's ride...



2009-02-26

GetExpandedStringValue: Type Mismatch

This is another post about time painfully spent trying to solve a simple bug.

I'm currently migrating a lot of VBScript code to a VB.NET application. Before you email me about how I should be using C#, please take note that migrating code is already tricky enough as it is so I decided for a language that shares a lot of syntactic similarities with VBScript.

Back to the bug, I was migrating this VBScript code:


Const C_HKEY_LOCAL_MACHINE = 2147483650
Dim Server
Dim strValue
Dim objReg
Dim strKeyPath
Server = "localhost"
strValue = ""

Set objReg = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & Server & "\root\default:StdRegProv")
strKeyPath = "SYSTEM\CurrentControlSet\Control\ComputerName\ActiveComputerName"

If objReg.GetExpandedStringValue(C_HKEY_LOCAL_MACHINE, strKeyPath, "ComputerName", strValue) <> 0 Then
Error ("error reading registry")
End If


Well, not actually this code but something like this. This works in VBScript but this code in VB.NET that compiles and seems quite the same, generates a "Type mismatch" exception:


Const C_HKEY_LOCAL_MACHINE = 2147483650
Dim Server
Dim strValue
Dim objReg
Dim strKeyPath
Server = "localhost"
strValue = ""

objReg = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & Server & "\root\default:StdRegProv")
strKeyPath = "SYSTEM\CurrentControlSet\Control\ComputerName\ActiveComputerName"

If objReg.GetExpandedStringValue(C_HKEY_LOCAL_MACHINE, strKeyPath, "ComputerName", strValue) <> 0 Then
Error ("error reading registry")
End If


After wasting a lot of time reading forums and tweaking the GetExpandedStringValue arguments and reading it's documentation, I finally fixed it replacing
Const C_HKEY_LOCAL_MACHINE = 2147483650
with
Const C_HKEY_LOCAL_MACHINE = &H80000002


The final fixed code would be something like this:


Const C_HKEY_LOCAL_MACHINE = &H80000002
Dim Server
Dim strValue
Dim objReg
Dim strKeyPath
Server = "localhost"
strValue = ""

objReg = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & Server & "\root\default:StdRegProv")
strKeyPath = "SYSTEM\CurrentControlSet\Control\ComputerName\ActiveComputerName"

If objReg.GetExpandedStringValue(C_HKEY_LOCAL_MACHINE, strKeyPath, "ComputerName", strValue) <> 0 Then
Error ("error reading registry")
End If




You might also run into this problem, off-course, with functions like GetStringValue or EnumValues.
2008-10-25

PowerShell: which for windows

As Stephen Mills points out in a comment below, all this is unnecessary because there's a cmdlet that does this even better.

Get-Command


So please disregard this post :P



Fresh code out of the oven. I was on my Windows XP box and needed to know where I had an specific executable file stored. Some Windows boxes have a similar command called where but it wasn't availed in my box. So I just coded one in PowerShell.



function which ($file){
$return = @()
if (Test-Path ".\$file" -ea silentlycontinue){ # check current dir
$return +=@(".\$file")
}
foreach ($path in (get-content env:path).Split(";")){ # check all paths in PATH env var
if (Test-Path "$path\$file" -ea silentlycontinue){
$return += @("$path\$file")
}
}
return $return
}



Off course it can be refined - and probably should - but it's good enough for most situations.

Ex:

> which notepad.exe
C:\WINDOWS\system32\notepad.exe
C:\WINDOWS\notepad.exe



PS:
I store the output in a string array instead of printing out the string because to be able to use the output as an object and easily do whatever I want with it.
2008-10-16

PowerShell: very simple FTP client

One of my first experiments with PowerShell, besides ls and HelloWorld.ps1, was trying the object oriented features and the .Net Framework. I was at the time thinking about a problem I was having with an FTP client I was using and I asked myself: "Would it be possible to make an FTP client with PowerShell?"

I knew there was a .Net class that implemented an FTP client so I googled and bit and found it. After reading the examples shown and a little try-and-error, I came up with this extremely simple FTP client in PowerShell below. Note that by "extremely simple" I mean "with very few features and no error detection code". This only gets one file from the FTP server. I probably wouldn't use this in production without a few changes. But it's still a nice example on how to use the .Net framework with PowerShell and this class in particular


$localfile = "c:\file.txt"
$remotefile = "/folder/file.txt"
$ftphost = "ftp://ftpserver"
$URI = $ftphost + $remotefile
$username="username"
$password="password"


function Get-FTPFile ($URI,$localfile,$username,$password){

$credentials=New-Object System.Net.NetworkCredential($username,$password)



$ftp=[System.Net.FtpWebRequest]::Create($URI)

$ftp.Credentials=$credentials
$ftp.UseBinary=1
$ftp.KeepAlive=0


$response=$ftp.GetResponse()
$responseStream = $response.GetResponseStream()

$file = New-Object IO.FileStream ($localfile,[IO.FileMode]::Create)

[byte[]]$buffer = New-Object byte[] 1024

$read = 0
do{
$read=$responseStream.Read($buffer,0,1024)
$file.Write($buffer,0,$read)
}
while ($read -ne 0)
$file.close()
}



PS:
By the way, this is not me. It's just someone that is using my words without citation.
2008-10-14

PowerShell

A few weeks ago I had a chance to attend a workshop about PowerShell entitled "Using PowerShell" by Ed Wilson. Fantastic teacher and fantastic person to talk to about anything.

This was my first time using PowerShell. I had tried one of Monad's first versions but it took so much time to load and crashed so many times I decided to wait for a final version. I ended waiting a bit more than the final version though. My first impression was "This is fantastic!!". And it really is.

Being a lover of the command line and an avid user of Bash and other UNIX shells, I welcome PowerShell. I'm not going to compare Bash and PowerShell and try to guess the best shell. In my humble opinion, they're not comparable. For that kind of entertainment, google for PowerShell vs Bash. You'll probably get a lot of results with links to endless and pointless discussions between UNIX and Windows zealots.

PowerShell and Bash have very few similarities. Quoting Ed Wilson "PowerShell was done by UNIX guys. Pony tail and all" so it's not so much a surprise to see aliases to PowerShell commands (called cmdlets) that look like Bash commands but they actually do different things even though the visible result might look similar. But that's it.

For instance, ls in Bash gives you a list of files in the current directory. In PowerShell, ls is an alias to Get-ChildItem. This cmdlet "Gets the items and child items in one or more specified locations." (type man get-chilitem or man ls in PowerShell for this :) ). If the location is a directory, you get a list of objects of the type FileInfo. Each of this objects have properties (ex: Name, LastWriteTime) and methods (ex: Open, MoveTo). This info, by default, is displayed as a basic listing of files but it's much more than that.

PowerShell and Bash are fundamentally different and are built for different platforms using different resources. In PowerShell everything is an object. PowerShell brings the power of Microsoft .Net Framework at you finger tips, makes using WMI easier and is built for Windows systems, integrated and one. Bash lives of individual third party tools and binds them together.

I plan to post some of my first experiments with PowerShell as examples.
2008-05-26

Missing Win32_Product

Today I spent some frustrating time debugging a VBScript looking for an authentication problem that didn't exist. I was trying to a install an MSI file on a remote server and kept getting "SWbemServicesEx: A security package specific error occurred."

Googling the error only drove me further in the wrong way. Only when I decided to run the script locally on the target computer I realised it wasn't an authentication problem. At least not any more.

The error was now "SWbemServicesEx: Not found". I then noticed the Win32_Product class I was using to install the MSI files did not exist. I googled for the error and found lots of people with the same problem but no straightforward solution. Well... it turns out this was not the first time this happened to me... Maybe the next time this happens I google it and find my own blog post with the answer. It's as simple as this:

  • "Add/Remove Programs"
  • "Add/Remove Windows Components"
  • "Management and Monitoring Tools"
  • check "WMI Windows Installer Provider."
  • "OK", "Next" and "Finish"

You might need the installation disk or at least the i386 folder.