Development and deployment strategy

Posted on by
Reply

How does development affect deployment, and how can deployment have an effect on development?

Over the course of the last few months, I’ve started to realize how much emphasis should be placed on the deployment of an application very early in the development process, and I’ve started to put serious thought into how as a developer, I need to make that a painless step.

Scenario

BIgStompy Corp has a new widget tracking system in the works. They are currently using  a spreadsheet and a widget catalog to track all the widgets they have on order, in-stock and being fulfilled.  Management starts with identifying what they need in their shiny new WidTrack application.

“We need to be able to enter in a part number and description”

“We need to be able to count how many widgets we have on order” 

And so on.

“When you get it close to up and running, we want to see it. We’d also like to roll it out to some of our more experienced staff, so they can make sure it’s all there, and then we aim to put it into place by Q3 this year”

The developers take these requirements.

They decide that they’re going to create tables, they’ve got an idea that they want to use Entity Framework, and they’re going to use (as it fits quite nicely) ASP.NET MVC to develop a nice, friendly front end to WidTrack. They start writing the code. They check it into their repository of choice.

As they proceed through the development, several packages are required, NUnit for testing, json.net for serialization, and all manner of other packages. This is going great, they develop, they check it in to the repository, feature after feature starts to appear. Two months in, they’ve got something worth showing to management. They copy the site to a test server in the office, install the database and get it running. Management love it, they want to start testing this with a couple of users straight away. They copy all the files over to a new staging server, where it’s set on almost immediately by a few users.

Development continues, the team deprecated some packages, updated the version of a few others. Pushes out to the test server happen quite often to show progress. Occasionally these changes are put to the staging server. Cycle continues. One day, the team are told to push it, they like it so much that they have stopped using the old way and are exclusively on WidTrack on the staging server. What was the last version that was pushed to the staging system? What changes did that have? Did that have the huge schema redesign that you needed to do to introduce the “tweet when in-stock” feature? Development slows to a crawl, because now the team has to figure out how to get the newest version updated onto the server.

My thoughts

It’s easy to concentrate on the development. The design was there, the product was there and the users were loving the result. From a development stand-point it was clean, stable and well tested. The downside was that now the development team are scrambling to figure out how to migrate data between schema, how to bring about changes they’ve got no method of identifying.

The deployment of a system seems like pie in the sky when you’re starting out developing, why should you worry about how it’s going to be put somewhere, when you really don’t know where that somewhere is right away. In this scenario, here’s what the development team knew before development:

- That they would need to show management a version that wasn’t development

- That they would have to put a version into production

- That no software is perfect, bugs happen. You’re going to have to update the software.

My scenario above might seem a little weak, but it should be possible to identify these requirements from it.

This is not a small challenge to overcome though. In the big design up front world, we would still have spent more time trying to figure out the features that needed to be in there than we would have spent figuring out how to deploy. In the agile world, we don’t know what we need up front, we do the simplest possible thing and iterate over that.

In retrospect, I believe that I would tackle the choice of deployment tools, software version and work tracking as critical parts of the development. How you manage the code in the repository, how you manage dependencies on third party libraries and how you keep them  in check. They all have to become part of your up-front standards.

So, development has an effect on deployment in so much as the methods chosen, the process of that development can clearly limit how you deploy. Deployment has an effect on development not so much in how you should code and what technologies you should use, but in so much as the process of development. How you check-out/commit code to your repository, how you branch, how you version or tag commits.

What’s my answer for all this? I don’t have one. Sorry.

I’m still learning a lot each day, and trying to form my opinions and solid ideas on how to achieve a smoother work-flow. I’m not certain right now whether there is too much tooling around some of these processes, or not enough transparency in the tooling that is there. Deployment into a live system is a complex task when your application is more sizable, but that doesn’t mean it shouldn’t be transparent as to what’s there, or how it’s done.

I’m going to stop this post here, and come back to it over the next few days. It’s something I’m going to spend more time considering. Hopefully getting some reasonable suggestions and ideas for others.

 

Taking my responsibility seriously

Posted on by
Reply

One week ago today, I became a United States Citizen. It took a while and a bit of preparation to get there. But, after what amounted to an entire day in a government office, I walked out with a certificate stating I am a full citizen of the United States of America.

When I moved to the US almost five years ago, I gave up more than just a life in the UK. I could not work, I could not vote. It was almost 7 months before I was back earning again. Even when I was permitted to work again I only moved to being a permanent resident. Throughout that year, I saw something I had never had the opportunity to witness. A presidential election.

For the longest time in England, I felt like my vote made little to no difference. I worked, I paid taxes. I went home, slept and did the same the next day. I did get in several discussions with friends who felt a little more seriously about our responsibility to play a stronger part in democracy.

What I saw in 2008 was something unlike any election I had witnessed. I actually found myself caring, I wanted to play a part, I wanted to rise up and cast my vote! But I couldn’t.

In December of 2008, our daughter was born, in 2009 we bought our first house. Now, 4 years on from that presidential election I am a fully qualified citizen. I have a vote.  Although I have a lot more at stake now than I ever did while living in the UK, I feel that if British politics had ever felt this energized I would probably have been able to cast a vote with more feeling than I ever had.

But, to those who have impressed upon me the importance of my vote in the past, to those I have ignored or dismissed. I apologize. I think we have a responsibility to our nation, to our future, to cast our vote.

I promise to take my responsibility seriously. I promise to cast my vote, let my voice be heard.

Software Craftsmanship

Posted on by
Reply

In recent months, I’ve had the chance to watch a lot of the excellent series, Clean Code, produced by Robert Martin (Uncle Bob). If you haven’t had the chance to take a look at these, I highly recommend watching the series if only for the entertainment value alone! Uncle Bob can be a little “quirky” but he gets his point across. Watching the videos has allowed me to reflect on my own coding practices and come to some conclusions I’m not sure I would have reached alone.

This last week, I listened to a Herding Code podcast with David Starr, in which they go to the topic of Software Craftsmanship. The question was raised as to if software development can truly be treated as a craft. I’ve long considered what I do comparable to an art. Given a subject, no two artists will come out with the identical piece of work. Both achieve the goal of producing a recreation of the subject, both consider theirs to be the more correct. The same is true of software development. Two programmers will have differing opinions, thoughts and ideas on the subject, they will probably both achieve the goal laid out in front of them, but who is to say which is correct?

Craft an art, trade, or occupation requiring special skill, especially manual skill: the craft of a mason.

I think by virtue of possessing the skill necessary to be a programmer, you could consider yourself a craftsman. But I definitely think it’s more than that. If you search for the term Software Craftsmanship you’ll come up with a range of results. Many of them resolve around the idea that it’s all about the quality of what you produce. Uncle Bob says “We are tired of writing crap.”

When a carpenter or a plumber wants to learn how do what they want to do, they don’t go to school for four years, read some books, do some projects, get a diploma and go work. They undertake an apprenticeship, they go and work with another, more experienced member of their chosen profession. This person will teach the apprentice all they know. In the same, the apprentice, having spent time learning may bring some newer ideas and techniques to the teacher. In this regard, I feel that apprenticeships for software development would be better suited to bringing about the craftsman of tomorrow than any internship could.

Having watched many of the Clean Code videos. Having mocked my colleagues and been mocked about our variable naming, our methods doing way too many things and everything Uncle Bob points out as bad, I can say with certainty I do not consider myself a software craftsman. But, after spending time reflecting on how the lessons in the videos apply to me, I have come to a conclusion that I can not go down the road alone.

For the past twelve months, I’ve had the great opportunity to work not as a single developer, but as a part of a team. One of the most difficult changes to adjust to (and I’m not sure I’m even there yet) was the idea of paired programming. As part of paired programming, we no longer just sit at a keyboard, producing the work of art we believe we need to make. There is only collaboration, discussion and cooperation. The piece of work produced at the end of the day has no individual stamp on it, it has a two programmers ideas, achievements and mistakes. At the end of the project, the team will have their stamp on it. Not mine. To the point of software craftsmanship, I don’t believe I could see the mistakes I make, or the shortcuts I might take if it weren’t for my team. And if delivering high value code, efficient designs and great customer product were to be achieved, it is only because all of us aim to be as professional as we can be.

I think that to become a craftsman, I have to continue wanting to learn more about what I do, I have to keep striving to be better than I am. I do have to continue taking pride in my code, and making it as clean, re-usable and extendable as I can. More importantly, I have to want to help my colleagues learn to do the same, and not be afraid to ask questions when I don’t know how to do something. Only when we can look at the definitions laid out by Uncle Bob and others and answer, “yes we can”, could I possibly begin to consider myself a craftsman worthy of the label.

The satisfaction of working with code you know

Posted on by
Reply

The other day, I got the opportunity to return to a project I worked on very early in its life-cycle and had had a lot of influence over the coding. It was only for an hour or so, but after working with a new code-base for a few weeks, it was nice to reflect back on an old friend and realize how much I missed working in it.

Working in another developers code, when you’ve not followed the decisions made from the start is a frustrating practice. You’ll see code you don’t entirely agree with, or that you think is just horrible code. But you don’t know whether the programmer that wrote it didn’t have any better techniques in-front of them, or whether they just weren’t very good programmers. It’s easy to claim that the programmer just wasn’t very good, or that you could do better.

I’ve spent the last couple of hours adding functionality to a website created using my bounce framework for PHP. I didn’t realize how friendly it was to me. I decide to add a form here, or fix a bug there and I’m just happily jumping around in the code. I’m trying to develop bounce to be a usable framework, not just for me, but for other developers coming upon it. My issue is, I know the framework, the decisions are in my head, and the code reflects what I thought was good. I took a little inspiration here, and a little influence there, but overall it’s what I think is good programming.

I’ve spent a lot of time criticizing what has come before, whether that be me being a little elitist, or just being incredibly picky about the code I read and write, but I’ve never stopped to fully appreciate the challenges other developers might have gone through. I always try and pass it off as, “I just don’t like lazy programming”, but I think I’m just a little too picky and critical.  Maybe next time I will stop and try and appreciate the design decisions other programmers have made, or try and understand the limitations of the time. Maybe I can improve some code using techniques now available, or that I have learnt through my own experience. Even better, I can pass on what I know to other programmers and be more open to learning from them.

Do write your own ORM

Posted on by
Reply

As part of my daily blog reads, I came across this article posted by Jimmy Bogard on Los Techies, and I find myself agreeing with the point, but disagreeing with the moral of the story. I understand the point he is making, that the problem has been solved and there are some great existing ORMs out there that can be leveraged, why re-invent the wheel yet again? But, as a developer, as someone who likes to take a problem head on and try and solve in the most efficient way I might want to tackle writing my own ORM. I think that if the business affords you the time to go ahead and write what you need, and you’ve looked at the existing ORMs and found them lacking, go ahead, go write your own.

Using an existing ORM affords you the time to concentrate on your domain, knowing the track-record of the software you choose gives you a comfort level that you just can’t get from your own software. Your SpiffyORM has to prove itself over time, you have to identify a whole subset of bugs and deficiencies not related to your core application. And to Jimmy’s point, this adds “cognitive weight” to the overall application.

Jimmy starts with saying that if you’re not Ayende or Greg Young that you just shouldn’t bother trying to write your own, but then proceeds to list three pieces of software that aren’t from these guys. For example, Massive from Rob Connery which I’ve read good things about. He didn’t stop at EF or NHibernate and say, “It’s done, we can’t improve”. He went ahead and did his own thing, and the result is a viable alternative ORM.

I don’t think we should be discouraged from writing software for ourselves, because others have already done it for us. If we as a race decided to stop driving forward with new ideas, where would we go? We already invented the car, it gets us from A to B, why should we try improving on that? Human beings learn through trial and error, from the moment we’re born, we investigate the world around us. So, why does this not apply to software?
Not everyone out there is going to set off down this path of discovery and come out with the next best thing, in fact I believe few of us are going to achieve that. But, if we don’t stop and ask ourselves “Can I?” (which should be closely followed by “Should I?”) then we might as well accept that we’re just monkeys tapping at keys, because the few pure genius developers among us do all the hard stuff, we don’t need to think for ourselves.

Introducing bounce PHP

Posted on by
Reply

When I started this site, I did so with the intention of making a site that would showcase what I could do. Over time, life took over and I never had the drive to put the site together the way I would truly like. I moved to a WordPress blog a while ago and bar some annoying ups and downs, I’ve quite enjoyed the experience of having an accomplished and friendly, extendable CMS for the site.

But, that doesn’t mean I haven’t stopped developing, and breaking my own resolution to concentrate soley on my Microsoft technologies and not keep hopping around, I’ve made some pretty big improvements to a PHP framework I began development of several years ago. I started by calling this framework Amity, which means ‘friendship’. But, considering my lack of focus and my ability to bounce back and forth between projects, bounce makes a little more sense.

I’ve decided to continue sharing, so you can find the project over on Github,

https://github.com/tbd-develop/bounce

What makes a good teaching article?

Posted on by
Reply

Recently, I had the chance to revisit an article I wrote on Code Project. I had never written what I would call a tutorial before that article and I haven’t written one since, but it did get me thinking on why I wrote it, what I knew and what I learned from it. I thought I would share that here.

I have been bouncing around Code Project for years, and had seen many articles, read many comments. I rarely took part, as I didn’t feel I had the ability to comment or I just didn’t want to write something critical of another persons work. There was a post by a member that the articles contained on the site were of an increasingly poor quality. And I agreed. But, I had never contributed, is that fair of me?

I set off down the path of writing an article on interfaces. I planned on creating a tutorial with solid, working examples that could be applied to every day development. What I learned from this experience is that it is incredibly difficult to put together a real-world example of a concept such as interfaces. You’ll see if you take a look at the article that I eventually ended up on little more than a text book example. Further to that, the comments after publishing show that I really didn’t understand as much about interfaces as I thought I did, or that I just couldn’t apply the knowledge to an example.

So, what makes it so difficult?

From a real-world example perspective, the issue is that we can never provide enough code (either through proprietary restraints or pure code scale). Our brains work better at breaking things down into small, bite-size pieces. And showing a more fleshed out example of interfaces without spanning multiple articles you’re not going to get to show everything about them.

Some authors would tackle a sample system, something that allows you to see how the pieces fit in in the overall system. So, I might have expanded the article by several installments, put in a WinForm app that had buttons and you can see the results as we go. The issue with this is that those sample apps suffer another issue, they are too focused on the subject and never show enough about how the pieces fit into a real world system. Sure, they might show some very solid examples of interface usage, but a real-world system is rarely a single component. The time necessary to write up the other code, without explaining it outweighs the benefits of the sample in the first place.

After editing my article, I received an extremely lengthy comment from another poster telling me that I perhaps should have waited, that I didn’t understand interfaces well enough. And to an extent, I do agree, I didn’t understand enough to be able to show absolute proficiency with the subject. But, I did choose to try and share what I did know and understand with others. I put together an article that laid out an example, that described the usage and I hope that it helped a couple of people. I appreciated the comments from this new poster, and I took the time to read the response and will hopefully produce a better, more rounded article next time I choose to post something.

 

Expressions

Posted on by
Reply

A huge part of my experimentation with Roast Potato is down to a recent fascination with the Expression. I’ve known since I started working in the .NET environment that it could do a lot with reflection. And through the development of a filtering system at work, I’ve found some of the extents to which that can be taken and seen some very impressive results.

With Roast Potato, my direction is more of an educational jaunt. If the library turns out to be useful in some capacity, then that’s awesome, I’d take a great sense of pride in knowing my work helped someone else along. But, I want to explore what can be done with expressions, how far I can understand them. This is the start of my ramblings on what I’m finding.

Once I had settled on a syntax for RP and spent a couple of nights fleshing out the Regex and other such fun, I got down to writing the tests, and I soon realized just how quickly some of the things I wanted to do could be achieved. For example;

The expression above does the following;

  • Convert the object value into the type of the property we’re requesting (Price[decimal])
  • Put that value into a constant value (just an Expression to wrap around the value and allow it to be constructed)
  • Returns a constructed lambda expression using the GreaterThanExpression and passing it both the property we wish to be greater than and the value we wish to be greater than.

Given the query;

“Price gt ’21′”

The resulting Lambda expression;

x => x.Price  > 21

We can then apply that against a collection of objects and filter out a list, and that’s the starting point of the roast potato library. The actual code to go and parse the fact that it’s a GreaterThan is more complicated than the part that does the heavy lifting once we’re there.

I don’t want to try and go too deep. I barely have a working understanding on Expressions so far and I want to understand so much more. Also, I want to get a working Roast Potato sample up and running so I can show what it can do. Then, I can hopefully learn what it is to understand.

Tonight I uploaded the first version of Roast Potato Console, just a little sample app with some test data that you can type the query into and see it return the results. This app will eventually go away and I’m starting to believe that most of the need is for server side anyway, not a client/server side solution. But, I’ll leave that for another post and another day.

Open Source Contribution

Posted on by
Reply

I’ve always been a disorganized person, both in my general management of time and resources and in my projects. Professionally, I have the job I’m doing to focus on, and although my thoughts are scattered at times and I struggle to always convey them, I get the job done. And I hope, well.

Personally though, I lose focus quickly, I find my mind being drawn to other topics quickly. One minute I might want to write the next great sim game, the next I want to write blackjack. I lack focus and will-power. Sometimes, I give up because I get bored, sometimes because I realize that I’m not coming up with anything new. Mainly, I just lack the attention span required to develop all components of a project.

Recently, in the course of my professional life, I’ve been very focused on providing a way to query data in all manner of ways, quickly and efficiently. But as the development has gone on, I’ve realized there are so many ways in which we can query data now, that there aren’t enough hours in the day to provide the ultimate solution. We’re going to have to bend a little here, and rewrite a little there. This has given rise to a new personal challenge for me, and one I’m committed to staying the course on. Now, I know I can’t take what I did for my employer and just run with that, that wouldn’t be allowed, but what I can take is the lessons I learned throughout the course of the project and see if I can’t come up with something that will benefit others trying to tackle these issues, working on them from a totally different perspective.

In order to try and focus myself a little more, I’ve chosen to make what I do an open-source  project. This way, if I stop doing it, there’s a chance what I’ve done will help someone else. It might also attract the attention of other developers that want to contribute, and at that point I have a commitment to my peers and thus I have to be more professional in my behavior.

The project I have started has, as all new projects must, a quirky name. I called it, Roast Potato. What, really? Why?

Well, to Roast is to query. And Roast Potatoes are among my favorite foods. So, I’m going to Roast a Potato (a source of data) using a Recipe. I haven’t figured out all the other quirky things yet, and I’ve only just started. But, this one is interesting me more and more, and with each .NET project I start personally, I keep struggling at the same points, what is it to get data from the server controlled by the client.

What I’m trying to aim for is a way to represent the data on the server in a standard way, and be able to query that data from the client without having to worry about all the wiring in between.

Ever heard of OData?

Yes, I’ve heard of it, I’ve used it and I really like what I can do with it. And I’m not trying to replace or reinvent the wheel here. What I’m trying to do is make a reusable and simple client side interface for some pretty complex server side functionality that I can understand. I’m also thinking of how I put this into a PHP form as well as .NET, so I intend to write client code once and two versions of the server.

Anyway, I’ll see where it goes and if nothing else, there’s some code out there to show people how to extract an expression from a string.

Battlefield 3 – Best and worst FPS ever

Posted on by
Reply

Although game development and application development differ in many ways, the delivery of a product, to a standard is still the end game.

Since its release in late October, I’ve had both the distinct pleasure and single-most painful gaming experience with the video-game, Battlefield 3.

If next week, after having run a beta test on our software for a month, knowing there were issues in it, went on a massive marketing campaign and released the software, I’m pretty sure the company would be hurt badly. But, in the same, I think we would do everything we could to repair the damage done, and work hard to get the problems fixed.

When EA/Dice released the beta of Battlefield 3, I was quick to give it a try. I loved its predecessor and looked forward to playing it. Once downloaded, I accepted certain quirks, certain difficulties in the interface and the potential for bugs. Overall, it ran well and my machine was coping great. The feedback all seemed to be on the missing parts from Battlefield 2, dislike of the UI, but nothing particularly unmanageable.

1 month later, after the release of the product, the forums were full of “I can’t get this to play”, “keeps crashing”, and comment upon comment showing anger and frustration at having spent money on a game that (seemingly) nobody could play. To make matters worse, EA/Dice were boasting about being one of their biggest game launches ever, that they had sold so many copies of this game, and if you logged in you could see that. Reading through the issues, I find some people having success with port forwarding, that they are able to get consistency that way. Looking at the list of ports, this doesn’t sound like a good idea. I have to forward a whole bunch of ports (include 80 and 443) to my machine, I have to disable UPnP . I’m liking this less and less. To make matters worse, it doesn’t even make a difference.

Considering the number of people on the beta, I find it hard to understand how an issue such as this didn’t delay the game. They must have seen it. The beta showed some issues up, but nothing you couldn’t put down to test software. And, if they didn’t find something that means changes to net code were done after the beta code had been finalized that broke this in a really bad way.  Maybe EA noticed it and decided to release anyway,. because they had sunk millions into advertising. I’ve downloaded two huge updates for this game, and still nothing seems to work. I get one maybe two games in (if I stay in a session, that works), but if I log out, or try and do something else on MY computer, I have to restart the machine and try again. This is ridiculous.

Yesterday, I spent a great deal of time wandering around the system trying to find some logs that could show me what was failing. But, it seems that the web based system EA/Dice have chosen to use doesn’t really do logging. The game doesn’t seem to produce any logs, and I can’t find a way of enabling a debug mode to at least show me what’s failing. This suggests that EA has been completely useless at providing any sort of advanced troubleshooting and/or this feature doesn’t exist, because this information would make it out quick considering the number of issues out there. What makes it all the more frustrating is the error messages. They’re downright insulting. “An error occurred”, “You’ve been disconnected from EA”. WHY?!!

Assuming that the issue isn’t hardware based (I have a machine capable of taking the game on the highest specs), and it isn’t software based outside of EA/Dice software (drivers are all updated, Windows is updated). Where does the issue lie? Adding to the frustration is that the single player game is “online” too, and crashes within 2 minutes of playing. I didn’t buy the game for its single player campaign, so this doesn’t bother me too much, but I can’t get any satisfaction out of this product.

I can’t send the game back, you get a code that you activate when you try to install the game. Once that’s activated, it’s bound to your account. I can’t sell the game for the same reason. So, the normal consumer action of returning the product, not fit for purpose, is unavailable to me. I’ve seen logs of conversations with EA support, and they seem as useless as a chocolate fire-guard.

EA have taken some really horrible approaches to their software lately, all in the name of anti-piracy and whatever marketing goals they have. I don’t need everyone to see that I achieved the single worst score on a server, or that I couldn’t shoot my way out of a paper bag. I don’t need to share with someone that I managed to beat their time in a race. If Madden 2013 uses this sort of system, I know for sure I’m boycotting all things EA. Go back to making and publishing awesome video games, leave the online gaming arena to the guys at Valve. They would have made this far less painful for you EA, I have never had issues like this with games distributed through Steam.  I don’t care about all the fluff, I do care about being able to play a couple of hours of a game. Especially one that ranks as one of my favorites if I can get past the issues.