Saturday, May 21, 2016

Delphi Programmer Thinks about the Go Programming Language and Mandatory Source Code Organization

If you follow one of the usual tutorials for Go programming they will start by trying to dump a load of things you have to do on you.  This is perhaps something that you, as a long-time-Delphi geek have become inured to in your own environment.  Let us imagine a developer who has been given access to a source code repository or server, perhaps a big subversion server, and has no familiarity with the Delphi codebase at CompanyX, where CompanyX is basically every delphi-using company ever.  Let's make a quick list of the first tasks our developer would face:

  •  Setting up a working copy of source code so it builds, and so the forms open up without errors due to missing components.

  • Associating package set X required  to build version Y of product Z.

  • Setting up library paths that might be completely undocumented.

  • Individual  things done at company X like mapping a fake drive letter with SUBST, or setting up environment variable COMPANYX to point at some shared folder location.

  • At some companies they will just look at you blankly if you ask "can you build this entire product and every part of it from the command line on every developer's PC"?  Other companies have exactly ONE computer ( on which this feat is frequently possible.    Still others (the sane ones) have made the process so unspectacular, and merely reliable that they think the ones who gave you the blank look just haven't realized how insane they are yet.
  • At some companies it might be considered acceptable if the build scripts and projects and sources ASSUME you will always check your code out to C:\COMPANYX. When you want to have a second branch you simply clone and copy a tiny little 120 gigabyte VM and fire that up.

Has any of that ever seemed insane to you? It does to me.   And so when I look at new languages one of the things that I look for is if the problems above have been thought about and resolved in that language and its related tools, including its build system, if it has one, and its module system.

Go has been known to me for some years as a famously opinionated language, characterized by the removal of features that its designers felt were problematic in C++, and were thus removed from GO:

  • There are no exceptions in Go, only error returns, and panics.

  • There are no generics in Go.

  • There is no classic object oriented  programming with Inheritance, there is only composition, and there are only Interfaces, there are no base classes (because no inheritance).

  • The module structure is pretty much mandatory.  Here's me starting a brand new Go project from a command line, what is happening should be pretty clear to most geeks:

~/work> export GOPATH=~/work
~/work> export PATH=$PATH:$GOPATH/bin
~/work> mkdir -p $GOPATH/src/
~/work> cd $GOPATH/src/
~/work/src/> vi hello.go 

package main

import "fmt"

func main() {
        fmt.Printf("Hello, world.\n")

~/work/src/> go install
~/work/src/> hello
Hello, world.
What is the thinking process that goes into designing the module system, with the following structure:

I think the above has the benefit of being about as nice a structure as I could imagine.  The folder names above tell me even where on Github I might find this project.   Source code is now globally unique and mapped by these conventions so that I know where to find any public code I want.  If I want to use or bitbucket, or if it was stored on a private server inside my company named, I would move my code into different folders to make that choice clear.  For a language which is intended to be used in large systems, it's an appropriate design choice.  Let's contrast this with Perl or Python where the intended use starts with one to ten line scripts that are basically used for any kind of little task you can imagine, and where this kind of ceremony would be stupid.

I have worked in enough large code-bases in Delphi, and where any form of organization is accepted, it will inevitably become horribly complicated.

Let's briefly discuss the forms of folder/directory organization one might try to do in delphi:

* Ball Of Mud: Everything for one application is in one or some small number of directories - It is extremely common that one directory contains 90% of the files that are not third party components and that directories are only used to hold files not written here.  No sensible use of directories is made.  The source should all be in one directory with 10K files in it.  Usually ball of mud folder structure goes nicely with ball of mud source code organization inside your code files. That form with 5K lines of untestable business logic mashed into the form? That "controller" object that is more of a "god" object that direct references everything else tightly via USES statements? Ball of mud.

* MVC or MVVM: Views in their own folder. Models in their own folder. Controllers in their own folder.   Additional folders for shared areas, and per application area.  I've heard that this is possible, but I've never seen a Delphi codebase coded according to MVC.  Ideally, if you're going to do this, you also don't have your Views reference your controllers or even have access to the source folder where the controllers live.  Your models ideally don't reference anything, they're pure model classes and don't take dependencies on anything.

* Aspirational:    This is the most common condition of Delphi codebases.  There is some desire to be organized and some effort has been made, but it is fighting an uphill battle that may be unwinnable, because the barn door was already opened, and that cow of accidental complexity is already out and munching happily in your oat field.    You have a desired modular approach and it's expressed in your code, but every unit USES 100 other units, and your dependency graph looks like something a cat coughed up.

So given that I have seen large systems get like the above, I have a lot of sympathy for languages like C# where at least you can get your IDE and tools to complain when you break the rules, and I even have more sympathy for Java where namespaces are required on classes, and where the classes must live in directories which are named and hierarchically ordered.   In Go we have named modules, and the modules contain functions and can define interfaces, but they're not really Classes like in Java.  But the idea of order and organization has been preserved as important.  In Delphi we have the ability to use unit prefixes which are weaker than true Namespaces but still potentially useful.  In Delphi but most code I have seen does not attempt to adopt them.  It seems to me that having a codebase that uses unit prefixes, and that has source organized into these folders is a worthy future goal for a Clean Delphi Codebase, but existing legacy codebases are all we have, and so getting there is not something I'm going to hold my breath on.   One has to have practical achievable plans, and not tilt at windmills.

My first reaction to Go's requirement to use a fixed structure was predictably the same reaction that I had when I first realized the horrors of Forced Organization that java was forcing on me when I first tried Java in 1996.  Now, it's 20 years later, and I think we can say that Java's designers were right.   Java has proven to be especially useful in constructing extremely large systems.

The go package dependency fetching system (go get X) works precisely because of this forced organization, and it's all very well thought out.  There's NO reason that a clever Delphi programmer couldn't learn the lessons of how GOPATH and how go get and go install work and use that to fashion a guaranteed well organized and maintainable and clean delphi codebase, incrementally, by a phased approach.

You don't gain much if you close the barn door after the cow's got out, and you can't stop everything and rewrite, but if you can build some tools to help you tame accidental complexity gradually, you can restore order, over time, while you work on a system.

What goals might you start with?  I'm not going to tell you.    All I'm going to do is say that if your brain lives in a box that has a begin at the beginning, and an end at the end, and you can't read and think outside that box, you're sadly limiting yourself as a developer.  Becoming a better developer (in any language) requires what the old timers even older than me called a "Systems  Approach".  A view of what you build, and your project and its goals that is larger than your daily work, longer in scope than whatever form of agile "sprints" you're doing, and which has a sustainable high quality engineering methodology behind it.

You can't build that kind of mentality in at a language level (in Go or Java or Pascal), but I think it does help to have the bar set progressively higher as you can, so that once code becomes cleaner and more maintainable, there is at least the potential to detect when someone has made things worse.

Thus far we have seen many programmers throw up their hands at the 5 million line big-ball-of-mud projects and consider rewriting it from the beginning.  My feeling is that the bad patterns in your brain are still there, and if you rewrite it all in the same language or a different one, you're going to make all those mistakes again, and some new ones, unless you start learning some ways to approach system design that promotes clean decoupled programming.   Studying and research phases are required. Do not race to reimplement anything, either in Pascal or any other language.  Spend time and sharpen your sword. And remember Don Quixote.


  1. Well said, thanks for this reality check on what we Delphians call "work" (as usual)

  2. You can do BBM folder design in go, once you are in your own "private" folders.
    The fact that there is no forced folder hierarchy in Delphi does not forbid you do setup one.
    And IMHO you should - as you clearly stated here.
    It is up to company's software design people to setup simple and efficient rules.
    We usually let DDD "clean architecture" be reflected in our source code tree.

  3. What would be useful to add to Delphi is:

    * True namespaces, and package structures where the namespace matches the module names, that could be enforced, even when you don't use runtime bpls. This is incredibly useful in C# and Java.

    * A nuget-like system with ability to have our own private feeds (like myget). GetIt is at a very early stage, but will hopefully evolve in that direction.

  4. I fully agree with your concerns.

    Simply make the use term folder sensitive, so that I can write code like:

    Delphi can use absolute folders already in the project source code and rename it,
    so how complicated can that be to implement in the IDE/compiler ?
    Would not break any existing code but get me free to place units how I want.


  5. Hey Rollo, check my article again. Having D:\1234 placed in source code DIRECTLY is already currently allowed in the .dpk and .dpr file in Delphi and this is in my view a BAD thing. One we can not change now, but it should never have been allowed. Second thing we can do now is have library paths search and find code that belongs to the Application (abuse of library paths). This is a bad thing and one we can not fix now. Third thing we can do now which is bad is use project search paths to make everything find everything. This is a bad thing we can not fix now. I don't think you understood my perspective, because the problem here is once a tool allows something you can never un-allow it. It is for this reason that language designers have to NOT add new features when people like you and me ASK for them. And when you and I build systems we need to find the areas where there's a mess and try to figure out how to avoid it by discipline and care. Coupling (direct using) is a BAD thing. Please study some more and then realize I'm not saying "let people do what they want is good". I'm saying it's BAD.