Wednesday, April 14, 2021

Rant: Git is not a version control system. But you're going to use it as one anyways.

I can hear you saying it.

" You're wrong, Delphi Code Monkey, Git is a version control system.".

Hear me out.

Git is not a version control system, it's a directed acyclical graph management system that happens to be possible (sometimes) to do version control with.

Every time you find something Git can't do, that it ought to do, for all of the billion people who probably use it and know it well,  if you keep searching for ways, however torturous, to handle your issue, there is probably a way.

There is stashing, and submodules, branching, and merging, there are cherry picks, and there is rebasing, there are branching models and workflows like Git Flow.        There's recursive submodule hell.  There's local git cached state hell.  There's ignore hell.   There's Git's unix roots are showing on windows hell.
There's git config hell.    There's unable to check out a branch because of local changes, and yet git status shows no files have changed, hell.   If I sat here and thought back through the years of horrible things Git has done to me, one by one, I would probably scream.

There are a thousand fresh hells waiting for you.  A thousand inscrutable (but usually googleable) error messages.

Here's the worst of the Git hells.  Git lockin hell.  Once you choose Git,  well, it's Hotel California.  You can "check out" but you can never leave.   The world of software development is now, and probably always will be locked into some version of Git or other.

Git is not actually (much) of a version control system, but you're going to use it like it was, and you're going to learn to like it.


Update:  The point of this rant (and it mostly is just a ranty rant) isn't that you SHOULD NOT use Git, it's that the nature of Git is to expose all the guts to you, and to blow up with errors that are because the internals are in a bad place and Git leaves it for you to put Humpty Dumpty back together again,  and that you are going to do that a LOT when you live and work all day, especially in a team of 5+ people, with Git.    Perhaps if you are saying to yourself, "I don't have all these problems that other people talk about", it could be because (a) you're not using submodules, (b) you're not working with teams of 5+ comitters, and (c) you don't use a lot of branching, merging, rebasing, cherry-picking nor complex workflows like git-flow.



Friday, October 23, 2020

A few quick tips about Downloading Delphi 10.4.1 (sydney) and all future Delphi Releases for current active customers.

Probably this is old news but I didn't see the news whenever it hit, and so I'm posting this here in hopes that other people who hadn't realized that EDN/CodeCentral is no longer where you get your Delphi downloads from, will Get The Memo.   Did you Get the Memo?



 1.  Don't bother trying to find 10.4.1 Sydney in the old EDN products portal, the one with the url starting cc.embarcadero.com/reg/rad_studio.  10.4.1 isn't in there.  I wouldn't be surprised if this site went away at some point. 

2. If you have an active subscription the place to get 10.4.1 Sydney is from the My Embarcadero Portal using the same login credentials as your active subscriptions and your licenses are associated with.  https://my.embarcadero.com/

3.  Even very old stuff like Delphi 7 isos is now on the new site. It does seem that the same file pile of your purchases from the Borland, or CodeGear, or the pre-Idera Embarcadero era, are all on there, and there's no loss of continuity, and as far as I can see, there's no reasons to complain about the change, it seems good.

It's nice to see that the ancient EDN stuff is finally being rebuilt.  I still have some complaints about the download and install experience. For example, why isn't TeeChart available both from GetIt instead of only as a one-shot-miss-it-and-it's-gone a checkbox during initial install?  




Wednesday, July 22, 2020

Git with Delphi: The Basics

I wrote this blog post as I ran into a Delphi coder who was using some non-verson-control tool like Zip or DropBox to store their code.  Now I'm not saying you shouldn't ever store code in zip files or upload code to dropbox, but that should not be your source of truth.  This is a highly opinionated fly through of the basics that a non-git user should be aware of.

WHY  GIT?   GIT IS THE STANDARD FOR SOURCE CONTROL

What is source control?  Source control means having a single  SOURCE OF TRUTH for what your code is.    Ideally that single source of truth should be on a private or public repository hosted on a git server, like bitbucket, or gitlab.

Why GIT?  There is literally nobody policing this, if you're a solo developer, pick whatever version control system you want, but if you pick anything other than Git, you're going to eventually probably  regret it.   There are network effects (an economics term, really) here. I'm not going to explain network effects. If you have to google that, go ahead.

What is a source of truth?  Git provides a source of truth because all git commits have a "revision hash", a fingerprint that marks them uniquely.  Git provides a way to say, for sure that if you have git commit A939B9.... (more digits omitted), you know exactly what content every single file in your produce source code contained.   In order to be really sure about it you should actually also be using Continuous Integration instead of building things on your developer PC.

What is CI (continuous integration)?   Another reason to use git is so that you can have your code built automatically whenever you change it and have that build happen in a controlled way, in a controlled and repeatable environment so that you actually know what you sent your customers, if you have customers.  If you are working for a company that sells software, and the software binaries you put on customer PCs were built on your own desktop computer, and not on a build server, and your company is more than just one person, you also should be using Git, so you can get ready to be building your software from an automated build (CI) environment.

What are these unit and integration tests things people talk about? You have those right.  If you have Git, and CI, and unit tests, you can be running those things every time your code builds. Are you beginning to see possibilities unfolding like flowers?  If you are a single developer, having CI and unit testing, and continuous unit testing on every git commit, can really help you.

How do I get started? The rest of this post lays out a basic plan for getting started with Git.

Wait I have objections...    Post them in the comments. I prefer subversion is an opinion, not an objection, but feel free to tell me that if you like. 

Step 1.  Sign up for an account on bitbucket.

Step 2. Download Source Tree. (The easiest Git client to learn, not the fastest, or most powerful, and not one you will stay with forever, SourceTree is good at holding new user's hands and getting them some critical early experience with version control, especially if you haven't used version control before and are coming from a background of using Zip or Dropbox to "back up"  your code.)

Step 3.  Create a dummy repository in bitbucket and clone it with source tree.  Follow their tutorials.

Step 4.  Create a delphi application with File -> New Application and commit that source code for the brand new main form right after the first time you save the project and main form and give them the NAME they are going to have for a while.   

Step 5. Using your  git GUI, Add and Commit the files to the local git repository. You have only now been working with your own files on your own hard drive so far.

Step 6. Push your commits up to the repo you created in bitbucket.  Log into the bitbucket website and find the code on your bitbucket account in the repositories page.  

Step 7.  Go to a different computer (if you can) or a different folder on the same computer (if you only have one) and clone the repo to that location.  Observe for yourself that Bitbucket is like a Transporter for Code.  The code on my computer is now code on another person's computer.  Yet another use for Git. Code transporter.  Team coding enabler. Backup system.   Historical log of what your code was over time.  And a thousand other things.  Once you have internalized and understood the value of even 5% of what a good version control tool can do for you, you will never work without one.

Step 8.   Make the application do something.  Fibonacci sequences.  Sorting interview question. Anything.   Once it works, Commit the working code, explain the code you wrote in your commit messages always. Get into good habits to start with so you don't have to learn the hard way.  Now mess the code up on purpose. Now using your Git Client (SourceTree or others) find the change and DO NOT COMMIT it, instead revert it. You have now observed another use for Git.  The Time Machine.  Any time you move your code in a way that makes it worse, you can just revert the change before you commit it.     

Step 9.   Make a bad change and commit it even though you know you just broke your app. Now find the way to revert a commit (get your code back to the way it was before the bad commit).   I'm not going to tell you what the command is called, because I want you to figure it out for yourself.   So git can not only be a time machine BEFORE BAD COMMITS it can also be a time machine to get you back code after bad commits.


Step 10. Now learn about .gitignore.  Do this before you import any real projects into git.  Find out what .gitignore is and how to write expressions into it like *.dcu and *.local.   Do not commit binaries (exes, dcus, dcps, bpls) into Git, unless you really really have to.  In general, you should NEVER EVER commit a DCU for a .PAS file that is in the repo as you should be BUILDING that .DCU file from source.  If your projects are badly configured (DCU Output folder is NOT set) then get used to setting the DCU Output folder option in Delphi to a sensible default for all your projects. Mine is .\dcu\$(Platform)\$(Config)   so that I can have win32, win64, and even other non windows platforms all build and not have my dcus get mixed up.     .gitignore is the place to code these dcu and exe output folders in so that when you add your source code into a git repo you do not commit the exes.   If you really really want to store your ssl dlls in git, nobody is going to come and arrest you, but please for the love of the flying spaghetti monster, learn what .gitignore is and use it. Don't commit DCUs. Sermon endeth.

Step 11. After you understand git ignore then take your source code folder and do a git init there, from source tree.  Add things in small chunks. Not 10K files in one commit. Add one top level folder that contains less than 50-80 megabytes of content, at a time, if you can.  Trust me you'll be glad you did this, if you have a 1 gigabyte project, please do not add 1 gigabyte of files in a single commit.  After each commit, add more files until your repo is built.   If you were previously using mercurial or subversion there are tools to keep your subversion and mercurial history and convert over to git.  I'm not going to cover them but I know they exist and that they work as I've used a variety of them.  I generally don't bother. I just start over with new history in git, and I go back and look at the old version control systems to see old versions.

Step 12.  Get used to comitting all your code after you write it and writing GOOD COMMIT MESSAGES. A good commit message is a message that explains the purpose of your changes.   If you are lazy here, your future self will hate you. You don't need that kind of rejection in your life. WRITE GOOD COMMIT MESSAGES NOW.

Step 13. Push after every commit until you have made it a habit.

Step 14. Delay using and learning about submodules, branching, tagging, and other topics in Git until everything I have said above is well ingrained and is part of your habits.  Learn to use Diff tools, learn to merge, before you learn to branch.     Imagine you learn to drive a car but you concentrated on learning how to drive fast before you're very good at steering and braking.     Steering and braking are the ways you stay out of trouble.  Merging, and diffing are ways to get out of trouble. Submodules, and branching are ways to get into it.  Until you have the tools to sort yourself, just don't make the crazy advanced messes. Don't try to deploy and manage a Gitlab instance at cluster scale when you don't even know how to Diff two files, compare them, and solve merge conflicts.

I believe the above will be helpful as I've written it, but I reserve the right to amend the advice above if anyone can point out ways in which the above might lead a new Git user astray. The happy path, of a contented Git user, is a gradual growth in competence with a powerful tool.

Can git be frustrating? Yes. Is it complicated?  Yes.  Is it worth it to learn it? Also Yes.

Are most of us going to do things wrong, and learn everything the hard way? Yes, because you are human. It's okay. 






Friday, May 29, 2020

Please DO NOT use the Jedi JVCL JvCsvDataSet component. I wrote it and it's not safe.

Once upon a time I was a Jedi JVCL developer. I contributed JvCsvDataSet, to the Jedi JVCL project.

Unfortunately, I believe it has several internal design flaws and needs a total rewrite, and also may be incompatible with modern delphi Dataset field management.

I do not have time to go debug the horrible memory and pointer bugs that are lurking deep inside it, but I believe the component should be deprecated.

I wrote some of this code, some of it is based on very old code from a book written in 1998, and it's also been altered, both by me, and by other people, over its long history.   I think it should be moved into a legacy folder and removed from a future JVCL version if no longer maintained.

Like I am saying above, and I'm saying again, I don't have time to fix it, and I believe it's broken in Delphi 10.0, 10.1, 10.2, 10.3, and 10.4.

If it's being used in a little utility program fine, but if you're building your app around this component, please rethink that.  Please rip it out and use FireDac TFDMemData and write some other CSV import and export functions.

Thanks.

Tuesday, March 24, 2020

Delphi Seattle becoming almost unusable on Windows Build 1909

I have previously posted about fun adventures running insider preview builds of Windows 10, with Delphi.    Now, I no longer run insider preview builds because the problems are just not worth it.

However, as Microsoft believes your computer belongs to them once you put Windows 10 on it, you now have to deal with your "windows 10" (pronounced "Windows is like a box of chocolates, you never know what you're gonna get") system will daily, or weekly, break stuff and die randomly on you.

In the midst of a global panic over a real virus, I would like to add that I feel that Windows 10 is a virus.

I can't rely on Windows 10 not to :

1. Ship new stuff I don't want.

2. Turn back on options that I turn off, like windows defender realtime protection.

3. Break production binary applications, for which I have no source code, most importantly Delphi itself, which I need to do my job.

4. Work the same way on all the hardware I own.  One build of windows will be a disaster and crash repeatedly, but work fine on other hardware I own. Build 1903 worked on one of my PCs but not the other, and now build 1909 works on one of my PCs and not the other.

Most recently, I can no longer operate Delphi 10 seattle which is the version my employer's production codebases are relying on. (When you work in a team, by the way, one person can't just move versions, so if you suggest that I should move to a new delphi version, I'll just ignore you as we're going to do that when we all do that, as I belong to a team.)

Here are three of the fun problems that I'm seeing:

1. If ever you had to set up a new PC on a new windows build and get it going with Seattle, and Windows build 1909, good luck.  To get a preview of the fun you'll have, you could rename (god forbid you should ever ever delete) the registry BDS\17 registry key, Delphi can not create a valid new setup and start up without crashing. You get an access violation in coreide230.bpl.  If you have a valid backup of a working BDS\17 registry key, you can recover it.  Anyone using Seattle on Windows 10, please keep backups of your BDS\17 key under your Current User registry settings.

2.  (UPDATE:Fixed in recent Win10 quality update) Delphi is unable to build our projects without dying during the write DCU phase randomly. This was always bad on some peoples computers, but usually not on mine, and now I can no longer operate the IDE and compile or build the main projects I need to compile for my day job.

3. (UPDATE: Happens less in build  1909 than build 1903, but still a problem.) Delphi's debugger when reset, is hanging the Windows 10 1903 and 1909 kernel hard, on some systems, for some applications, mostly ones that seem to have certain DLLs loading and unloading in them.


I'll post solutions if I find them, but if I don't, I'm going to move into a Win7 VM for all development, because Windows 10 is not something you can trust.  What's most odd is I have a Win10 VM running build 1909 and this issue is NOT happening in the VM.

UPDATE:  On the latest quality update of Windows 10 build 1909, on the main machine that was having these issues, the problems with the bds.exe executing delphi compiler within the IDE  (point 2) is resolved.

UPDATE2: Still no workaround known for Point 1 other than to keep your working backups of registry in current user.  

UPDATE3: A new issue that is less grievous but still annoying has appeared.  The USB User Mode Driver Framework has started hanging. The main symptom is that all USB attached peripherals including external mouse and keyboard stop working for about 30 seconds while some core Windows user mode driver framework bits restart themselves.  I can trigger it by resetting the delphi debugger. Something about terminating the process being debugged, which may be holding onto a handle for some device driver related object somehow, seems to trigger a crash in a usb driver, which somehow then crashes the host process. 

 

UPDATE4: Although my main work machine is unaffected, build 1903 and build 1909 are "bricking themselves" (failing to boot) after something in the windows update process corrupts the boot time data storage area (BCD) of the Windows boot volume (Drive C).  This happens repeatably on certain Lenovo Thinkpad hardware. I suspect a kernel-mode panic during boot due to a driver bug in the chipset drivers.




Monday, December 23, 2019

Bare Minimum Expert and Wizard Demos

Bitbucket, instead of giving me free Mercurial hosting forever has decided to kill my Mercurial repos, of which I have hundreds.

I converted two little ones today and wanted to make a quick post because I think many people may have looked for these minimal wizard and expert examples and have had a hard time finding them.

Originally for the the Toronto Delphi user group I wrote some examples on how to start writing experts and wizards for Delphi:

http://www.tdug.com/2016/04/april-meeting-follow-up-2/

I have recently converted the sample code from mercurial to git repos, and they are still on bitbucket but have new links:



https://bitbucket.org/wpostma/helloworld_delphi_expert/src/master/ (git)

https://bitbucket.org/wpostma/helloworld_delphi_wizard/src/master/ (git)


The abridged notes on the above examples are:


1. You probably don't want to write a wizard.  You probably want an expert. But you may want to know what a wizard is, versus an expert, and you may conceivably find some thing that really is better as a wizard.  But mostly you probably want IDE experts if you want to add some feature to the Delphi IDE as a plug-in.

2. The Wizard example is probably the one you want to download if you only grab one of the above. But go grab both. Free. No warranty expressed or implied.  And no built in guarantee that I'll provide you with free technical support.  You probably want to join an OTAPI focused Delphi mailing list or web forum, if you want help and support writing OTAPI (open Tools API) DLLs.


Here are my notes on these code samples:

The recommended way to use the helloworld samples is to open them up and build them, then install them and try them, then poke around the code and find how an expert or wiard works by reading the samples/demos.

There’s not so much code there, and almost all of it is important, except you can ignore the castalia parser code which was put in there just to make it easier to get it integrated when you want to have a pascal parser library in a real world expert, as shown in the last of the three demos.

The first of our three demos, the hello world Wizard sample shows in Package (BPL) form, how to use the open tools api Wizard interface.

Wizards may seem easier to use but the limitations and annoyances of a BPL wizard are severe and it is not a recommended approach for most IDE plugin requirements.

The second demo, the hello world Expert sample shows in an Expert DLL form, how to use the most central open tools api  Expert  interfaces.  It shows how to export the registration function from the DLL, how to integrate with the IDE about box and IDE splash screen including a little icon that shows while Delphi starts up, and how to add a menu item in the main menu and add a menu item in the project right click menu, how to integrate with the IDE Insight feature, and how to make keyboard shortcuts (hotkeys) work.


Finally at the TDUG meeting, I showed a third demo that actually tries to do something interesting using the open tools APIs. Since the above helloworld expert is only a skeleton, and while useful as a starting place for someone who wants to write a wizard, doesn’t actually do anything that you might want to do and is certainly not something you would find you couldn’t live without, I wanted to make a demo that while not yet an invaluable tool, shows some of the potential for wizards. This one implements a custom parser, grabs the currently selected editor window and checks the code inside for two kinds of problems. The first is incorrect use of the WITH statement. All uses of WITH are incorrect, and should be killed with fire.  Secondly, it tries to line up your begin and end statements and see if they match up. If they don’t match up, it tells you. If your code is nicely formatted and free of With statements, it is pronounced good.

Tuesday, October 1, 2019

The Single Worst Mistake any Software Product Based Company Can Make

I am a long time follower and reader of Joel Spolsky. Way way back in 2000, he penned what has become a seminal blog post.

Things You Should Never Do, Part 1

It should be required reading for all leadership teams in all software companies, especially the small ones, because those are the ones most likely to commit Hari-Kari in the way described in the above post. I will quote it.

We’re programmers. Programmers are, in their hearts, architects, and the first thing they want to do when they get to a site is to bulldoze the place flat and build something grand. We’re not excited by incremental renovation: tinkering, improving, planting flower beds.There’s a subtle reason that programmers always want to throw away the code and start over. The reason is that they think the old code is a mess. And here is the interesting observation: they are probably wrong. The reason that they think the old code is a mess is because of a cardinal, fundamental law of programming:It’s harder to read code than to write it.This is why code reuse is so hard. This is why everybody on your team has a different function they like to use for splitting strings into arrays of strings. They write their own function because it’s easier and more fun than figuring out how the old function works.

Here's something I've seen happen time and time again in my career.

Manager Dave:   Steve, why did we ship version 3.2 and break all our customer's ability to do business again? I thought once we hired QA people that would stop happening.
Team Lead Steve (haggard, has had 3 hours sleep):   Um, Dave, regressions in 3.2 were caused by the requirement in that release to rewrite a core module of our system to please one of our customers, that broke all three hundred installed customers.
Manager Dave:  Can you explain to me why that is?
Team Lead Steve:  Goes on to explain the concept of technical debt, makes reference to configuration spaces (parameter 310 is on, parameter 311 is off, parameter 312 is set to X, parameter 313 is set to Y).
Manager Dave (cradling his head in his hands): So what do we do, Steve?
Team Lead Steve:  Well we could rewrite everything get rid of all the technical debt, refuse to add parameters to turn features on and off in 900 ways.

In my view, the thing that saves people from making the single worst mistake (the ground up rewrite, discussed in the blog post from Joel Spolsky) is that most people realize the economic disaster before they commit all their resources to it.  Wait, we can't ship any minor updates to our customers for THREE YEARS? We won't HAVE any customers in three years.

Another way the rewrite conversation comes up is when it's time to grow your team from four delphi devs to to five, and you can't find one that will move to Spooksville Indiana, and join your local team, and you aren't going to add remote team members. If you moved to C#, and rewrote, all your problems will be over, there are reams of unemployed C# developers everywhere, and you can probably grow your team to 20 people no problem, you just need to rewrite 9 million lines of code in C# that are currently in Delphi. 

There are still teams running on Delphi 7 and Delphi 5 because they can't figure out the complexities of moving up to modern unicode Delphi  who think they can't manage the unicode transition, but they can successfully rewrite everything in C#.

Note that there ARE times when I think you can rewrite, ground up, same language or different, same database or different. They are when all the following stars align:

1. When you can keep your existing team working on your classic product, and start a parallel effort,. and afford to develop both in parallel for the years to a decade it will take before the new product can replace the old.

2. When you don't just think you know your requirements, your designs and your use cases but you actually do know them.  Most of the time you have a confirmation bias telling you that you know this stuff, but guess what, you're probably wrong here.

The next thing, if the economics don't kill you, in a rewrite, that's gonna kill you is second system effect. Second System Effect is a term coined by Fred Brooks and is the title of one of the essays/chapters in the seminal book "The Mythical Man Month" that every senior developer or software team leader/manager should read.

Joel says the same thing thus:

It’s important to remember that when you start from scratch there is absolutely no reason to believe that you are going to do a better job than you did the first time. First of all, you probably don’t even have the same programming team that worked on version one, so you don’t actually have “more experience”. You’re just going to make most of the old mistakes again, and introduce some new problems that weren’t in the original version.

I have noticed that old code is scarred with bug fixes. And we don't like the look of it we wish it looked clean and rather like pseudo-code. The thing is the real world has lock conflicts, network timeouts, retries,  user errors, windows defender locking and even altering your on-disk files, disk performance and corruption issues, video card bugs,  USB driver glitches.   Need I go on?

The real world is a mess, and if your product works at all for your customers, it's amazing that you got that far. Don't blow it now.  Fix your bugs, and clean your messes up. And you better stop digging new holes (increasing technical debt) before you can expect to see the old technical debts get paid off.

One of the leading causes of technical debt is crazy features for ONE customer.   You could rethink when and where you add custom hacks for one customer who is loud and persistent, and perhaps could accomplish their goals another way without breaking your product in half.

Thinking carefully about Risk management, and careful heads-up planning is necessary to avoid killing your software products or your whole software company.

Don't rewrite your code ground up.