Please DO NOT use the Jedi JVCL JvCsvDataSet component. I wrote it and it's not safe.

Once upon a time I was a Jedi JVCL developer. I contributed JvCsvDataSet, to the Jedi JVCL project.

Unfortunately, I believe it has several internal design flaws and needs a total rewrite, and also may be incompatible with modern delphi Dataset field management.

I do not have time to go debug the horrible memory and pointer bugs that are lurking deep inside it, but I believe the component should be deprecated.

I wrote some of this code, some of it is based on very old code from a book written in 1998, and it's also been altered, both by me, and by other people, over its long history.   I think it should be moved into a legacy folder and removed from a future JVCL version if no longer maintained.

Like I am saying above, and I'm saying again, I don't have time to fix it, and I believe it's broken in Delphi 10.0, 10.1, 10.2, 10.3, and 10.4.

If it's being used in a little utility program fine, but if you're building your app around this component, please rethink that.  Please rip it out and use FireDac TFDMemData and write some other CSV import and export functions.

Thanks.

Delphi Seattle becoming almost unusable on Windows Build 1909

I have previously posted about fun adventures running insider preview builds of Windows 10, with Delphi.    Now, I no longer run insider preview builds because the problems are just not worth it.

However, as Microsoft believes your computer belongs to them once you put Windows 10 on it, you now have to deal with your "windows 10" (pronounced "Windows is like a box of chocolates, you never know what you're gonna get") system will daily, or weekly, break stuff and die randomly on you.

In the midst of a global panic over a real virus, I would like to add that I feel that Windows 10 is a virus.

I can't rely on Windows 10 not to :

1. Ship new stuff I don't want.

2. Turn back on options that I turn off, like windows defender realtime protection.

3. Break production binary applications, for which I have no source code, most importantly Delphi itself, which I need to do my job.

4. Work the same way on all the hardware I own.  One build of windows will be a disaster and crash repeatedly, but work fine on other hardware I own. Build 1903 worked on one of my PCs but not the other, and now build 1909 works on one of my PCs and not the other.

Most recently, I can no longer operate Delphi 10 seattle which is the version my employer's production codebases are relying on. (When you work in a team, by the way, one person can't just move versions, so if you suggest that I should move to a new delphi version, I'll just ignore you as we're going to do that when we all do that, as I belong to a team.)

Here are three of the fun problems that I'm seeing:

1. If ever you had to set up a new PC on a new windows build and get it going with Seattle, and Windows build 1909, good luck.  To get a preview of the fun you'll have, you could rename (god forbid you should ever ever delete) the registry BDS\17 registry key, Delphi can not create a valid new setup and start up without crashing. You get an access violation in coreide230.bpl.  If you have a valid backup of a working BDS\17 registry key, you can recover it.  Anyone using Seattle on Windows 10, please keep backups of your BDS\17 key under your Current User registry settings.

2.  (UPDATE:Fixed in recent Win10 quality update) Delphi is unable to build our projects without dying during the write DCU phase randomly. This was always bad on some peoples computers, but usually not on mine, and now I can no longer operate the IDE and compile or build the main projects I need to compile for my day job.

3. (UPDATE: Happens less in build  1909 than build 1903, but still a problem.) Delphi's debugger when reset, is hanging the Windows 10 1903 and 1909 kernel hard, on some systems, for some applications, mostly ones that seem to have certain DLLs loading and unloading in them.


I'll post solutions if I find them, but if I don't, I'm going to move into a Win7 VM for all development, because Windows 10 is not something you can trust.  What's most odd is I have a Win10 VM running build 1909 and this issue is NOT happening in the VM.

UPDATE:  On the latest quality update of Windows 10 build 1909, on the main machine that was having these issues, the problems with the bds.exe executing delphi compiler within the IDE  (point 2) is resolved.

UPDATE2: Still no workaround known for Point 1 other than to keep your working backups of registry in current user.  

UPDATE3: A new issue that is less grievous but still annoying has appeared.  The USB User Mode Driver Framework has started hanging. The main symptom is that all USB attached peripherals including external mouse and keyboard stop working for about 30 seconds while some core Windows user mode driver framework bits restart themselves.  I can trigger it by resetting the delphi debugger. Something about terminating the process being debugged, which may be holding onto a handle for some device driver related object somehow, seems to trigger a crash in a usb driver, which somehow then crashes the host process. 
 

UPDATE4: Although my main work machine is unaffected, build 1903 and build 1909 are "bricking themselves" (failing to boot) after something in the windows update process corrupts the boot time data storage area (BCD) of the Windows boot volume (Drive C).  This happens repeatably on certain Lenovo Thinkpad hardware. I suspect a kernel-mode panic during boot due to a driver bug in the chipset drivers.

Bare Minimum Expert and Wizard Demos

Bitbucket, instead of giving me free Mercurial hosting forever has decided to kill my Mercurial repos, of which I have hundreds.

I converted two little ones today and wanted to make a quick post because I think many people may have looked for these minimal wizard and expert examples and have had a hard time finding them.

Originally for the the Toronto Delphi user group I wrote some examples on how to start writing experts and wizards for Delphi:

http://www.tdug.com/2016/04/april-meeting-follow-up-2/

I have recently converted the sample code from mercurial to git repos, and they are still on bitbucket but have new links:



https://bitbucket.org/wpostma/helloworld_delphi_expert/src/master/ (git)

https://bitbucket.org/wpostma/helloworld_delphi_wizard/src/master/ (git)


The abridged notes on the above examples are:


1. You probably don't want to write a wizard.  You probably want an expert. But you may want to know what a wizard is, versus an expert, and you may conceivably find some thing that really is better as a wizard.  But mostly you probably want IDE experts if you want to add some feature to the Delphi IDE as a plug-in.

2. The Wizard example is probably the one you want to download if you only grab one of the above. But go grab both. Free. No warranty expressed or implied.  And no built in guarantee that I'll provide you with free technical support.  You probably want to join an OTAPI focused Delphi mailing list or web forum, if you want help and support writing OTAPI (open Tools API) DLLs.


Here are my notes on these code samples:

The recommended way to use the helloworld samples is to open them up and build them, then install them and try them, then poke around the code and find how an expert or wiard works by reading the samples/demos.

There’s not so much code there, and almost all of it is important, except you can ignore the castalia parser code which was put in there just to make it easier to get it integrated when you want to have a pascal parser library in a real world expert, as shown in the last of the three demos.

The first of our three demos, the hello world Wizard sample shows in Package (BPL) form, how to use the open tools api Wizard interface.

Wizards may seem easier to use but the limitations and annoyances of a BPL wizard are severe and it is not a recommended approach for most IDE plugin requirements.

The second demo, the hello world Expert sample shows in an Expert DLL form, how to use the most central open tools api  Expert  interfaces.  It shows how to export the registration function from the DLL, how to integrate with the IDE about box and IDE splash screen including a little icon that shows while Delphi starts up, and how to add a menu item in the main menu and add a menu item in the project right click menu, how to integrate with the IDE Insight feature, and how to make keyboard shortcuts (hotkeys) work.

Finally at the TDUG meeting, I showed a third demo that actually tries to do something interesting using the open tools APIs. Since the above helloworld expert is only a skeleton, and while useful as a starting place for someone who wants to write a wizard, doesn’t actually do anything that you might want to do and is certainly not something you would find you couldn’t live without, I wanted to make a demo that while not yet an invaluable tool, shows some of the potential for wizards. This one implements a custom parser, grabs the currently selected editor window and checks the code inside for two kinds of problems. The first is incorrect use of the WITH statement. All uses of WITH are incorrect, and should be killed with fire.  Secondly, it tries to line up your begin and end statements and see if they match up. If they don’t match up, it tells you. If your code is nicely formatted and free of With statements, it is pronounced good.

The Single Worst Mistake any Software Product Based Company Can Make

I am a long time follower and reader of Joel Spolsky. Way way back in 2000, he penned what has become a seminal blog post.

Things You Should Never Do, Part 1

It should be required reading for all leadership teams in all software companies, especially the small ones, because those are the ones most likely to commit Hari-Kari in the way described in the above post. I will quote it.

We’re programmers. Programmers are, in their hearts, architects, and the first thing they want to do when they get to a site is to bulldoze the place flat and build something grand. We’re not excited by incremental renovation: tinkering, improving, planting flower beds.There’s a subtle reason that programmers always want to throw away the code and start over. The reason is that they think the old code is a mess. And here is the interesting observation: they are probably wrong. The reason that they think the old code is a mess is because of a cardinal, fundamental law of programming:It’s harder to read code than to write it.This is why code reuse is so hard. This is why everybody on your team has a different function they like to use for splitting strings into arrays of strings. They write their own function because it’s easier and more fun than figuring out how the old function works.

Here's something I've seen happen time and time again in my career.

Manager Dave:   Steve, why did we ship version 3.2 and break all our customer's ability to do business again? I thought once we hired QA people that would stop happening.
Team Lead Steve (haggard, has had 3 hours sleep):   Um, Dave, regressions in 3.2 were caused by the requirement in that release to rewrite a core module of our system to please one of our customers, that broke all three hundred installed customers.
Manager Dave:  Can you explain to me why that is?
Team Lead Steve:  Goes on to explain the concept of technical debt, makes reference to configuration spaces (parameter 310 is on, parameter 311 is off, parameter 312 is set to X, parameter 313 is set to Y).
Manager Dave (cradling his head in his hands): So what do we do, Steve?
Team Lead Steve:  Well we could rewrite everything get rid of all the technical debt, refuse to add parameters to turn features on and off in 900 ways.

In my view, the thing that saves people from making the single worst mistake (the ground up rewrite, discussed in the blog post from Joel Spolsky) is that most people realize the economic disaster before they commit all their resources to it.  Wait, we can't ship any minor updates to our customers for THREE YEARS? We won't HAVE any customers in three years.

Another way the rewrite conversation comes up is when it's time to grow your team from four delphi devs to to five, and you can't find one that will move to Spooksville Indiana, and join your local team, and you aren't going to add remote team members. If you moved to C#, and rewrote, all your problems will be over, there are reams of unemployed C# developers everywhere, and you can probably grow your team to 20 people no problem, you just need to rewrite 9 million lines of code in C# that are currently in Delphi. 

There are still teams running on Delphi 7 and Delphi 5 because they can't figure out the complexities of moving up to modern unicode Delphi  who think they can't manage the unicode transition, but they can successfully rewrite everything in C#.

Note that there ARE times when I think you can rewrite, ground up, same language or different, same database or different. They are when all the following stars align:

1. When you can keep your existing team working on your classic product, and start a parallel effort,. and afford to develop both in parallel for the years to a decade it will take before the new product can replace the old.

2. When you don't just think you know your requirements, your designs and your use cases but you actually do know them.  Most of the time you have a confirmation bias telling you that you know this stuff, but guess what, you're probably wrong here.

The next thing, if the economics don't kill you, in a rewrite, that's gonna kill you is second system effect. Second System Effect is a term coined by Fred Brooks and is the title of one of the essays/chapters in the seminal book "The Mythical Man Month" that every senior developer or software team leader/manager should read.

Joel says the same thing thus:

It’s important to remember that when you start from scratch there is absolutely no reason to believe that you are going to do a better job than you did the first time. First of all, you probably don’t even have the same programming team that worked on version one, so you don’t actually have “more experience”. You’re just going to make most of the old mistakes again, and introduce some new problems that weren’t in the original version.

I have noticed that old code is scarred with bug fixes. And we don't like the look of it we wish it looked clean and rather like pseudo-code. The thing is the real world has lock conflicts, network timeouts, retries,  user errors, windows defender locking and even altering your on-disk files, disk performance and corruption issues, video card bugs,  USB driver glitches.   Need I go on?

The real world is a mess, and if your product works at all for your customers, it's amazing that you got that far. Don't blow it now.  Fix your bugs, and clean your messes up. And you better stop digging new holes (increasing technical debt) before you can expect to see the old technical debts get paid off.

One of the leading causes of technical debt is crazy features for ONE customer.   You could rethink when and where you add custom hacks for one customer who is loud and persistent, and perhaps could accomplish their goals another way without breaking your product in half.

Thinking carefully about Risk management, and careful heads-up planning is necessary to avoid killing your software products or your whole software company.

Don't rewrite your code ground up.

Sivv Open Source Projects by Jason Southwell

Just wanted to point out these open source projects written by Jason Southwell, who in addition to being a well known Delphi developer out there is also my boss! Hi!

Anyways, I use these libraries where we both work, and I think you should all check them out.


SIVV Open source is on bitbucket ...

A few example projects:

Chimera  Networking and JSON Library

Chimera project includes chimera.json, a very fast JSON library, and a bayeux (pubsub) network protocol  client, and other things.

CocinAsync

Networking/threading/async library. It recently grew some new "flux" like capabilities for event driven programming.

DuckDuckDelphi

A nifty duck-typing facility built over delphi RTTI facilities.  As the comments state in the project source:

//  Instead of:
//    if obj is TControl then
//      TControl(obj).Visible := True
//
//  You can simply call
//    obj.duck.setTo('Visible',True);

The call will do nothing silently instead of blowing up if you had done a bad runtime cast.  It's a nice pattern.



Discussion and questions on these components is at this site:

https://sivv.com/

Source Control, Version Control, Change Control, and the Underlying Disciplines Thereof

So, out there in the world of developers, Delphi and others, there are many schools of professionalism, around:

• Version Control Practices
• Continuous Integration Practices
• Release Management and Change Controls

Let's imagine there's a dial from 0 to 10, and at each number on the dial, there's a developer or team that is functioning at that Level of Versioning/CI/Release/Change practices.

0.  Dude with laptop, ships code from his laptop to customer pcs. 

1.  Dude had a bad day and lost data. Decides to up his game to using Zip files.  (For most of us old timers, I hope this day was back around 1991 for you.)

2. Dude sees the beauty of being able to check in, revert, commit.  Dude is saved and has seen the Version Control System light.  (This happened to you in about 1999, hopefully and your first version control system was Visual Source Safe, hopefully you are not still using that.)

3.  Dude realizes that checking in frequently is great.  Dude has not realized that Branching is useful, or is using a tool in which branching is not very useful (like Subversion).   Dude needs to learn new tools and practices, but doesn't want to.

4. Dude is member of a team of Developers.  This team works well together, so they have to know what branches are, and merges. None of them know what Continuous Integration is.  The team builds and ship from a designated lead developer PC.


5. Dude & Team had a bad day, and learned to love CI.    The CI system is a bit sad but hey, we've got one, and that makes our Kung Fu better than teams that don't.   No longer does The Dude ship code from his/her laptop to customers.

6. The Dude and Team, do CI, and honestly tried to start Unit Testing, but that never really worked.  There are millions of warnings and hints in the code.    When the code somehow builds without an error, we hand it to QA then ship it.   If you don't have dedicated QA people, you are still level 5, hire a QA, and have CI, and you're level 6. Congrats, you're probably above average in the Delphi world.

7.  The Dude and the Whole Team are conscientious about cleaning up hints and warnings and is working on getting code under unit test.   This is now better than about 60% of delphi shops.

8.   Dude and Whole Team Always know what was shipped to customers, as each version like Version 1.6.7.85080 is  known to correspond to a certain CI build which in turn corresponds to a known revision in version control.   All the executables are re-tagged with versioninfo updating tools during builds.  This is now better than about 70% of Delphi shops.

9. All the prior best practices are followed and also, All code that we build also passes tests or it doesn't go to human QA. These tests cover 10% or more of real bugs that QA would find. You are probably in the top 20 best Delphi shops in the whole world.


10. All the prior best practices are followed and also,All code that we build also passes tests or it doesn't go to human QA. These tests cover 80% or more of real bugs that QA would find.  You are a unicorn, and you probably don't exist, or you're working in Go or C++, not Delphi. 

So far, I'm not aware of ANY delphi shop that can claim to be in category 10, for even the top four largest codebases at your company.   Typically a few products are in good shape, and the others are pretty rough. From comparing notes with other Delphi professionals, most best of breed Delphi shops are probably in categories 6-9 above, all of which are Pretty Good to Really Good, in my opinion. 

But Pretty Good and Really Good is still basically much lamer than many Go, Ruby, and Java open source projects out there, that you can go inspect and see, are being built with levels of professional practices that far exceed what I am personally aware of within Delphi based teams.   If your Delphi teams blow away all the rest, please please write a blog post and explain how you do it.

Some truly impressive open source projects you can observe a whole higher level of practices going on include the C# roslyn compiler and the .net core open source project, the gitlab version control system (built in go, and ruby), Google Chrome,  Firefox, and SQLITE, which has a truly impressive set of unit tests and change control practices around its C/C++ core. 

I want to move on to the question of WHY the best practices matter.

 A lot of the principles that underly the best practices above should be obvious to most professional software developers, but sometimes these principles are left implicit,  and not spelled out.

Here are some principles that lie beneath the practices above.  I have seldom worked on a team that couldn't do something a bit better.  The best teams were always asking, "how can we improve?". The worst teams were always trying not to think about it. 

1.   Change Control Disciplines : We want to not make accidental changes, or even have people intentionally commit things, that were not wanted.   How are we going to minimize risk of accidental or just random developer "felt like doing something" and that ending up in our product without anyone being aware, or testing for it?

2.  Quality Control Disciplines:  How do we ensure we know the risk and the proper testing scope, for any release of software we work on?

3. Version Tagging Disciplines:   Does everything contain what it says on the tin?  Is version 1.0.3.10231 enough information to recover the exact source code set that was used? (If you used CI, and build 106 in jenkins, became installer-1.0.3.10231 , and you can see which git or svn revision that was, you can recover sources to any version in the field, and accurately patch it.)

4. Code Hygiene and Code Quality Disciplines : Do we fix warnings (especially the ones that lead to direct bugs, like uninitialized local variables) before we ship to customers?   Do we keep warnings and hints very close to zero so that really bad stuff doesn't hide "in the weeds" in plain sight?  Once all the real ugliness is out of the way, some teams graduate to Code Smells, and Refactoring, to get the architecture in a better place so the product can grow and change and not collapse under its own weight.  Code Hygiene and Code Quality disciplines are a never-ending asymptotic series of goals and ideals, but within those disciplines is a path out of drudgery and fire-fighting leading towards coding quality software that your customers will love. 

There are more details and things to say about the kinds of disciplines involved in the points above, that are not even talked about, there's a lot to be said about how to move from ball of mud (classic delphi) to TDD and well-factored code that is not a ball of mud.   Those are topics for another day.

New Idera/Embarcadero Forums

Folks who might have still been on the ancient Newsgroups might not have ever used the reboot of the Embarcadero Support Forums, and those are gone now, and have been replaced by some very modern and nice looking Embarcadero/Idera support forums which are located here:

https://community.idera.com/developer-tools/general-development/


My old community login did not appear to get moved over but it only took a second to get registered, and the site appears to be fast, responsive and easy to use.   I like it.