On Data Files and Configuration in Delphi Programs

On a Delphi programmer's group on Facebook the other day, someone asked a question about configuration files. They were going to use an INI file to hold their data, and someone suggested the Registry.  I chimed in that I find INI files superior to Registry files, with the provision that perhaps one exception to that might be if your application needs to configure where its textual configuration files are. The reasons why textual configuration files (in json or ini format) are better than the registry are:

• Because it's easier to back up and restore application state.
• Because for your support people, reading and inspecting a text file is easier than finding and inspecting the registry, which is in the end like one enormous INI file that describes the state of your entire system.
• Because even if the system won't boot any more, it's easy to get that configuration data and copy it off that machine, just by booting from a live CD image like GPARTED or by slapping a completely dead PC's disk in another machine after the motherboard died.  In my career this has turned out to be a good choice when customers were told to make a backup and they didn't. Data recovery has saved my customer's bacon when they didn't do the backups they were taught and told to do.
• In my automated bug report submission systems, using MadExcept (or you could use EurekaLog), I find it helpful to attach log files (showing what happened to the app recently, on the client and the server side) and even some of the configuration files, so you know what options the user has configured.
• In some domains we might even check the configurations into version control, or keep backup copies of configurations. It's harder to do that with a registry.

Some interesting technical questions that remain:

• Where should data files, including configuration files be stored?  For state which is different for each user, I believe in the user's appdata folders, either Local or Roaming, under a Subfolder named by your company name, with a Subfolder under that for your product name.  
  Similarly folders under c:\ProgramData should be used for global data that is not different for each user who logs in. So for AwesomeSoft's product SuperTool, on my computer, C:\Users\warren\AppData\Local\AwesomeSoft\SuperTool is used. If the installer version is known and this folder are backed up, then your entire application state can be backed up and restored without even backing up the program files folder.  Perhaps a local database or data file might also automatically live here if it hasn't been configured and moved elsewhere.
• Even back in Windows XP era (2002) Microsoft has given clear guidance that non-system privileged processes should not have write access to the Program Files folders.  In order to preserve system stability, only the installer, which operates in an Elevated privilege level, and is started by a person with appropriate system administrative role, should be able to write to the program files folder.  I consider hacking NTFS permissions on Program Files to be a bad practice, on end user machines.   What I think is kind of hilarious though is that in 2016, Microsoft SQL Server 2016, still defaults to primary SQL Table data file storage under Program Files folders, instead of moving that global state data to Program Data where Microsoft's own guidance would have suggested they put it.   What do you do when even Microsoft's left hand doesn't follow Microsoft's right hand?  I would say, you do what is right, what is a solid engineering principle.  Protect users and their systems by separating your binary code and non-user-defined static state (maintained by installers and patch updaters) from your user defined static state (your data folders).
• When if ever are binary configuration files appropriate? I would say, only when the problem domain or technical requirements cannot be met with a text file. For example, if the configuration files contain a million entries and really just needs to be a key-value store. In that case, I would say, you should use a proven binary container like SQLite, and not invent your own shabby one, or use some unmaintainable binary blob technology that you found on the internet, like that b+Tree algorithm you found somewhere randomly on the internet.  Binary files are opaque, testing binary file reads/writes is a harder problem, and inspecting that binary file for damage becomes a difficult task, unless you choose something like SQLite that already has tests, and file integrity checking already built.

Ask that "Dumb" Question, and No You Aren't Dumb. Or maybe if you are, that's OK.

I found this blog post called "Being a 'Dumb' Girl in Computer Science" incredibly awesome, so I have to share it with everybody.  As someone who wants to see more women in STEM (Science, Tech, Engineering, Math) in school and in the workforce, I'm encouraged to see how brave and resourceful, and actually totally not dumb, some of the young women in university these days really are.

University was a long time ago for me, but there are some things that haven't changed whether it's school, or the workforce.  A really big problem I have seen in the workplace, is when people don't ask questions.  We leave people behind. We don't get buy-in, we don't have a shared understanding of the problem domain. We all lose. The company loses. The customer loses. The team is less effective. The software product quality suffers.

Ask the dumb questions. They are not dumb questions after all.  One of the comments on the issue puts it perfectly:

20-year veteran here, CTO of my company, yadda yadda yadda. It’s my publicly-proclaimed policy to ask the “dumb” questions. Get those answered and there seldom remain any “smart” questions to be asked. Skipping the “dumb” questions usually means asking irrelevant ones. Beware the “geniuses”—they write the worst code. At first you’ll feel dumb not understanding it, but you’re blaming the victim. Simple, easily-understood code is the product of hard work more than cleverness. Mysterious code is a menace. I hope you keep your beginner’s mindset. You’ll learn far more and be a better team player.

This fits perfectly with something that Confucius said a long time ago:

If I am walking with two other men, each of them will serve as my teacher. I will pick out the good points of the one and imitate them, and the bad points of the other and correct them in myself.

Don't be the person who thinks he's a "genius" and has nothing more to learn. Learn from Cat that you should ask the question.   Learn how not to be the people in her class that made her feel bad for asking the question.  Correct in yourself the pride and smugness you find. You are like a full cup, and you can receive nothing more.  Begin with the knowledge that you know nothing, feel free to call yourself "dumb", just don't call other people dumb.  And know that if you learned something, it wasn't a dumb question at all.   Maybe someone else at your workplace or your school wanted to ask that question and didn't understand at all either, and needed you to step forward.

I went through a phase where I preferred to write the most inscrutable code, and considered that great.  If people can't make powerful concepts simple, and build useful systems out of understandable parts, and share knowledge, then our schools are failing, and our companies will fail soon after.  Ask the question if you have it. Give the answer if you have it.  The knowledge economy is a gift economy.

Innovation: You're Doing It Wrong

I'm reading a book called The Idea Factory: Bell Labs and the Great Age of American Innovation by Jon Gertner. (2012).  

Bell Labs, and its scientists were neither perfect, nor morally impeccable, and this book does not feign the sort of pseudo-scientific hagiography that the subject matter often inspires in technophiles.  But the book does contain some kernels of what might be the nearest thing to magic that Science itself permits.  The human process of invention, the Eureka moment, provides an example that the economic value of the work we do does not correlate highly to the appearance of business.  In fact, much the reverse is true.  Let me quote a bit:

"Around the time [director] Kelly was giving his speech to the phone company executives [on the fact of Bell Labs being founded not upon magic but hard science], a metallurgist named Bill Pfann was mulling over how to raise the purity of germanium to improve it further for transistor production. Pfann had returned to his office after lunch -- 'I put my feet on my desk and tilted my chair back to the window sill for a short nap, a habit then well established,' he recalled. He had scarcely dozed off when he suddenly awoke with a solution. 'I brought the chair down with a clack I still remember,' he said. Pfann envisioned passing a molten zone - a coil of metal in effect creating a superheated ring - along the length of a rod of germanium; as the ring moved it would strafe the impurities out of the germanium. Kelly would eventually tell people that Pfann's idea was called 'zone refining'.... one of the most important inventions of the past 25 years. Kelly didn't tell people it resulted from a man sleeping on the job."


So, I'll skip the essay I could write here and instead close with this thought; Nap rooms. Good enough for Bell Labs. If I ever have a startup, I'm going to institute them.

Finding Delphi HotFixes on EDN

A problem that some Delphi users may not even know they have is "how do I navigate the EDN website and find Delphi fixes".

Three problems stand in your way:

1. The EDN site doesn't always have things posted under the heading Delphi.  You should probably be looking for your hotfixes using the RAD Studio heading.

2.  The EDN site is slow and quirky and has a user interface probably designed in 1993-era HTML.

3. It's hard to just say "I want to see Delphi hotfixes".

So, here is a nice link you can bookmark that does effectively that:

RAD Studio Recent Uploads

There is only one hotfix of interest to most Delphi 10 Seattle users who are already on Update1 product level:

First is a general IDE stability hotfix (memory leak in modern theme):
http://cc.embarcadero.com/Item/30453

I believe this DCC64 hotfix for Delphi 10 Seattle RTM (pre-update1) is already included in update1:

http://cc.embarcadero.com/Item/30429

Here's hoping someone at Embarcadero has some spare cycles to improve the experience at EDN, which I find a bit dated.

Also please clean this section of the CC articles up. Here's what you should be displaying in the "to download this" section:

To download this, you must have registered [one of]:
     Delphi 10 Seattle, C++ Builder 10 Seattle, or any subscription or package that includes
    one of these products, including RAD Studio, and All Access suites.

I'm not going to copy and paste the current list here, but let's just say it runs to 289 lines and includes SKUs and product bundle offerings listed in detailed format that not even a product or marketing manager could love.

When All You Have Is a Hammer (Part Deux) - A Cry For Professionalism

A couple years ago I wrote a blog post called "When All You Have Is A Hammer, Everything Looks like a Nail".  I was bemoaning the way that software developers (including me) will sometimes use the one thing they know and avoid learning new things.  I have further come to believe that even though we start out passionate, and with a desire to learn more, and to find ways to keep getting better, many of us still periodically have days or weeks where our output decreases in quality, sometimes we even make a complete hash of things, we who perhaps might be expected to know better.  Most often it happens when developers "phone it in" for a day, a week, or a month. Maybe they've been demoralized by micro-management, or continual hectoring.  Let's be kind to people when they are in ruts, because we never know when it will be us.  Let's ask ourselves why we build bad things, and not hector others.   Somehow, our process and its outputs are no good.  Later on this will be discovered, and someone has to clean it up.  Sometimes I am the person who has to clean up my own mess, several weeks or months later, when it becomes clear my changes were a bad idea. Sometimes I am cleaning up someone else's mess.  I ought to do this without grumbling or attacking others. That's part of being a professional. But, since I have observed this category of problems over and over again, and like any hard problem in computing (naming things, being smart and good at your job without becoming an arrogant jerk), it admits of no final solutions.   But there are things we can do, and if the software you make is making the world a better place, then you should care enough to try to do your job better, every week, every month, every year.

First, begin your coding day by asking yourself if the work you're doing today matters. For some of you, that's easy.  Even if curing cancer is out of your reach, then at least increasing the effectiveness of breast cancer screening protocols counts, if you work at a place that does that.  But let's say you spend five minutes considering the business you work in and you still haven't thought of the reason why what you will do today (fix a bug, work on a feature) matters.   Here's my last line, if all else fails;   If what you do today keep your businesses (and your customer's businesses) running, then you're not only earning the dollars that feed you, and your children today, you're also helping other businesses that employ people who are hopefully going to be able to feed and clothe their children, and give those kids somewhere to sleep.  And if you like me are a regular contributor to the charities that matter, then every organization that employs the men and women who give to others, is not only doing its own work, and helping its own employees and customers, it's also, hopefully making the world a better place.  So, you're rooted now? You're ready to begin your day? You care. You are checked in fully.  This is step one to avoiding doing bad engineering work. You have to care about quality.  If you're anything like me, you need to connect all the wires up, from the battery, to the lightbulb, and do that almost daily to keep caring.

Second, all day long, when you find something stupid, that makes no sense, avoid the temptation to stop caring, and push on through, so you can close your bug-tracker ticket, and go home.  Ask the questions. Find the subject-matter experts. Engage in constructive, rational critique of the system. Don't jump into ripping it apart and rewriting it, and don't jump into "must not touch legacy code" fear mode, either. In between, find a way to make the customer's life better. First, the person who is forced by the conditions of their employment to use your software is your customer. Second, the person who signed the cheque to buy your software from your employer is your customer. That business has to succeed, to keep everybody working. Nobody succeeds until everybody succeeds.   This is the second step in avoiding making these mistakes.


Third, keep learning.  If you are stuck on Delphi 5, it's time to move up and onwards. You won't even know what all the reasons why are, until you do. That's just the nature of the game. If you are stuck in a rut, defending the rut won't get you unstuck.  Fine, your code is on a modern delphi version? Next, if all you know is Delphi, then go learn Python. If you do not know any version control tools at all other than zip, then you my friend are not a professional developer.  Go learn one now, I'll wait here.  If all you know is Subversion, go learn Mercurial. If you don't know how to automate your build, at least build a batch file and invoke msbuild and have that build your Delphi, C#, and C++ targets for you.  If you're still using the lame old solution of batch files, then level up, my friend. It's time to learn how to use FinalBuilder, or WANT. If you have a build tool but no CI server software experience, it's time to learn Jenkins (or Hudson).      If you have never done SmokeTesting, you better learn how.  A smoketest job runs on Jenkins (or Hudson) and makes sure everything that is checked into source control still builds, and runs after every single checkin.   Now you have arrived at this point, it might be a good time to notice if the unit tests you write are actually finding any bugs. Does the smoketest job go red not only for compilation failures, but also for unit test failures?   If you have zero unit test failures over 60 days, chances are that you actually have zero meaningful unit tests.  Go analyze why changes that resulted in regressions that your QA person found, were not found in the unit tests.

The more I think about the problems of "hammer-thinking", the more I think there is no real solution other than the basic patterns I see in all good engineers.  Ask why.  Grow and improve. Learn.  Add new tools to your toolbox.  Build new tools when none exist.  Never stop growing.

If you don't want to do this, you should go find a new line of work.  I'm not kidding.



Postscript:

Agile is Dead. Agile values have been totally lost behind the implementation.
Dave Thomas (of The Pragmatic Programmer)

How To Find and Fix Problems In Your Component Packages

A very useful technique that all Delphi developers should know, is how to debug a your own DLLs (IDE experts) or  BPLs (binary component packages) when they exhibit problems when they run inside Delphi itself.  You could think if it as "debugging delphi inside of delphi", except you're not really doing that.   For one thing, the Delphi main IDE executable (BDS.exe) is not your application and it doesn't have full debug symbols, so you can't see all the names of procedures and objects inside of it, but what you can do, and what matters for the purposes of this quick tutorial, is that you can set a breakpoint in a unit that is installed as a designtime Component, or as an IDE Expert, that you have built with full Debug information, and that you have the source code for.

If you have some bug in a commercial library (such as Developer Express, or Teechart), and you have the source code for it (your purchase included more than just BPL and DCP, and DCU files, and also included the .PAS files and the .DPK/.DPROJ project files), then you can find and fix problems with them, as well.

When Not To Use This Technique

• You could just use logging instead.  Inside your design-time component, you could just add some CodeSite logging, to help you figure out if the math or logic in your component is working properly at designtime.
• You could simulate this within your own code, at runtime.
• You could just install MadExcept into the Delphi IDE, and catch your exception and get a stack trace, and figure out from the stack trace, what calling sequence is causing the exception, and fix it without actually needing to debug.  If you don't have MadExcept yet, go buy it. I'll wait. You're back? Okay good.   (There's a free non-commercial version, but you are not the kind of limp-brain who refuses to support a pillar of the community like Matthias, are you? You bought it anyways right? Good for you.)

What is the difference between Runtime and Designtime, for your code?

• Every class that inherits from TComponent has a property called ComponentState.  It is a set of bit-flags, and one of those flags is called csDesigning. When csDesigning bit is on, we are in designtime mode. A common beginner mistake is to forget to protect code that should not run at designtime with guard blocks like this:

procedure TMyComponent.SomePropertySetter(Value:Integer);
begin
  FValue := Value; // designtime+runtime
  if not (csDesigning in MyComponent.ComponentState) then
    DoRuntimeOnlyThing();
end;

• Your objects are instantiated in designtime mode when they are being loaded by Delphi into a Form, when you are in the Design tab of the IDE.  This means your code can do some pretty amazing things for you.  You could add menus that appear when you right-click on your component, to automatically do some task for you that would be helpful when you're building applications using your component. These designtime actions are known as Verbs. The default one is known as Edit. You could have a dialog box open to help you configure your component.  TeeChart makes excellent use of these designtime actions. If you want to see how to add one to your component, there are many excellent tutorials around on the web, check out Zarko Gajic's excellent series on delphi.about.com.

Examine And Understand Your Current IDE and Project Settings, and Make Sure You Know How to Compile and Link A Package So that the BPL that Is Loaded Into The IDE Contains Full Debug Information

• Before you start changing anything, be sure you understand your working environment. Spend a minute to review the the following aspects of your IDE's configuration, and your project configuration:
• Look at your IDE Library  Path. What folders will be searched when you look for DCU and DCP files? 
• Look at your IDE Default DCP and BPL paths. What paths are used? Is it the default or a custom setting?
• Look at your project search path. What folders will be searched when building this project in debug mode, and then again, in release mode? Look at both.
• Look at your project BPL and DCP output path. Are you using the default (that field is blank in your project), or are you overriding it? Where do you expect the BPL to be output to when you build your package?  Is that the same place, when you are in debug mode, or in a different place?
• Look in the IDE Options at the Environment variables, and examine your path. What folders currently contain your BPL files? Do you understand what packages you want to load from what locations?
• You will want your package to be compiled in Debug mode, that is, compiler and linker settings set to include debug information.
• Somehow we need to get the IDE to load our debug package, and every package it depends on, from the set of folders in my PATH.  This might be as simple as hitting compile once on your package, while the Debug settings or the debug configuration are active. In ancient Delphi versions which lack a separate Debug/Release mode profile setting, this is, paradoxically, more straightforward, just check the Compiler debug info and Linker debug info checkboxes are on.  On modern versions it gets more tricky.  For example, if you are a tidy sort of person, and you have set up your packages so that your debug and release mode packages output their BPL files into different folders for your Release and Debug configurations, you may run into some headaches at this point, getting everything to load. In the end you have a set of DLLs that have the file-extension BPL instead. Now you want to load a mix of debug and release packages and say you have both the debug and release BPL output folders in your path. How do you suppose that's going to work out for you? Suppose your BPL depends on another set of BPLs.  The longer I consider the mess that I have sometimes made by having multiple BPL output folders, the more I don't like doing that. I would have only ONE package BPL output folder, but have separate DCP and DCU output folders.  Changing from Debug to Release mode when building a Package, I would want all my outputs to go to the same binary output BPL folder, so I only have ONE folder to manage in my PATH.   Switching executable/dll search PATH values in and out is a source of headaches and pain for me, and I suggest you avoid that pain.
• At this point, before you continue, you should be able to build your package, know that it is built with debug DCUs, and that it will load into your IDE, without errors. To verify that you have accomplished at least this much, you should click Component menu, then click Install Packages, and find your package in your installed packages list.  If the checkbox is unchecked, or if you saw some errors when starting your IDE, or loading your project which uses your component, you have not yet succeeded in compiling your package with debug information turned on, and still retaining a loadable state.  Make sure that BPL that is in the IDE has the same FULLY QUALIFIED FILENAME INCLUDING ITS ENTIRE PATH and not just a package with the same name, in a different directory. If your Debug version of your BPL is output to some different path than your release BPL, then you shall have pain here.  I said that already above, and I'll say it again, and still you'll plough on past me, and then wonder why you can't set breakpoints inside your BPL, and generally debug things.

Click Run → Parameters and Set the Host Application

For most modern delphi versions, type $(BDS)\bin\bds into the Host application edit box.
If you're a sad-sack using some ancient Delphi 5/6/7 version, put the path to your main Delphi32.exe into the Host Application box. Now click Ok.  Now click the debugger Run icon, and hold onto your hat, because Kansas is going Bye-Bye.  Things are about to get weird if this is the first time you've ever done this.

   

If you did this right, you'll start out seeing the Delphi splash screen, then probably your whole IDE (if it's maximized on a one monitor system) will be replaced by another IDE window.  I recommend you un-maximize, and then move your second Delphi instance over to a different monitor at this point.  Which one is which? Look carefully.  Which one has the Debugger desktop-profile selected, and has [Running] in the Caption of the window.  I always keep that one to the left of my workstation (either on a left monitor or to the left side of my main monitor) and I keep the application under test to the right of my monitor, or on a secondary monitor.

Now I can work with the Delphi form editor, perhaps create a new project, drop the component I'm debugging onto the form, and start setting properties via the property inspector. I could open up the Component Editor (which is a designtime-only bit of code that may contain a wizard to help me configure my component's properties), and debug that.  If I'm debugging an IDE Expert, I could click Tools → My Wizard → Do Something, and watch my Wizard Do Something, or have the IDE stop my wizard for me, on first chance exception, because my wizard is accessing an object which is NIL, so of course, I'll get an access violation. Now I can see where that code is, and examine it.

If you find you can not debug your package in this situation, then you messed up in the areas I talk about in the first half of this blog post. If you thought I was belaboring my points above, please rest assured, I was only trying to be helpful to you. Are you really aware of every detail of your environment? What your windows environment variable named PATH is? What every project has as its output folders, not only in Debug mode, but also in Release mode?   What all your compiler and linking settings are, and how they affect your environment? Sure, that's a lot of information. But what are you? Are you a developer, or a whiner?  Woman Up, or Man Up, as the case may be. Dig in. Solve your own problem. Nobody else has the tools in front of them to solve this. You do.  Go to it.

Thoughts on Delphi 10 Seattle, and Idera

Most people who follow what's going on in the Delphi world are aware that Embarcadero was sold by Thoma Cressey Bravo to Idera.  I had not heard of Idera and when the acquisition happened, like a lot of people, I googled the name, found the company website, and did a little reading.   I could see the synergy between their MS SQL Monitoring flagship product line and a lot of Embarcadero's database tooling but the Development Tools market appears to be kind of new to them.     I'm a member of the "MVP" program and I've sent my thoughts and ideas along to some of the folks in the new leadership team, and I'm looking forward to seeing some fresh ideas in growing Delphi's market share, and getting more people using Delphi.

In my opinion, Delphi "10 Seattle" (once I have gotten used to the change away from the XE naming convention, which I quite liked) is a good release, and worth upgrading to.  The extra memory available inside the IDE, due to the large address aware IDE core, is a welcome improvement. The IDE is also faster, and more stable than ever, partly due to the increased memory available, and also because it seems a lot of work has been done to make startup, shutdown, building, running, debugging, and loading and unloading packages work better, and faster.  

That being said, I have to admit, I still get Delphi crashes.  One of the difficult parts of being a Delphi developer is my inability to tell the difference between a Delphi crash caused by a bug in Embarcadero's code, and a crash brought on by the components and experts I've loaded into the IDE. Since I can't work without my components, and only really can choose to turn Experts on or off, it's really a difficult problem.  One hopes, that if stability is still an issue for people, that there might be some ways to tell the difference between a designtime package gone amok, and a real Delphi bug.  Maybe the designtime packages should be loaded into a different process, one that only contains the form designer, and not the rest of the IDE's code-space.   Then the form designer could say "oh snap! something's gone wrong!" without the rest of the IDE going snap too. Some day, right?

The best technique currently available is to run the IDE inside the IDE.  Delphi can debug itself.  Which is cool. Unfortunately, unless you already know a likely way to cause a crash, this is not a great help, as the inner IDE instance will run quite slowly.  I have taken to instrumenting my packages, adding OutputDebugString messages to my packages, in key module initialization and finalization sections, so I can see the packages loading and unloading.  That gets a bit difficult after a while, and in my case, with over 247 designtime and runtime packages (the designtime packages depend on runtime packages, so in total, I have 247 bpl files loading into my IDE) it's too much for me to figure out.  The codebase I work on right now is the largest I've ever worked on. It's impressive, and in Seattle it's working fine, but I have some doubts about the stability of the latest Developer Express packages inside Delphi 10 Seattle. There's something weird going on, and I'm not sure what's up. Once I isolate it, I'll report it both to Developer Express and to Embarcadero, but due to the complexity, it's not easily done.

I have reported some issues with High DPI systems, and I'm pleased to see some progress on this with the recent "Update 1".  It seems there is more still to be done in supporting the latest hardware, including 4K and 5K monitors, with their lovely crisp 150 dpi to 200 dpi pixel densities.   I'm using Windows 10, and Windows 10 is really quite slick on systems with multiple monitors and each monitor can have a different DPI scaling factor.  Delphi has some initial support for this, but where I work we already have our own classes to handle this, so we have the interesting problem of trying to decide how to merge the new VCL DPI awareness functionality with our own.

I'd like to hear your comments, if you're reading this, and you've tried Delphi 10. How are you finding it so far?